Image processing device and control method thereof, imaging device, and program

Abstract

To provide an image processing apparatus that can perform display control based on evaluation values acquired from a detection image of a subject.SOLUTION: An image processing apparatus detects a subject in a picked-up image (S301), acquires information on a first subject area and a second subject area (pupils, face, head) that is part thereof, and detects the feature of the subject areas (S302). The image processing apparatus calculates evaluation values from detected feature points (S303). When the second subject area is detected, control of displaying display frames corresponding to the subject area on a display 113 is performed (S308, S309, S310). When the second subject area is not detected, control of displaying ranging position candidate frames as display of evaluation values on the display 113 is performed (S312).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to display control based on an evaluation value of an image. [Background technology]

[0002] Image processing performed within an imaging device includes processing for detecting a subject area contained in a captured image, and a subject to be targeted for autofocus (AF) is detected. Image display devices such as liquid crystal display devices and electronic viewfinders display the subject detection results to visually present a focus area (AF frame, etc.) to a user. Patent Document 1 discloses a technology for acquiring a video signal from a tracking image sensor via a signal processing unit and detecting the movement of a subject targeted for AF based on the video signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-338352 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, when the subject is a person, it is difficult to identify the ranging position within the subject area when the size of the detected subject is relatively large, such as when the person's eyes cannot be detected from the image data and the entire body is detected, etc. On the display screen of the subject detection results, it may be difficult to visually determine which area within the subject area is set as the ranging position. An object of the present invention is to provide an image processing device capable of display control based on an evaluation value obtained from a detected image of a subject. [Means for solving the problem]

[0005] An image processing device according to an embodiment of the present invention comprises: subject In the image corresponding to First Area And, before recorded One of the photographs To the department In the corresponding image Second Area a detection means for detecting the Multiple From feature point information For each of the feature points A calculation means for calculating an evaluation value; Based on the evaluation value, information about a ranging area corresponding to a ranging position in the image is displayed on the image. and a control means for controlling the display means to output the information to the display means, wherein the control means The control unit displays information indicating the first area, and controls the display of multiple pieces of information indicating areas corresponding to multiple evaluation values ​​as candidates for the ranging area corresponding to the second area within a display area in the image corresponding to the first area, and changes the number of pieces of information by controlling a threshold value for the evaluation value. [Effects of the Invention]

[0006] According to the image processing device of the present invention, display control based on evaluation values ​​obtained from detected images of a subject is possible. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram showing an image processing unit in the first embodiment. [Figure 3] 5 is a flowchart showing evaluation value display control in the first embodiment. [Figure 4] 5 is a flowchart showing feature point detection processing in the first embodiment. [Figure 5] 4 is a flowchart showing a similarity calculation process in the first embodiment. [Figure 6] FIG. 10 is a schematic diagram of a feature calculation process in similarity calculation. [Figure 7] 10 is a flowchart showing a reliability determination process in the first embodiment. [Figure 8] 10 is a table showing the relationship between evaluation values ​​and each element in the first embodiment. [Figure 9] 10 is a flowchart showing control of the number of frames to be displayed in the first embodiment. [Figure 10] FIG. 10 is a schematic diagram of a frame display based on an evaluation value in the first embodiment. [Figure 11] 10 is a flowchart showing frame display control according to the density in the first embodiment. [Figure 12] 5A and 5B are schematic diagrams illustrating frame display control according to the density in the first embodiment. [Figure 13] 10 is a flowchart showing evaluation value display control in the second embodiment. [Figure 14] FIG. 11 is a schematic diagram showing color-coded display of evaluation values ​​in the second embodiment. [Figure 15] 10 is a flowchart showing distance measurement area control in the second embodiment. [Figure 16] 10A and 10B are schematic diagrams showing an example of distance measurement area control in the second embodiment. [Figure 17] 10A and 10B are schematic diagrams showing another example of distance measurement area control in the second embodiment. [Figure 18] 10 is a flowchart showing evaluation value display control in the third embodiment. [Figure 19] 10A to 10C are schematic diagrams showing differences in subject detection size in the third embodiment. [Figure 20] 10 is a flowchart showing evaluation value display control in the fourth embodiment. [Figure 21] FIG. 13 is a schematic diagram of an evaluation value display in the fourth embodiment. [Figure 22] 13 is a flowchart showing a tracking process in the fifth embodiment. [Figure 23] FIG. 13 is a diagram showing a search range for tracking a ranging frame in the fifth embodiment. [Figure 24] 13 is a flowchart showing display control of a distance measurement frame in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. An example in which an image processing device is applied to an imaging device such as a digital still camera or a digital video camera will be shown.

[0009] (First embodiment) 1 is a block diagram showing an example of the configuration of an image processing device 100 according to this embodiment. An imaging device to which the image processing device 100 is applied includes an imaging optical unit 101 and an image sensor 102. The imaging optical unit 101 includes optical components such as lenses and apertures that constitute an imaging optical system, a drive mechanism, and a drive circuit. The drive circuit of the imaging optical unit 101 is electrically connected to a bus 116. The image sensor 102 is a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor, which performs photoelectric conversion on a formed subject image and outputs an electrical signal corresponding to the subject image.

[0010] An A / D conversion unit 103 acquires the analog image signal output by the image sensor 102 and converts it into a digital image signal. An image processing unit 104 acquires the digital image signal from the A / D conversion unit 103 and performs various image processing. The image processing unit 104 is composed of various processing circuit units, buffer memory, etc., and appropriately performs gamma correction, white balance processing, etc. on the A / D converted digital image data.

[0011] The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that transfer data. The bus 116 is a system bus for transmitting control signals from a CPU (Central Processing Unit) 114 (described later) and the like. The bus 117 is a data bus for transferring image data and the like. The data transfer unit 105 is connected to the image processing unit 104 and the bus 117.

[0012] The memory control unit 106 controls the DRAM (Random Access Memory) 107. The DRAM 107 stores data such as still images, moving images, and audio, as well as constants and programs for the operation of the CPU 114. The memory control unit 106 is electrically connected to buses 116 and 117, and writes data to and reads data from the DRAM 107 in accordance with instructions from the CPU 114 or the data transfer unit 105.

[0013] The nonvolatile memory control unit 108 controls a ROM (Read Only Memory) 109. The ROM 109 is electrically erasable and recordable, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) or the like is used. The ROM 109 stores constants, programs, etc. for the operation of the CPU 114. The nonvolatile memory control unit 108 is electrically connected to a bus 116, and writes data to and reads data from the ROM 109 in accordance with instructions from the CPU 114.

[0014] The recording media control unit 110 controls the recording media 111. The recording media 111 is a recording medium such as an SD card. The recording media control unit 110 is electrically connected to buses 116 and 117, and records image data onto the recording media 111 and reads out the recorded data.

[0015] The display control unit 112 controls the display unit 113. The display unit 113 has a liquid crystal display, an electronic viewfinder, etc., and performs display processing of image data transferred from the image processing unit 104, menu screens, etc. When taking still images or moving images, the image processing unit 104 processes image data input from the A / D conversion unit 103 in real time, and the display control unit 112 controls the display of the processed data on the display unit 113. The display control unit 112 is electrically connected to buses 116 and 117.

[0016] The CPU 114 controls the operation of the image processing device 100 via the bus 116. The CPU 114 realizes various functions by executing programs stored in the ROM 109. The operation unit 115 includes switches, buttons, a touch panel, etc. that are operated by the user, and is used for turning the power and shutter on / off, etc. The operation unit 115 is electrically connected to the bus 116, and transmits operation instruction signals from the user to the CPU 114.

[0017] 2 is a block diagram showing an example of the configuration of the image processing unit 104. The image processing unit 104 includes a feature point detection unit 201, a saliency calculation unit 202, a similarity calculation unit 203, a density calculation unit 204, a reliability determination unit 205, and an evaluation value calculation unit 206. Each unit can refer to information acquired up to the previous stage as needed.

[0018] The feature point detection unit 201 acquires image data from the A / D conversion unit 103 and detects feature points within the image. The saliency calculation unit 202 acquires feature point information from the feature point detection unit 201 and calculates the saliency of the feature points. The similarity calculation unit 203 acquires information from the saliency calculation unit 202 and calculates the similarity between feature points. The density calculation unit 204 acquires information from the similarity calculation unit 203 and calculates the density of feature points within an image region. The density is an index that indicates how densely feature points are located within an image region. The reliability determination unit 205 acquires information from the density calculation unit 204 and performs reliability determination mainly using similarity information. The evaluation value calculation unit 206 calculates an evaluation value based on one or more of the saliency, density, and reliability determination results acquired in the previous stages.

[0019] Selective evaluation value display control according to subject detection results will be described in detail with reference to FIG. 3. FIG. 3 is a flowchart showing the overall processing flow of selective evaluation value display control according to subject detection conditions. In this embodiment, an example of frame display based on evaluation values ​​is shown. Known technologies can be used for subject detection methods and methods for detecting specific subject parts such as the pupils, face, and head. For example, subject detection methods include methods that use features such as the eyes, nose, and mouth, and detection methods that use learning algorithms such as neural networks.

[0020] In S301, the CPU 114 executes processing for detecting an object within an image. The object is, for example, a person or an animal. In S302, the feature point detection unit 201 detects feature points from the image area (object area) of the object detected in S301. A specific example of the feature point detection processing will be described later with reference to FIG. 4. In S303, the image processing unit 104 calculates saliency, density, and similarity, and determines reliability, and calculates an evaluation value for the feature points detected in S302. Specific processing will be described later. In S304, the CPU 114 determines whether the object detected in S301 is a specific object. An example of a specific object is a person. If it is determined that the detected object is a specific object, the processing proceeds to S305. If it is determined that the detected object is not a specific object, the processing proceeds to S311.

[0021] In S305, the CPU 114 determines whether the pupils of the specific subject can be detected. If it is determined that the pupils of the specific subject can be detected, the process proceeds to S308. If it is determined that the pupils of the specific subject cannot be detected, the process proceeds to S306. In S306, the CPU 114 determines whether the face of the specific subject can be detected. If it is determined that the face of the specific subject can be detected, the process proceeds to S309. If it is determined that the face of the specific subject cannot be detected, the process proceeds to S307. In S307, the CPU 114 determines whether the head of the specific subject can be detected. If it is determined that the head of the specific subject can be detected, the process proceeds to S310. If it is determined that the head of the specific subject cannot be detected, the process proceeds to S311.

[0022] In S308, S309, S310, and S311, CPU 114 performs control to display a frame in the detected region. That is, in S308, display unit 113 displays an pupil frame for the detected pupil region, and in S309, display unit 113 displays a face frame for the detected face region. In S310, display unit 113 displays a head frame for the detected head region, and in S311, display unit 113 displays a subject frame for the detected subject region.

[0023] After S308, S309, and S310, the process proceeds to S314, and after S311, the process proceeds to S312. In S312, the CPU 114 enables frame display according to the evaluation value calculated in S303 for a subject that was determined not to be a specific subject in S304 or for a subject for which it was determined that part of the subject (eyes, face, head) could not be detected in S305 to S307. In the next S313, a selection process for evaluation value display (frame display) is performed, and then the process proceeds to S314. Details of evaluation value display will be described later.

[0024] In S314, the CPU 114 determines the distance measurement area (focus state detection area) based on the frame display selected in S313. In S315, the CPU 114 performs control to display the distance measurement area determined in S314 on the display unit 113. Then, the series of processes ends.

[0025] The processing in Fig. 3 will be described in detail below with reference to Fig. 4 to Fig. 12. Fig. 4 is a flowchart of the processing performed by the feature point detection unit 201 and the saliency calculation unit 202. In S401, image data of the subject region detected in S301 in Fig. 3 is acquired. In S402, the feature point detection unit 201 specifies an area where feature point detection processing is to be performed on the image data of the detected subject region.

[0026] In S403, the feature point detection unit 201 performs horizontal first derivative filtering on the region specified in S402 to generate a horizontal first derivative image. In S404, the feature point detection unit 201 further performs horizontal first derivative filtering on the horizontal first derivative image acquired in S403 to generate a horizontal second derivative image.

[0027] The processes of S405 and S406 are vertical differential filtering processes of the image, which are executed in parallel with S403 and S404, respectively. In S405, the feature point detection unit 201 performs vertical first differential filtering on the region specified in S402, thereby generating a vertical first differential image. In S406, the feature point detection unit 201 further performs vertical first differential filtering on the vertical first differential image acquired in S405, thereby generating a vertical second differential image.

[0028] The processing in S407 is image differential filtering processing that is executed in parallel with S404. In S407, the feature point detection unit 201 further performs vertical first differential filtering processing on the horizontal first differential image acquired in S403, thereby generating horizontal first differential images and vertical first differential images.

[0029] After S404, S406, and S407, the process proceeds to S408. In S408, the saliency calculation unit 202 calculates the determinant (referred to as Det) of the Hessian matrix (referred to as H) of the differential values ​​acquired in S404, S406, and S407. xx The vertical second derivative obtained in S406 is expressed as L yy The horizontal and vertical first differential values ​​obtained in S407 are expressed as L xy The Hessian matrix H is expressed by the following formula (1), and the determinant Det is expressed by the following formula (2).

number

number

[0030] In S409, the saliency calculation unit 202 determines whether the value of the determinant Det calculated in S408 is equal to or greater than zero. If it is determined that the value of the determinant Det is equal to or greater than zero, the saliency calculation unit 202 proceeds to the processing of S410. If it is determined that the value of the determinant Det is less than zero, the saliency calculation unit 202 proceeds to the processing of S411.

[0031] In S410, the feature point detection unit 201 executes a process of detecting points where the value of the determinant Det is equal to or greater than zero as feature points, and then proceeds to the process of S411. In S411, the feature point detection unit 201 or the saliency calculation unit 202 determines whether or not processing has been performed for all of the subject regions input in S401. If it is determined that processing has been completed for all of the target regions, the feature point detection process of FIG. 4 ends. On the other hand, if there are unprocessed target regions, the process returns to S402, and the processes of S402 to S410 are repeatedly executed.

[0032] FIG. 5 is a flowchart of the processing performed by the similarity calculation unit 203. In S501, the similarity calculation unit 203 calculates a feature amount for the feature point detected by the feature point detection unit 201 in S410 of FIG. 4. FIG. 6 is a schematic diagram illustrating the feature amount calculation processing. A point of interest 601 in the image is indicated by a black dot. A plurality of random line segment patterns 602 are shown around the feature point of interest 601. In this embodiment, an example is shown in which the magnitude relationship between the brightness values ​​at both ends of each line segment is expressed by 1 and 0. The magnitude relationship between all 1s and 0s for the line segment pattern 602 is expressed as a bit string and calculated as the feature amount.

[0033] In S502 of Fig. 5, a determination process is performed as to whether or not feature calculation has been completed for all feature points detected in S410 of Fig. 4. If it is determined that feature calculation has been completed for all feature points, the process proceeds to S503. If it is determined that feature calculation has not been completed, the feature calculation process in S501 is repeated.

[0034] In S503, the similarity calculation unit 203 specifies a feature point for which similarity calculation processing is performed, and then in S504, specifies a counterpart feature point for comparing similarity with the feature point specified in S503. In S505, the similarity calculation unit 203 calculates the Hamming distance (denoted as D) of the feature amounts between the feature points specified in S503 and S504. The bit string of the feature amount of the feature point specified in S503 is denoted as A, and its elements are denoted as Ai. The bit string of the feature amount of the feature point specified in S504 is denoted as B, and its elements are denoted as Bi. The Hamming distance D, which indicates similarity, is expressed by the following formula (3).

number

[0035] In S506, a determination process is performed as to whether or not the calculation process of the Hamming distance D between the feature point focused on in S503 and all feature points has been completed. If it is determined that the calculation process of the similarity (Hamming distance) has been completed, the process proceeds to S507. If it is determined that the calculation process of the similarity (Hamming distance) has not been completed, the process returns to S504, and the processes of S504 and S505 are repeatedly performed.

[0036] In S507, a determination process is performed as to whether or not processing has been completed for all feature points. If it is determined that processing has been completed for all feature points, the similarity calculation process ends. If it is determined that processing has not been completed for all feature points, the process returns to S503, and the processes from S503 to S507 are repeatedly performed.

[0037] FIG. 7 is a flowchart of the process performed by the reliability determination unit 205. In S701, the reliability determination unit 205 specifies a feature point to be focused on in the reliability determination, and in S702, specifies a feature point to be compared with the feature point specified in S701. In S703, the reliability determination unit 205 compares the similarity between the feature points specified in S701 and S702 with a threshold (denoted as M) based on the similarity calculated by the similarity calculation unit 203. A determination process is performed to determine whether the similarity is equal to or greater than the threshold (M or greater). For example, when the number of bits in the feature bit string is 50, the similarity is maximized when the Hamming distance D is 0. The threshold M is set to, for example, 10. However, the number of bits in the bit string and the threshold M are variable. If it is determined that the similarity is equal to or greater than the threshold, the process proceeds to S704. If it is determined that the similarity is less than the threshold, the process proceeds to S705.

[0038] In S704, the reliability determination unit 205 performs a repetition pattern determination for the feature points determined in S703 to have a similarity equal to or greater than the threshold M. A repetition pattern is a situation in which there are multiple similar features within the same image, and indicates that the feature has a high possibility of false detection and low reliability. For example, this could be a shooting scene in which windows of the same shape are arranged consecutively in a building.

[0039] In S705, the reliability determination unit 205 determines whether or not processing for all feature points has been completed for the feature point focused on in S701. If it is determined that the reliability determination processing for the feature point has been completed, the process proceeds to S706. If it is determined that the reliability determination processing for the feature point has not been completed, the process returns to S702 and continues.

[0040] In S706, the reliability determination unit 205 determines whether the processing has been completed for all feature points. If it is determined that the processing has been completed for all feature points, the reliability determination processing ends. If it is determined that the reliability determination processing has not been completed, the processing returns to S701 and the reliability determination processing continues.

[0041] FIG. 8 is a diagram showing an example of a table of evaluation value elements used in the evaluation value calculation by the evaluation value calculation unit 206. The evaluation value calculation unit 206 determines the evaluation value based on the results obtained by the feature point detection unit 201, the saliency calculation unit 202, the density calculation unit 204, and the reliability determination unit 205. FIG. 8 shows an example of the relationship between "high" and "low" evaluation values ​​and the corresponding "high" and "low" saliency, density, and reliability. For example, if any of the saliency, density, or reliability is "high," an evaluation value of "high" is obtained.

[0042] 3, a process is executed in S312 to enable frame display according to the evaluation value for a subject that is not determined to be a specific subject or a subject in which an eye, face, or head is not detected. Hereinafter, the display frame according to the evaluation value is referred to as a ranging position candidate frame.

[0043] Fig. 9 is a flowchart illustrating display control when display of ranging position candidate frames is enabled. Fig. 10 is a schematic diagram showing display examples of the display unit 113 according to the detection status of the subject. Fig. 10(A) shows a display frame 1001 when the subject's pupil is detected, and Fig. 10(B) shows a display frame 1002 when the subject's face is detected. Fig. 10(C) shows a display frame 1003 when the subject's head is detected, and Fig. 10(D) shows a display frame 1004 when the subject (whole body) is detected. The display example of Fig. 10(E) shows a case where display of ranging position candidate frames is enabled, and multiple ranging position candidate frames 1005 are displayed.

[0044] In S901 of FIG. 9, the image processing unit 104 determines whether the saliency of the feature point calculated by the feature point detection unit 201 and the saliency calculation unit 202 is equal to or greater than a threshold (denoted as N). As a specific example, the value of the determinant Det of the Hessian matrix H corresponds to the saliency, and the threshold N is set to 0.5. The threshold N is a fixed value or a variable value. When the threshold N is set to a variable value, the number of ranging position candidate frames to be displayed can be controlled by changing the threshold N. If it is determined that the value (evaluation value) of the determinant Det is equal to or greater than the threshold (N or greater), the process proceeds to S902. If it is determined that the value of the determinant Det is less than the threshold N, the process proceeds to S903.

[0045] In S902, the display unit 113 displays the ranging position candidate frames. In the next S903, a determination process is executed to determine whether the display process for the ranging position candidate frames has been completed for all evaluation values. If it is determined that the process for all evaluation values ​​has been completed, the display process for the ranging position candidate frames ends (proceed to S313 in FIG. 3). On the other hand, if it is determined that the process has not been completed, the process returns to S901, and the processes from S901 to S903 are repeatedly executed.

[0046] FIG. 11 is a flowchart illustrating frame display control according to density when ranging position candidate frame display is enabled. FIG. 12 is a schematic diagram showing display examples on the display unit 113 with and without frame display control according to density. FIG. 12(A) shows a display example 1201 when the frame display control of FIG. 11 is not performed, in which ranging position candidate frames not based on evaluation values ​​(density, prominence) are comprehensively displayed. If there are a large number of ranging position candidate frames displayed, it may be difficult for the user to select one. In contrast, FIG. 12(B) shows a display example 1202 when the frame display control of FIG. 11 is performed. Ranging position candidate frames are displayed based on evaluation values ​​(density, prominence).

[0047] 11, the density calculation unit 204 specifies an area for which density is to be calculated. In this embodiment, an example is shown in which the subject area is divided into multiple blocks and density is calculated for each block. In S1102, the density calculation unit 204 calculates the density of feature points in the area specified in S1101. In this embodiment, the density is calculated by counting the number of feature points in the specified area.

[0048] In S1103, a determination process is performed as to whether the density calculated in S1102 is equal to or greater than a threshold value (denoted as P). The threshold value P is a fixed value or a variable value. If the threshold value P is a variable value, the number of ranging position candidate frames to be displayed can be controlled by controlling the threshold value P. If it is determined in S1103 that the density is equal to or greater than the threshold value (P or greater), the process proceeds to S1104. If it is determined that the density is less than the threshold value P, the process proceeds to S1106.

[0049] In S1104, the density calculation unit 204 specifies a feature point whose saliency is higher than the threshold value in the area determined in S1103 to have a density equal to or higher than the threshold value P. In S1105, the display unit 113 displays a ranging position candidate frame centered on the feature point specified in S1104. This makes it possible to display a representative frame with a high evaluation value for an area with high density. Furthermore, in S1106, the display unit 113 displays a ranging position candidate frame according to the saliency for an area determined in S1103 to have a density less than the threshold value P.

[0050] After S1105 and S1106, in S1107, a determination is made as to whether processing has been completed for the entire target region. If it is determined that processing has been completed for the entire target region, the frame display control according to the density is terminated. On the other hand, if it is determined that processing has not been completed for the entire target region, the process returns to S1101, and the processes from S1101 to S1106 are repeatedly executed.

[0051] In S313 of Fig. 3, a process is performed to select an arbitrary frame from the displayed candidate frames for ranging positions. An arbitrary frame is selected from the candidate frames for ranging positions displayed as in the example of Fig. 10(E) or Fig. 12(B). There are automatic selection processes based on predetermined conditions, manual selection processes, and semi-automatic selection processes based on the presentation of recommended options. Here, an example of frame selection by user operation is shown. In an embodiment equipped with a touch panel, the user can specify the selection by touching the frame they want to select as the ranging position from among multiple frames. As a selection operation method, button operation, stick operation, etc. may be used in addition to touch operation.

[0052] In this embodiment, an evaluation value is calculated using the saliency, density, and reliability of feature points detected in an image, and evaluation value display control, for example, display control of a candidate ranging position frame, is performed according to the subject detection situation. As a result, even if the eyes, face, or head of a specific subject cannot be detected, the frame display allows the user to visually recognize the candidate ranging position. Furthermore, by controlling the threshold for displaying the candidate ranging position frame, it is possible to display a region frame with a relatively high evaluation value. By selecting the ranging area that the user intends, it is possible to determine a ranging area that balances ranging performance and the user's intention.

[0053] (Modification of the first embodiment) In the first embodiment, a saliency calculation method based on feature point detection using a Hessian matrix was described, but other calculation methods such as edge detection or corner detection can also be used. Regarding feature point feature amount calculation, a calculation method based on the magnitude relationship between the feature point and its surrounding luminance values ​​was described, but a feature amount calculation method based on hue or saturation can also be used. Regarding frame display control based on density, a method was described in which a representative ranging position candidate frame was displayed by specifying an area with high saliency, but a representative ranging position candidate frame can also be determined using the center of gravity of the area, etc.

[0054] Although the first embodiment does not mention the color or line type of the display frame, in the modified example, a process is performed to change the color or line type of the display frame according to the calculated evaluation value. For example, the display unit 113 displays a first display frame for the entire subject in a first color or line type, and a second display frame for a part of the subject in a second color or line type according to the evaluation value of each part, making it easier for the user to see.

[0055] (Second embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, an example of frame display was shown as a display method when evaluation value display control is performed in accordance with the detection status of the subject. In contrast, in this embodiment, an example of color-coded display is shown as a method of displaying evaluation values ​​in accordance with the detection status of the subject. Note that detailed description of matters and configurations similar to those in the first embodiment will be omitted, and the description will focus on differences from the first embodiment. This method of omitting description will be the same in the embodiments described below.

[0056] Fig. 13 is a flowchart illustrating display control when color-coded display is enabled, and the display unit 113 is controlled by the CPU 114 and the display control unit 112. Fig. 14 is a schematic diagram showing an example of color-coded display on the screen of the display unit 113 according to the subject detection status.

[0057] In this embodiment, an example is shown in which the subject area is divided into rectangular blocks and each rectangular block is displayed in a color according to the saliency of the feature points. In the display area corresponding to the subject area, rectangular blocks with high evaluation values ​​are displayed in a first color, rectangular blocks with medium evaluation values ​​are displayed in a second color, and rectangular blocks with low evaluation values ​​are displayed in a third color. In FIG. 14, the color distribution is expressed in shades of gray. That is, rectangular blocks with the first color in the color distribution within the subject area are displayed in the darkest color, and rectangular blocks with the third color are displayed in the lightest color. While an example of a three-level color distribution (high, medium, low) based on the evaluation value is shown, this can be expanded to display evaluation values ​​in four or more levels.

[0058] 13, the saliency calculation unit 202 specifies a rectangular block to be displayed in a different color within the subject area. In S1302, the saliency calculation unit 202 determines whether the saliency within the rectangular block specified in S1301 is equal to or greater than a first threshold (denoted as α). If it is determined that the saliency within the rectangular block is equal to or greater than the threshold α, the process proceeds to S1304. If it is determined that the saliency within the rectangular block is less than the threshold α, the process proceeds to S1303.

[0059] In S1303, the saliency calculation unit 202 determines whether the saliency within the rectangular block specified in S1301 is equal to or greater than a second threshold (denoted as β, where "α>β"). If it is determined that the saliency within the rectangular block is equal to or greater than the threshold β, the process proceeds to S1305. If it is determined that the saliency within the rectangular block is less than the threshold β, the process proceeds to S1306.

[0060] In S1304, the display unit 113 displays the inside of the rectangular block whose saliency was determined to be equal to or greater than the threshold value α in S1302 in a first color. The density corresponding to the first color is density 1. In S1305, the display unit 113 displays the inside of the rectangular block whose saliency was determined to be equal to or greater than the threshold value β in S1303 in a second color. The density corresponding to the second color is density 2. In S1306, the display unit 113 displays the inside of the rectangular block whose saliency was determined to be less than the threshold value β in S1303 in a third color. The density corresponding to the third color is density 3.

[0061] In this embodiment, two threshold levels, α and β, are set, with the relationship "α>β." Three display color levels are set: density 1, density 2, and density 3. Density 1 corresponds to an evaluation value of "high," density 2 corresponds to an evaluation value of "medium," and density 3 corresponds to an evaluation value of "low." The threshold level can be set to any level to change the gradation of the display color.

[0062] After S1304, S1305, and S1306, in S1307, a determination is made as to whether processing has been completed for all target regions. If it is determined that processing has been completed for all target regions, the color-coded display processing according to the evaluation value is terminated. If it is determined that processing has not been completed for all target regions, the process returns to S1301, and the processing from S1301 to S1306 is repeatedly executed.

[0063] The process of determining the ranging area according to the display color will be described with reference to Fig. 15. Fig. 15 is a flowchart illustrating the process in S314 of Fig. 3, that is, the process of determining the ranging area according to the display color of the area selected in S313.

[0064] In S1501, the CPU 114 determines whether the density corresponding to the display color of the area selected in S313 of FIG. 3 is density 1 or density 2. Here, a determination process is performed to determine whether the evaluation value of the selected area is equal to or greater than a threshold value β. If it is determined that the density corresponding to the display color of the selected area is density 1 or density 2, the process proceeds to S1503. If it is determined that the density corresponding to the display color of the selected area is density 3, the process proceeds to S1502.

[0065] In S1502, if the density corresponding to the display color of the selected area is density 3 (evaluation value "low"), CPU 114 changes the ranging area to include an area of ​​density 1 (evaluation value "high"). Then, the process proceeds to S1503. In S1503, CPU 114 determines the ranging area according to the display color of the area selected in S313 of FIG. 3. After S1503, the process ends.

[0066] FIG. 16 is a schematic diagram showing differences in ranging areas according to the selected evaluation value. FIG. 16(A) shows the ranging area frame with density 1 (evaluation value "high") selected in S313 of FIG. 3. FIG. 16(B) shows the ranging area frame determined according to the display color (evaluation value) of the area selected in FIG. 16(A). FIG. 16(C) shows the ranging area frame with density 3 (evaluation value "low") selected in S313 of FIG. 3. FIG. 16(D) shows an example in which the ranging area frame has been changed to include an area with a high evaluation value, based on the display color (evaluation value) of the area selected in FIG. 16(C). In other words, the ranging area frame includes an area with a "low" evaluation value and multiple surrounding areas with a "high" evaluation value. Here, an example is shown in which the area used for ranging processing and subsequent subject tracking processing is changed by internal processing when the selected evaluation value is low.

[0067] FIG. 17 is a schematic diagram showing an example in which a plurality of candidate frames for ranging areas are displayed and reselected when the density corresponding to the display color of the selected area is density 3 (evaluation value "low"). In FIG. 17(A), the area with density 3 (evaluation value "low") selected in S313 of FIG. 3 is shown in the ranging area frame. In FIGS. 17(B), (C), and (D), ranging position candidate frames are displayed that include an area with density 3 (evaluation value "low") and surrounding areas with high evaluation values. The area reselected for the ranging position candidate frame is determined as the ranging area. In FIG. 17(B), an area with a high evaluation value is located to the lower right of the area with density 3 (evaluation value "low"), and in FIG. 17(C), an area with a high evaluation value is located to the upper right of the area with density 3 (evaluation value "low"). In FIG. 17(D), an area with a high evaluation value is located above the area with density 3 (evaluation value "low"). In either case, a ranging position candidate frame including an area with a relatively high evaluation value is displayed in accordance with the display color (evaluation value) of the ranging area frame shown in Fig. 17(A). The area reselected by the user operation is determined as the ranging area.

[0068] In this embodiment, the user can visually recognize the ranging position candidates by displaying them in different colors based on the evaluation values. Furthermore, by controlling the color-coded display using a threshold, areas with relatively high evaluation values ​​are displayed in a distinguishable manner. The user can easily select the desired ranging area, enabling the user to determine a ranging area that satisfies both ranging performance and the user's wishes.

[0069] (Modification of the second embodiment) In the second embodiment, a method of displaying ranging position candidate frames by coloring areas divided into rectangular blocks has been described. However, in a modified example, the display areas are not limited to rectangular blocks. For example, a grouping process is performed on areas with similar evaluation values, and ranging position candidate frames are displayed using the grouped areas. Furthermore, instead of coloring, icon display may be used so that the user can distinguish between high and low evaluation values.

[0070] In a modified example, the number of feature points, their strength, and information on the feature amount are used in the process of calculating the evaluation value. The number of feature points is the number of detected feature points, their density, etc. The strength of a feature point is an index that indicates the strength of a feature, such as a cross edge or corner, and the strength decreases when the image area has low contrast, blur, or shaking. The feature amount is an index that indicates the feature point and its surrounding situation, and tends to have similar values ​​for repeated patterns, etc. If there are many feature points, if the strength of the feature points is high, or if it is determined from the feature amount that there are no similar areas, the evaluation value calculated is "high."

[0071] (Third embodiment) Next, a third embodiment of the present invention will be described. In the above embodiment, an example was shown in which the detection status of a subject was used as a method for controlling the display of an evaluation value. Specifically, display control is performed according to an evaluation value obtained from an image depending on the detection status of the eyes, face, and head of a specific subject such as a person. In contrast, this embodiment shows an example in which the display of an evaluation value is controlled according to the size of a detected subject, regardless of the type of subject or the specific detection method.

[0072] 18 is a flowchart of evaluation value display control according to the size of a detected subject. In S1801, a subject area is detected from a captured image. The subject here may be, for example, a human head, a dog, or a flower. In S1802, the feature point detection unit 201 detects feature points from the subject area detected in S1801. The processing in S1802 is the same as S302 in FIG. 3.

[0073] In S1803, the evaluation value calculation unit 206 calculates the evaluation values ​​of the feature points detected in S1802. The processing of S1803 is the same as that of S303 in Fig. 3. In S1804, the CPU 114 performs processing to determine whether the size of the object region detected in S1801 is equal to or greater than a threshold value (denoted as Q). If it is determined that the size of the object region is equal to or greater than the threshold value Q, the processing proceeds to S1805. If it is determined that the size of the object region is less than the threshold value Q, the processing proceeds to S1809.

[0074] In S1805, the CPU 114 performs processing to enable the evaluation value display corresponding to the evaluation value calculated in S1803, and ranging position candidates are displayed. Evaluation value display methods include the frame display method described in the first embodiment and the color-coded display method described in the second embodiment. In S1806, processing is performed to select an arbitrary evaluation value display from the evaluation value displays displayed in S1805. The processing of S1806 is the same as that of S313 in FIG. 3. For example, as a manual selection method, selection is made in accordance with a touch operation, button operation, stick operation, etc. by the user.

[0075] In S1807, the CPU 114 performs processing to determine the ranging area based on the evaluation value display selected in S1806. The processing in S1807 is the same as the processing in S314 in Fig. 3. In S1808, the display unit 113 displays the ranging area determined in S1807. The processing in S1808 is the same as the processing in S315 in Fig. 3. After S1808, the processing ends.

[0076] FIG. 19 is a schematic diagram showing differences in the size of the detected subject area. FIG. 19(A) shows a situation in which the head 1901 of a subject person is detected. Because the size of the detected subject area is small, no evaluation value is displayed. FIG. 19(B) shows a situation in which the head 1902 of a subject person is detected, similar to FIG. 19(A). Because the size of the detected subject area is large, it is difficult to determine the ranging position within the subject area. In such an example, evaluation value display control is enabled according to the size of the subject area, and ranging position candidates are displayed.

[0077] 18, display unit 113 displays a subject outline in the detected subject region. If threshold Q is a variable value, it is possible to control the display of evaluation values ​​according to the size of the detected subject region by changing threshold Q. After S1809, the process ends.

[0078] In this embodiment, the evaluation value display is controlled according to the size of the detected subject area (subject size). To address the issue that the ranging area is visually difficult to understand when the subject size is equal to or larger than a predetermined size, it is possible to visually display ranging position candidates regardless of the subject detection method. Furthermore, by controlling the evaluation value display using a threshold value, areas with relatively high evaluation values ​​are displayed in a distinguishable manner. The user can easily select the desired ranging area, making it possible to determine a ranging area that satisfies both ranging performance and the user's wishes.

[0079] The above-described embodiment provides an image processing device that enables control of displaying candidate ranging positions based on evaluation values ​​obtained from a detected image of a subject, and selection of a ranging position. The display of candidate ranging positions is an example, and the present invention can be applied to display control of a subject tracking frame or the like as various information based on evaluation values ​​for a subject region. The present invention can also be applied to detecting multiple subjects. For example, a first and a second subject may be detected. Assume that a first subject region (e.g., the entire body) is detected for the first subject, and a second subject region (part) is detected within that region. In this case, control is performed to output first information indicating that the first subject region is a second subject region to a display unit. Assume that a first subject region (e.g., the entire body) is detected for the second subject, and a second subject region (part) is not detected within that region. In this case, control is performed to output second information based on an evaluation value calculated for the first subject region within that region to a display unit.

[0080] (Fourth embodiment) A fourth embodiment of the present invention will be described with reference to FIGS. 20 and 21. In the first to third embodiments, a method of selectively controlling the display of an evaluation value using whether or not a ranging area of ​​a subject is detected has been described. Specifically, when a local ranging area, such as the pupil or face of a specific subject, is detected, control is performed to disable the display of an evaluation value according to the evaluation value obtained from the image. Furthermore, when a local ranging area is not detected, control is performed to enable the display of an evaluation value according to the evaluation value obtained from the image. On the other hand, when the display of an evaluation value according to the evaluation value obtained from the image is disabled, there is a possibility that the ranging area determined solely based on the subject detection result will be an area not intended by the user. In this embodiment, a configuration is shown in which the display of an evaluation value is enabled and controlled regardless of whether or not a ranging area is detected by the subject detection means.

[0081] 20 is a flowchart of evaluation value display control in this embodiment. In S2001, a process is performed to detect a first subject from an image on the screen. The first subject here is the entire body of a person, a car, a train, etc.

[0082] In S2002, feature points are detected by the feature point detection unit 201 from the region of the first subject detected in S2001. In S2003, an evaluation value is calculated according to the feature points detected in S2002. As in the above embodiment, the evaluation value is calculated based on one or more of the saliency, density, similarity, and reliability based on similarity of the feature points in the image. The processes of S2002 and S2003 are the same as those of S302 and S303 in FIG. 3.

[0083] In S2004, a process is performed to display a subject outline corresponding to the subject area detected in S2001. In S2005, CPU 114 determines whether a second subject can be detected within the subject area detected in S2001. The second subject here refers to a local area such as a person's eyes or face, or the driver's seat of a train. If it is determined that the second subject can be detected, the process proceeds to S2006. If it is determined that the second subject cannot be detected, the process proceeds to S2007.

[0084] In S2006, display unit 113 displays a subject frame corresponding to the detected second subject region. In S2007, CPU 114 enables evaluation value display (display of ranging candidate frame) according to the evaluation value calculated in S2003. FIG. 21 is a diagram showing a case in which the second subject detection result and evaluation value display are displayed side by side in this embodiment. This example shows an example in which the first subject is a train and the second subject is the train's driver's seat. A detection frame 2101 of the train, which is the first subject detection result, is displayed, and within detection frame 2101, a detection frame 2102 of the train's driver's seat, which is the second subject detection result, and an evaluation value display frame 2103 according to the calculated evaluation value are displayed. Multiple evaluation value display frames 2103 are included in detection frame 2101 corresponding to the first subject region, and the calculation result of the evaluation value is displayed in an area different from the area of ​​the driver's seat, which is the second subject region. The evaluation value display method is not limited to a frame display method, and icon display methods, color-coded display methods, etc. may also be used.

[0085] In S2008 of Fig. 20, a process is performed to select an arbitrary area from the detection frame corresponding to the second subject area displayed in S2006 and the evaluation value display information displayed in S2007. The process of S2008 is the same as S313 of Fig. 3, and the selection method can be a touch operation, a button operation, a stick operation, or the like.

[0086] In S2009, the CPU 114 determines the ranging area based on the area selected in S2008. In S2010, the CPU 114 performs processing to display the ranging area determined in S2009 on the display unit 113. The processing in S2009 and S2010 is common to the processing in S314 and S315 in FIG.

[0087] In this embodiment, regardless of whether or not a ranging area is detected by the subject detection means, the evaluation value display is controlled to be valid, thereby making it possible to determine a ranging position that reflects the user's intention, regardless of the subject detection result. In other words, it is possible to prevent the ranging area determined based on the subject detection result from being an area that differs from the user's intention. Furthermore, as in the previous embodiment, threshold control related to the evaluation value display allows the user to select an area with a relatively high evaluation value, making it possible to determine a ranging area with high ranging performance.

[0088] (Fifth embodiment) A fifth embodiment of the present invention will be described with reference to Fig. 22 to Fig. 24. In the first to fourth embodiments, detection of a subject, detection of ranging area candidates, a control method for evaluation value display, selection of a ranging area by a user, etc. will be described. In this embodiment, in addition to processing for searching for and tracking a subject area between frames, processing for searching for and tracking a ranging area between frames when the ranging area is an area different from the subject area will be described.

[0089] 22 is a flowchart illustrating the processing of subject tracking control in this embodiment. The processing of this flowchart starts when a subject is detected by the subject detection means. In S2201, the image processing unit 104 performs tracking processing of the detected subject area. The tracking processing is a process in which correlation processing is performed between an image of the subject area to be tracked and the latest frame image for frames of captured images acquired at a certain imaging cycle, and the area or position with the highest correlation is determined as the subject movement area or position of the tracked subject.

[0090] Next, in S2202, the CPU 114 determines whether a ranging area is defined other than the subject area. Regarding the ranging area, a part of the detected main subject area or a subject area of ​​a size suitable for ranging that includes a part of the detected main subject area may be defined as the ranging area. Alternatively, the user may select a ranging area from candidate ranging areas determined from the evaluation value and define it as the ranging area. If a ranging area is not defined, the process ends, and if a ranging area is defined, the process proceeds to S2203.

[0091] In S2203, tracking processing of the ranging area is performed. The tracking processing of the ranging area is performed by the image processing unit 104, similar to tracking of the subject area. In the tracking processing, correlation processing is performed between the image of the ranging area to be tracked and the latest frame image, and the area with the highest correlation is determined as the tracking result of the ranging area. This will be specifically described with reference to FIG. 23.

[0092] 23 is a diagram showing an example of displaying the tracking result of the ranging area as a ranging frame, and an example of the range in which correlation processing is performed on the latest frame image when tracking processing of the ranging area is performed. A captured image 2301 is the entire captured image and is an example of a displayed image. A subject frame 2302 is an example in which the main subject area detected from the captured image 2301 is displayed as a subject frame.

[0093] A ranging frame 2303 is an example in which a ranging area determined as an area including part of the subject area is displayed as the ranging frame. The ranging frame 2303 is set so that the user can clearly distinguish it visually from the subject frame 2302, for example, by using a different frame color or a different line type from the subject frame 2302, such as a dotted line. The area within the ranging frame 2303 (ranging area) may be determined as an area detected from the captured image 2301, or may be determined by the user selecting from candidate ranging areas determined from evaluation values.

[0094] For the area (subject area) displayed by the subject frame 2302 and the ranging area displayed by the ranging frame 2303, by tracking the subject area and ranging area as described in Figure 22, a stable frame display can be provided to the user between frames.

[0095] Area 2304 shown in FIG. 23 is an example of a search area for which correlation processing is performed on captured image 2301 when tracking processing of the ranging area is performed. In tracking processing of the ranging area in S2203 shown in FIG. 22, correlation processing may be performed between the image of the ranging area and captured image 2301, which is the latest frame image. At this time, there is a method of performing correlation processing on search area 2304 determined based on subject outline 2302, without performing correlation processing with the entire captured image 2301. Search area 2304 may be determined, for example, as an area with a fixed size outside subject outline 2302, or as an area with a size that is a fixed ratio to the size of subject outline 2302. Not using the entire frame image as search area 2304 in tracking processing of the ranging area not only shortens processing time and reduces power consumption, but also improves the accuracy of tracking processing of the ranging area by limiting the range of correlation processing. Figure 23 shows a format (frame representation format) in which the subject area is represented by a subject frame 2302 and the ranging area is represented by a ranging frame 2303, but each area may also be presented to the user in a different representation format, such as a representation format that shows only the corners of each area.

[0096] FIG. 24 is a flowchart illustrating display control of the ranging area. The processing of this flowchart begins when a subject is detected by the subject detection means and subject tracking begins. In S2401, the CPU 114 determines whether a ranging area is defined other than the subject area. As with S2202 in FIG. 22, a subject area of ​​a size suitable for ranging that includes a part of the detected main subject area may be defined as the ranging area, separate from the detected main subject area. Alternatively, the user may select from candidate ranging areas determined from the evaluation value and define the ranging area. If a ranging area is not defined, the processing proceeds to S2405; if a ranging area is defined, the processing proceeds to S2402.

[0097] In S2402, the ranging frame corresponding to the ranging area is displayed in a blinking manner for a fixed period of time. After that, the ranging frame continues to be displayed. The blinking display notifies the user that the ranging frame has started to be displayed in addition to the subject frame. The blinking display for a fixed period of time is one example, and the start of the display may be expressed in other ways, such as displaying the ranging frame in a color different from the color used during normal display for a fixed period of time, or gradually darkening the display. Next, the process proceeds to S2403.

[0098] In S2403, the CPU 114 determines whether the ranging area is still being defined. If the ranging area is still being defined, the determination process in S2403 is repeated until the ranging area is no longer defined. When the definition of the ranging area separate from the subject area is completed, the process proceeds to S2404. The definition of the ranging area is completed, for example, when no subject area of ​​a size suitable for ranging is detected other than the detected main subject area, or when a part of the main subject area is not detected. Alternatively, this may be the case when the user cancels the specification of the ranging area, or when the ranging area tracking control is lost, i.e., the target is lost.

[0099] In S2404, the ranging frame is flashed for a fixed period of time. Thereafter, the ranging frame is erased and display of the ranging frame is terminated. The flashing display notifies the user that display of the ranging frame will soon be terminated. Flashing for a fixed period of time is one example, and the end of display may be indicated in other ways, such as displaying the ranging frame for a fixed period of time in a color different from the color it is normally displayed in, or gradually fading the display. In the next S2405, the CPU 114 determines whether or not the subject area is being tracked. If the subject area is being tracked, the process proceeds to S2401, and if tracking control of the subject area has ended, the process of this flowchart ends.

[0100] In this embodiment, when the ranging area is defined separately from the subject area, tracking control of the ranging area is performed. Stable tracking control of the ranging area can be performed between frame images not only when the ranging area coincides with the subject area, but also when the ranging area is a different area from the subject area. As a result, more stable ranging control can be achieved.

[0101] Although the preferred embodiments of the present invention have been described, the present invention is not limited to the above-described embodiments, and various modifications, changes, and combinations are possible within the scope of the gist of the present invention.

[0102] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0103] 100 Image processing device 104 Image processing unit 112 Display control unit 113...Display section 114 CPU

Claims

1. A detection means for detecting a first region in an image corresponding to a subject and a second region in the image corresponding to a part of the subject; a calculation means for calculating an evaluation value for each of a plurality of feature points in an image from information of the feature points; a control means for controlling, based on the evaluation value, outputting information about a ranging area corresponding to the ranging position in the image to a display means that displays the image; The control means controls the display means to display information indicating the first region, and to display a plurality of pieces of information indicating regions corresponding to a plurality of the evaluation values ​​as candidates for the ranging region corresponding to the second region in a display region in the image corresponding to the first region, and controls to change the number of the plurality of pieces of information by controlling a threshold value of the evaluation value.

1. An image processing device comprising:

2. The control means selectively controls outputting to the display means, together with information indicating the first area detected by the detection means, either first information indicating the second area detected by the detection means or second information based on the evaluation value calculated by the calculation means.

2. The image processing device according to claim 1, wherein:

3. When the first area is detected by the detection means and the second area is detected by the detection means in the first area, the control means controls to output the first information in the area to the display means, and when the first area is detected by the detection means and the second area is not detected by the detection means in the first area, the control means controls to output the second information in the area to the display means.

3. The image processing device according to claim 2.

4. The detection means detects the entire subject as the first region and detects a part of the subject as the second region.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. acquiring information about feature points in the first region; The control means determines an evaluation value calculated from the feature points by the calculation means and controls output of the second information to the display means.

4. The image processing device according to claim 2, wherein the image processing device is a computer.

6. the display means for displaying the second information in a display area corresponding to the first area when the second area is not detected by the detection means; 4. The image processing device according to claim 3.

7. The calculation means calculates the evaluation value based on at least one of the saliency, density, and similarity of feature points in the image, and the reliability based on the similarity.

7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. The control means controls to output, as the first information, information of a display frame indicating that the area is the second area to the display means, or to output, as the second information, information of a display frame corresponding to the evaluation value to the display means.

4. The image processing device according to claim 2, wherein the image processing device is a computer.

9. The control means controls changing the number of display frames corresponding to the evaluation value by comparing the evaluation value with a threshold value.

9. The image processing device according to claim 8,

10. The control means performs control to select one of the display frames corresponding to each of the plurality of evaluation values, and determines a distance measurement area corresponding to the selected display frame.

10. The image processing device according to claim 8 or claim 9.

11. the control means controls the display means to output, as the second information, information on a display color corresponding to the evaluation value; The display means displays a color distribution based on the evaluation value.

4. The image processing device according to claim 2, wherein the image processing device is a computer.

12. The control means controls the change of the display color by comparing the evaluation value with a threshold value. The image processing device according to claim 11 .

13. The control means performs control to select one of the display colors corresponding to each of the plurality of evaluation values, and determines a distance measurement area corresponding to the selected display color.

13. The image processing device according to claim 11 or 12.

14. The control means compares the size of the second area with a threshold value to determine whether to control output of the second information to the display means.

4. The image processing device according to claim 2, wherein the image processing device is a computer.

15. When the size of the second area is equal to or larger than a threshold value, the control means controls to output, to the display means, information on a display frame or a display color corresponding to the evaluation value as the second information.

15. The image processing device according to claim 14.

16. The calculation means calculates a similarity between feature points in the image from feature amounts, determines a reliability based on the similarity, and determines the evaluation value based on the reliability determination result.

16. The image processing device according to claim 1,

17. The control means controls to change the number of display frames or display colors corresponding to the evaluation value by changing a threshold value for the density of feature points in the image.

8. The image processing device according to claim 7,

18. The control means controls the display means to display the display frame for the distance measurement area corresponding to the feature point whose saliency is higher than a threshold in the area where the density is equal to or higher than a threshold.

18. The image processing device according to claim 17,

19. the display means acquires the second information and displays a color distribution based on the evaluation value; The control means controls the display means to display a distance measurement area including a plurality of display colors corresponding to a plurality of different evaluation values.

4. The image processing device according to claim 2, wherein the image processing device is a computer.

20. The control means controls the display means to display the display frame for a distance measurement area including a plurality of distance measurement positions.

9. The image processing device according to claim 8,

21. The detection means detects the pupil, face, or head of the subject as the second region.

5. The image processing device according to claim 4.

22. The control means performs tracking control of a subject area corresponding to the subject by correlation processing between a first image specified based on the detected subject and an input second image.

22. The image processing device according to claim 1,

23. The control means determines a part of the detected first area as a ranging area and performs tracking control of the ranging area.

23. The image processing device according to claim 22.

24. An image processing device according to any one of claims 1 to 23 is provided. An imaging device characterized by:

25. A detection step of detecting a first region in the image corresponding to a subject and a second region in the image corresponding to a part of the subject; a calculation step of calculating an evaluation value for each of a plurality of feature points in an image from information of the feature points; a control step of controlling, based on the evaluation value, to output information about a ranging area corresponding to a ranging position in the image to a display means that displays the image; In the control step, the display means is controlled to display information indicating the first region, and to display a plurality of pieces of information indicating regions corresponding to a plurality of the evaluation values ​​as candidates for the ranging region corresponding to the second region in a display region in the image corresponding to the first region, and a number of pieces of information is changed by controlling a threshold value of the evaluation value.

2. A method for controlling an image processing apparatus comprising:

26. The steps according to claim 25 are executed by a computer of an image processing device. A program characterized by:

Images