Image processing apparatus, imaging apparatus, image processing method, and program

Abstract

To more efficiently remove a subject different from the intention of a user from an object to be detected.SOLUTION: An image processing apparatus comprises: detection means that detects a subject in an image; display control means that causes display means to display the image and a detection area for the subject; designation means that designates the detection area displayed by the display control means as a non-detection area; holding means that holds information on the non-detection area designated by the designation means; and determination means that, on the basis of the information on the non-detection area, determines a non-detection area in another image that is displayed on the display means later than the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Object detection processing, which detects objects from images, is applied to the functions of imaging devices such as digital cameras. Until now, object detection processing has focused on specific categories of objects, such as human faces, facial organs (eyes, nose, mouth), and the entire human body. In recent years, with the development of deep learning, technology has been realized to detect animals, vehicles, and other objects by training detectors to determine object-likeness using information from various categories of objects. In digital cameras, object detection processing is applied to autofocus (AF) technology, which automatically focuses on the detected object as the subject, autoexposure (AE) processing, which controls the exposure to the correct level, and autowhitebalance (AWB) processing, which performs color correction.

[0003] In an AF function using object detection processing, the camera focuses on an object recognized as a subject, but that object may differ from the camera user's intention. A case where the object differs from the user's intention is, for example, when the camera's object detection processing erroneously detects an object that is not a face and recognizes it as a face, resulting in the camera focusing on the non-face object. Another example is when, in a horse racing scene, the user wants to focus on the horse's face, but the camera focuses on the jockey's face.

[0004] As a method for avoiding focusing on an object other than the user's intention, Patent Document 1 allows the user to designate a specific subject detected in a live image displayed on the camera display as a non-priority subject, thereby controlling the camera not to focus on the non-priority subject. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-157675 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, if the non-priority subject can be tracked even after the frame in which it was specified, focusing on the non-priority subject is not performed. On the other hand, if the non-priority subject goes out of frame and the camera can no longer track it, it is necessary to re-specify the non-priority subject.

[0007] At a shooting site, a single subject may be photographed multiple times to obtain a good photographed image. In such cases, the photographed image may need to be checked after each photograph. In this case, in Patent Document 1, it is necessary to specify a non-priority subject that is different from the user's intention for each photograph, which may make the operation during photographing cumbersome.

[0008] Therefore, an object of the present invention is to more efficiently exclude subjects that are not intended by the user from detection targets. [Means for solving the problem]

[0009] In order to achieve the object of the present invention, an image processing device according to one embodiment of the present invention comprises the following configuration: detection means for detecting a subject in an image, display control means for displaying the image and a detection area of ​​the subject on display means, designation means for designating the detection area displayed by the display control means as a non-detection area, storage means for storing information on the non-detection area designated by the designation means, and determination means for determining a non-detection area in another image to be displayed on the display means after the image, based on the information on the non-detection area. [Effects of the Invention]

[0010] According to the present invention, subjects that are not intended by the user can be more efficiently excluded from detection targets. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus according to a first embodiment. [Figure 2] 4 is a flowchart showing processing of the image processing apparatus according to the first embodiment. [Figure 3] FIG. 2 is a view showing an example of an input image according to the first embodiment. [Figure 4] 10A and 10B are diagrams showing detection areas according to Modifications 1 and 2 of the first embodiment. [Figure 5] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing apparatus according to a second embodiment. [Figure 6] 10 is a flowchart showing processing by an image processing apparatus according to a second embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing apparatus according to a third embodiment. [Figure 8] 10 is a flowchart showing processing of an image processing apparatus according to a third embodiment. [Figure 9] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing apparatus according to a fourth embodiment. [Figure 10] 10 is a flowchart showing processing of an image processing apparatus according to a fourth embodiment. [Figure 11] FIG. 13 is a view showing an example of an input image according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] First Embodiment In the first embodiment, the image processing device is a digital camera. In the present embodiment, an example will be described in which AF focusing is not performed on a non-detection area of ​​a next input image (another image) that is determined based on the image feature amount of the non-detection area specified in the first input image. In the present embodiment, the image processing device is a digital camera, but this is not limited to this. Furthermore, the processing that is not performed on a non-detection target in the next input image is not limited to AF processing, and may be auto exposure (AE) processing that controls exposure to a proper level and auto white balance (AWB) processing that performs color correction for a light source.

[0014] <Device configuration> FIG. 1 is a diagram illustrating an example of the functional configuration of an image processing apparatus according to the first embodiment.

[0015] The image processing device 100 has a function of detecting a person's face as a specific subject from captured image data and focusing on the detected person's face.

[0016] Light rays representing a subject image are collected by an imaging lens 101, and the light rays are incident on an imaging element 102 made up of an image sensor such as a CCD or CMOS.

[0017] The image sensor 102 outputs an electrical signal corresponding to the intensity of the incident light for each pixel. This electrical signal is a video signal. The video signal output from the image sensor 102 is processed by an analog signal processor 103 using analog signal processing such as correlated double sampling (CDS).

[0018] The video signal output from the analog signal processing unit 103 is converted into a digital data format by the A / D conversion unit 104 and input to the shooting control unit 105 and the image control unit 106 .

[0019] The image control unit 106 performs image processing such as gamma correction and white balance processing. In addition to normal image processing, the image control unit 106 performs image control using detection area information in an image supplied from the subject detection unit 109, which will be described later. The image control unit 106 is also a display control means that superimposes a square detection frame (hereinafter referred to as a bounding box) on a subject (for example, a person's face) detected by the subject detection unit 109.

[0020] The image control unit 106 determines whether to display or hide a bounding box for the object based on the reliability of the object detected by the object detection unit 109. For example, if the reliability of the object exceeds a predetermined threshold, the image control unit 106 superimposes and displays a bounding box on the object. When superimposing and displaying a bounding box on the object, the image control unit 106 determines the display position and size of the bounding box based on the position and size of the object detected by the object detection unit 109. Furthermore, when the user specifies a non-detection area using the area specification unit 110 (described below), the image control unit 106 can explicitly notify the user that a non-detection area has been specified by changing the color, thickness, line type, etc. of the bounding box. The video signal output from the image control unit 106 is sent to the image display unit 107.

[0021] The image display unit 107 is a display means such as an LCD or an organic EL display, and displays a video signal. By sequentially displaying chronologically consecutive captured images on the image display unit 107, the image display unit 107 can function as an electronic viewfinder (EVF).

[0022] The video signal is recorded on a recording medium 108 (for example, a removable memory card). The video signal may be recorded in the camera's built-in memory or in an external device communicably connected to the camera. The video signal output from the image control unit 106 is supplied to a subject detection unit 109.

[0023] The subject detection unit 109 is a detection means for detecting specific subjects in an input image, and specifies the number and area of ​​the subjects. The specific subject is, for example, a human face. The method for detecting the specific subject is a known face detection method. Known face detection techniques include a method that uses knowledge about the face (skin color information, features such as the eyes, nose, and mouth) and a method that configures a detector for face detection using a learning algorithm, such as a convolutional neural network (CNN). To improve the accuracy of face detection, it is common to perform face detection by combining these methods.

[0024] The output information of the subject detection unit 109 includes, for example, image features, position, size, tilt, reliability of the detection result, and detection category of the subject for the number of detections. The detection category is information indicating which detection task (human detection task, animal detection task, vehicle detection task, etc.) the detector used to detect the subject. The image features include, for example, color information, brightness information, and CNN features of the detection area. In this embodiment, a human detection task is described as an example, but any combination of human detection task, animal detection task, and vehicle detection task may also be used. In this case, the detection category of the detection area can be recognized based on the reliability output by the detector. Furthermore, when multiple objects in an image are captured in different detection categories, the subject detection unit 109 can also determine which detection category the object detected in is to be the main subject based on the priority of the detection tasks.

[0025] Area designation unit 110 is an input interface such as a touch panel, a button, or a joystick. The touch panel may be integrated with the screen of image display unit 107. A user can designate a non-detection area from the detection area of ​​the subject detected by subject detection unit 109 via area designation unit 110 (designation means).

[0026] For example, a bounding box is displayed superimposed on the face of a person detected by subject detection unit 109 on image display unit 107. The user can designate the subject within the bounding box as an undetected subject by pressing inside the bounding box displayed on the touch panel. Alternatively, the user may designate an undetected subject by operating a joystick or button, but this is not limiting.

[0027] The subject detection unit 109 stores image features within a detection area (i.e., a bounding box) that includes the position on the image specified by the user using the area specification unit 110 in a detection area information storage unit 111 (storage means).

[0028] The photographing control unit 105 is a photographing control unit that controls a focus control mechanism and an exposure control mechanism (not shown) of the imaging lens 101 (imaging unit) based on the video signal output from the A / D conversion unit 104. The photographing control unit 105 uses information on the detection area supplied from the subject detection unit 109 to control the focus control mechanism and the exposure control mechanism. The photographing control unit 105 controls the imaging lens 101 (imaging unit) so that it does not focus on a non-detection area.

[0029] The feature amount comparison unit 112 compares the image feature amount of the detection area detected by the object detection unit 109 with the image feature amount of the non-detection area stored in the detection area information storage unit 111, from the frame (image) onwards in which the user has designated a non-detection object. Here, the object detection unit 109 is an acquisition unit that acquires the image feature amount of the detection area detected by the object detection unit 109 and the image feature amount of the non-detection area stored in the detection area information storage unit 111. In addition, the feature amount comparison unit 112 is a determination unit that determines, as the non-detection area, a detection area having an image feature amount that is highly correlated with the image feature amount of the non-detection area.

[0030] The subjects in the determined non-detection area are excluded from the subjects detected by the subject detection unit 109, so that the display of bounding boxes and focusing on non-detected subjects are not performed. This state in which the display of bounding boxes and focusing on non-detected subjects is not performed continues from the next frame (next image) onwards.

[0031] At this time, if the user presses a predetermined button on the area designation unit 110 while the non-detection area designation is continuing, the setting to continue the non-detection area designation may be canceled and the non-detection area may be set to a state in which it can be detected again, i.e., set to a detection area. Furthermore, even if the power of the image processing device 100 (digital camera) is turned off, the detection area information storage unit 111 may be able to store the image feature amount of the non-detection area, so that the designation of the non-detection area may be maintained permanently.

[0032] <Processing flow> Next, the processing flow in this embodiment will be described with reference to Figures 2 and 3. Figure 2 is a flowchart showing processing of the image processing device according to the first embodiment. Note that the processing in Figure 2 is realized when, after the image processing device 100 (digital camera) is powered on, the CPU of the image processing device 100 loads a control program stored in the ROM into a work area in the RAM and executes it.

[0033] The photographing control unit 105 reads a photographed image as an input image 300 (S201).

[0034] Here, Fig. 3 is a diagram showing an example of an input image according to the first embodiment. Fig. 3(a) shows an input image 300 depicting a kitchen. Because the subject detection unit 109, which will be described later, erroneously detected part of the kitchen equipment as a person's face, a detection area 301 (illustrated as a bounding box) is displayed on part of the kitchen equipment. Note that although Fig. 3(a) shows one detection area 301, multiple detection areas 301 may also exist. Returning to the explanation of Fig. 2.

[0035] The subject detection unit 109 obtains a detection result including a detection area 301 in the input image 300 (S202). The detection result includes the number of detection areas, and the image feature amount, position, size, and reliability of each detection area. The image display unit 107 notifies the user of the detection result by displaying the detection area 301 in the input image 300, as shown in FIG. 3(a).

[0036] Here, the user checks the detection area 301 of the input image 300 and realizes that the detection area 301 does not include the subject intended by the user (i.e., a person's face). The user does not want the detection area 301 (part of the kitchen as a non-detection subject) to be focused on. In this specification, a non-detection subject is a subject that the user does not want the image processing device 100 to detect. Also, a non-detection area is an area that surrounds a non-detection subject with a bounding box. Therefore, when the user specifies a specific detection area 301 from the detection areas of the input image 300 using the area specification unit 110 (YES in S203), the image control unit 106 changes the detection area 301 to a non-detection area. Hereinafter, the detection area 301 refers to a non-detection area.

[0037] The detection area information storage unit 111 stores the image feature amount of the non-detection area designated by the user (S204).

[0038] If the user has not specified a non-detection region (NO in S203), the subject detection unit 109 determines whether there is an image feature of the non-detection region stored in the detection region information storage unit 111 (S205).

[0039] If there is no image feature amount of a non-detection region stored in the detection region information storage unit 111 (NO in S205), the subject detection unit 109 advances the process to S211.

[0040] If there is an image feature of a non-detection area stored in the detection area information storage unit 111 (YES in S205) and after processing in S204, the subject detection unit 109 determines whether the user has designated the non-detection area to be cancelled (S206). Here, the designation to cancel the non-detection area refers to a designation to cancel the designation of the non-detection area, and more specifically, a designation to change the non-detection area to a detection area. The user can designate the non-detection area to be cancelled, for example, by pressing any button on the area designation unit 110 (cancellation means).

[0041] If the user has designated cancellation of the non-detection region (YES in S206), the subject detection unit 109 deletes the image feature amount of the non-detection region from the detection region information storage unit 111 (S207), and the process proceeds to S208.

[0042] If the user has not designated cancellation of the non-detection region (NO in S206), the subject detection unit 109 advances the process to S208.

[0043] The imaging control unit 105 determines whether or not a non-detection area has been designated by the user (S208).

[0044] If the user has not designated a non-detection region (NO in S208), the shooting control unit 105 determines the focus point (S211). Here, the subject closest to the image processing device 100, the subject occupying a large area in the input image 300, or the subject present in a position close to the center of the input image 300 is set as the main subject (i.e., the focus point). On the other hand, if there is no subject in the input image 300, the center of the input image 300 or an object positioned close to the image processing device 100 is set as the focus point. For example, FIG. 3(b) shows the center 302 of the input image 300 as the focus point.

[0045] The image processing device 100 focuses on the focusing point (for example, the center 302 in FIG. 3(b)) determined by the shooting control unit 105 in S211 (S212). After the image processing device 100 has completed focusing, the shooting control unit 105 reads the next input image 300. Now, we return to the description of the case where the determination in S208 is YES.

[0046] If a non-detection area is specified by the user (YES in S208), the feature comparison unit 112 compares the detection area of ​​the input image 300 with the image features of the non-detection area (here, the detection area 301) stored in the detection area information storage unit 111 (S209).

[0047] If there is a detected region (not shown) that has a high correlation with the image feature of the non-detected region (YES in S209), the feature comparison unit 112 determines the detected region as a non-detected region (S210), and proceeds to S211.

[0048] If there is no detected region (not shown) that has a high correlation with the image feature of the non-detected region (NO in S209), the feature comparison unit 112 advances the process to S211.

[0049] 2, steps S203 to S207 are the process of designating a non-detection area and the process of canceling the non-detection area. Steps S208 to S212 are the process of determining a detection area on the next input image that has a high correlation with the non-detection area as the non-detection area. The order of these steps may be reversed.

[0050] According to the first embodiment, it is possible to prevent detection of a subject that is different from the user's intention and to prevent focusing on this subject.

[0051] <Variation 1> Modification 1 of the first embodiment will be described. Only the differences from the first embodiment will be described using Fig. 1 and Fig. 4. Fig. 4 is a diagram showing detection areas according to Modifications 1 and 2. Fig. 4 shows an input image 400 depicting a kitchen, similar to input image 300 in Fig. 3.

[0052] In the first modification, the subject detection unit 109 is an acquisition unit that acquires image features of a detection area 402 including a detection area 401 of a non-detection subject (part of the kitchen) designated by the user and the surrounding area of ​​the detection area 401. Note that the detection area 401 and the detection area 402 are designated as "non-detection areas" by the user as non-detection areas. On the other hand, if the detection area 401 and the detection area 402 are not designated as non-detection areas by the user, they remain detection areas. As shown in FIG. 4, the surrounding area of ​​the detection area 401 is the entire surrounding area of ​​the detection area 401 (shown as the shaded area), but it may also be a portion of the surrounding area of ​​the detection area 401. The subject detection unit 109 acquires image features of the detection area 402, which has a larger area than the detection area 301 in FIG. 3(a). As a result, only detection areas having image features highly correlated with the image features of the detection area 402 are determined as non-detection areas. If the face of the person the user wants to detect is outside the peripheral area of ​​the detection area 401 in FIG. 4, the accuracy of detecting the face of the person will not be impaired.

[0053] According to the first modification, it is possible to prevent detection of a subject other than the user's intention and to prevent focusing on this subject, without impairing the detection accuracy of the subject that the user wants to detect.

[0054] <Variation 2> Modification 2 of the first embodiment will be described. Modification 2 is an example in which the detection region of Modification 1 is modified, so only the differences from Modification 1 will be described. As with Modification 1, the description will be made using FIG. 4.

[0055] In Modification 1, subject detection unit 109 acquires image feature amounts of detection area 402 (= detection area 401 + peripheral area of ​​detection area 401) which is larger than detection area 401 of the non-detected subject in Fig. 4. In Modification 2, subject detection unit 109 acquires image feature amounts of the shaded area between detection area 401 and detection area 402 which is larger than detection area 401. In other words, subject detection unit 109 acquires image feature amounts of only the peripheral area of ​​detection area 401. In this way, as with Modification 1, if the face of the person to be detected is located outside the peripheral area of ​​detection area 401 in Fig. 4, the accuracy of detecting the person's face can be maintained.

[0056] In this way, according to the second modification, it is possible to prevent detection of a subject that is different from the user's intention and to prevent focusing on this subject, without impairing the detection accuracy of the subject that is originally intended to be detected.

[0057] <Variation 3> Modification 3 of the first embodiment will be described. Modification 3 describes an example of control in which image feature amounts of a non-detection area (for example, detection area 401 in FIG. 4) designated by the user are learned, thereby preventing detection of an undetected subject in the next input image. Note that the image processing device 100 is assumed to include a learning unit (learning means) (not shown) that causes the detector to learn image feature amounts of an undetected subject.

[0058] The image feature amounts of a non-detection area designated by the user (for example, detection area 401 in FIG. 4) are continuously stored in the detection area information storage unit 111. Using the image feature amounts stored in the detection area information storage unit 111 and a learning unit (not shown), the detector of the object detection unit 109 is trained so as not to detect an undetected object from the next input image. By having the detector learn the image feature amounts of the undetected object, it is possible to prevent the detection of an object different from the user's intention and the focusing on this object from being performed.

[0059] Second Embodiment In the first embodiment, the image processing device 100 is a digital camera, and based on the image feature amounts of the non-detection area, a detection area highly correlated with the image feature amounts of the non-detection area is determined as the non-detection area. Also, an example was described in which the shooting control unit 105 controls the imaging lens 101 (imaging means) not to perform AF focusing on the determined non-detection area. In the second embodiment, similar to the first embodiment, an example is described in which the image processing device 100 is a digital camera, and based on the "reliability of the non-detection area," detection of a subject that is different from the user's intention is not performed. In the second embodiment, similar to the first embodiment, a scene is assumed in which the subject detection unit 109 erroneously detects part of a kitchen as a person's face, as shown in the input image 300 in FIG. 3(a).

[0060] <Device configuration> 5 is a diagram showing an example of the functional configuration of an image processing apparatus according to the second embodiment. The image processing apparatus 100 according to the second embodiment includes a reliability threshold setting unit 512 as a determination unit different from that according to the first embodiment.

[0061] Like the image processing device 100 of the first embodiment, the image processing device 100 has a function of detecting a human face as a specific subject from an input image and focusing on the detected human face.

[0062] The same components as those in the first embodiment are given the same names and numbers, and the explanation of the same components will be omitted to avoid duplication.

[0063] Here, the reliability of the non-detection area is one of the pieces of information output by the subject detection unit 109, and is a value indicating the likelihood of the non-detection area being a subject. For example, in the case of human face detection, it is a value that quantitatively indicates the likelihood of the detected subject being a human face. Subjects with a reliability exceeding the reliability threshold of the image processing device 100 are to be detected. The higher the threshold value, the lower the false detection rate, but also the correct detection rate. The lower the threshold value, the higher the correct detection rate, but also the false detection rate.

[0064] The subject detection unit 109 causes the detection area information storage unit 111 to store the reliability (for example, 150) of the non-detection area that includes the position on the input image that the user specified using the area specification unit 110.

[0065] The reliability threshold setting unit 512 sets a reliability threshold (for example, 170) greater than the reliability (for example, 150) held by the detection area information holding unit 111 for frames (images) in which a non-detection area is specified and thereafter. By setting a value higher than the initial reliability threshold (for example, 140) as the next reliability threshold (for example, 170), it is possible to suppress erroneous detection of a subject unintended by the user.

[0066] <Processing flow> Next, a method for specifying a non-detection area in this embodiment will be described. Fig. 6 is a flowchart showing the processing of an image processing device according to the second embodiment. The processing in Fig. 6 is realized when, after the image processing device 100 (digital camera) is powered on, the CPU of the image processing device 100 loads a control program stored in ROM into a work area in RAM and executes it. In Fig. 6, steps of the same processing as those in Fig. 2 of the first embodiment are indicated by the same step numbers. A description of the steps of the same processing will be omitted.

[0067] In S604, the image control unit 106 changes the detection area specified by the user using the area specification unit 110 to a non-detection area, and the detection area information storage unit 111 stores the reliability of the non-detection area. The reliability threshold setting unit 512 sets a value higher than the initial reliability threshold as the next reliability threshold.

[0068] In S605, the subject detection unit 109 determines whether there is reliability of the non-detection region stored in the detection region information storage unit 111, and if there is no reliability of the non-detection region (NO in S605), the process proceeds to S211.

[0069] If the subject detection unit 109 has a reliability for the non-detection region (YES in S605), the process proceeds to S206.

[0070] In S607, the subject detection unit 109 deletes the reliability of the non-detection region from the detection region information storage unit 111, resets the reliability threshold to the initial value, and proceeds to S208.

[0071] In S609, the reliability threshold setting unit 512 compares the reliability of the detection area on the input image with the reliability threshold, and if the reliability of the detection area exceeds the reliability threshold (YES in S609), the process proceeds to S211.

[0072] If the reliability of the detected region on the input image does not exceed the reliability threshold (NO in S609), the reliability threshold setting unit 512 determines the detected region as a non-detected region in S610.

[0073] According to the second embodiment, by changing the reliability threshold based on the reliability of the non-detection area, it is possible to prevent detection of a subject that is different from the user's intention and to prevent focusing on this subject.

[0074] <Third embodiment> In the third embodiment, as in the first and second embodiments, an example will be described in which the image processing device 100 is a digital camera and erroneous detection is suppressed based on the position and size of a non-detection area on an input image. As in the first and second embodiments, the third embodiment assumes a scene in which part of a kitchen is erroneously detected as a person's face in the input image 400 of Fig. 4.

[0075] <Device configuration> 7 is a diagram showing an example of the functional configuration of an image processing apparatus according to the third embodiment. The image processing apparatus 100 according to the third embodiment includes a non-detection region setting unit 712 as a determination unit different from that according to the first embodiment.

[0076] Similar to the image processing device 100 of the first embodiment, the image processing device 100 of FIG. 7 has a function of detecting a human face as a specific subject from an input image and focusing on the detected human face.

[0077] The same components as those in the first embodiment are given the same names and numbers as those in the first embodiment. The explanation of the same components will be omitted to avoid duplication.

[0078] The subject detection unit 109 causes the detection area information storage unit 111 to store the center position and size of the non-detection area that includes the position on the input image that the user has specified using the area designation unit 110 .

[0079] The non-detection area setting unit 712 can suppress false detection by setting an area having the center position and size of the non-detection area stored in the detection area information storage unit 111 as the non-detection area on the next input image after the frame (image) in which the non-detection area is specified.

[0080] <Processing flow> Next, a method for specifying a non-detection area in this embodiment will be described. Fig. 8 is a flowchart showing the processing of an image processing device according to the third embodiment. The processing in Fig. 8 is realized when, after the image processing device 100 (digital camera) is powered on, the CPU of the image processing device 100 loads a control program stored in ROM into a work area in RAM and executes it. Note that in Fig. 8, steps of the same processing as those in Fig. 2 of the first embodiment are indicated by the same step numbers. A description of the same steps of the processing will be omitted.

[0081] In S804, the image control unit 106 changes the detection area specified by the user using the area specification unit 110 to a non-detection area, and the detection area information storage unit 111 stores the center position and size of the non-detection area. The non-detection area setting unit 712 sets an area based on the center position and size of the non-detection area as a non-detection area in the input image.

[0082] In S805, the subject detection unit 109 determines whether the center position and size of the non-detection area are stored in the detection area information storage unit 111, and if the center position and size of the non-detection area are not stored (NO in S805), the process proceeds to S211. If the center position and size of the non-detection area are stored in the detection area information storage unit 111 (YES in S805), the subject detection unit 109 proceeds to S206.

[0083] In S807, the subject detection unit 109 deletes the center position and size of the non-detection area from the detection area information storage unit 111, cancels the designation of the non-detection area, and proceeds to S208.

[0084] In S809, if the center position of the detection region of the input image is included in the non-detection region (YES in S809), the non-detection region setting unit 712 determines the detection region as the non-detection region in S810. If the center position of the detection region of the input image is outside the non-detection region (NO in S809), the non-detection region setting unit 712 proceeds to S211.

[0085] In this embodiment, the criterion for determining a non-detection area is whether the center of the subject (the center of the detection area) is included in the non-detection area, but the non-detection area may be made two or three times larger in area. Furthermore, the size of the subject may also be taken into consideration, and the size difference between the subject and the non-detection area may be added to the criterion for determining a non-detection area. Alternatively, the IoU (Intersection over Union) based on the detection area and the non-detection area may be used as the criterion for determining a non-detection area, but this is not limitative.

[0086] According to the third embodiment, by setting an area based on the center position and size of the non-detection area in the input image as the non-detection area, it is possible to suppress erroneous detection of a subject that is different from the user's intention.

[0087] <Fourth embodiment> In the fourth embodiment, similarly to the first, second, and third embodiments, the image processing device 100 is a digital camera. An example will be described in which a subject (detection area) different from the user's intention is determined as a non-detection area based on the detection category of the non-detection area.

[0088] <Device configuration> 9 is a diagram showing an example of the functional configuration of an image processing apparatus according to the fourth embodiment. The image processing apparatus 100 according to the fourth embodiment includes a detection task priority setting unit 912 as a determination unit different from that according to the first embodiment.

[0089] The image processing device 100 in FIG. 9 has a function of detecting any one of a person, an animal, and a vehicle as a specific subject from an input image, and focusing on the detected subject.

[0090] The same components as those in the first embodiment are given the same names and numbers as those in the first embodiment. The explanation of the same components will be omitted to avoid duplication.

[0091] The subject detection unit 109 causes the detection area information storage unit 111 to store the detection category of the non-detection area that includes the position on the input image that the user has specified using the area designation unit 110 .

[0092] The detection task priority setting unit 912 (setting means) lowers the priority of the detection task corresponding to the detection category held in the detection area information holding unit 111 compared to other detection tasks from the frame (image) in which the non-detection area is specified onwards. Alternatively, the detection task priority setting unit 912 suppresses the detection of the non-detection area by excluding the detection task corresponding to the detection category of the non-detection area from the detection tasks executed by the subject detection unit 109.

[0093] Here, Fig. 11 is a diagram showing an example of an input image according to the fourth embodiment. Fig. 11(a) is an input image capturing a horse riding scene. The user wants to detect the face of a horse, but the subject detection unit 109 has detected a person's face in the input image 1100. A detection area 1101 is displayed on the person's face. Here, the detection task priority setting unit 912 performs a setting to change the priority of the detection task, so that the subject intended by the user, i.e., the horse's face (detection area 1102), can be detected, as shown in Fig. 11(b).

[0094] <Processing flow> Next, a method for specifying a non-detection region in this embodiment will be described. Fig. 10 is a flowchart showing the processing of the image processing apparatus according to the fourth embodiment. The processing in FIG. 10 is realized by the CPU of the image processing device 100 expanding the control program stored in the ROM into the work area of ​​the RAM and executing it after the image processing device 100 (digital camera) is powered on.

[0095] The same process steps as those in Fig. 2 of the first embodiment are designated by the same step numbers, and the explanation of the same process steps will be omitted.

[0096] In S1004, the image control unit 106 changes the detection area specified by the user using the area specification unit 110 to a non-detection area, and the detection area information storage unit 111 stores the center position and size of the non-detection area. The detection task priority setting unit 912 changes the priority setting of the detection tasks. That is, if the current detection tasks have the following priorities in descending order: human detection task, animal detection task, and vehicle detection task, the detection task priority setting unit 912 lowers the priority of the human detection task. As a result, the detection tasks have the following priorities in descending order: animal detection task, vehicle detection task, and human detection task. Alternatively, the detection task priority setting unit 912 excludes the human detection task from the set of detection tasks (including the animal detection task, vehicle detection task, and human detection task) executed by the subject detection unit 109.

[0097] In S1005, the subject detection unit 109 determines whether there is a detection category for the non-detection area stored in the detection area information storage unit 111, and if there is no detection category for the non-detection area (NO in S1005), the process proceeds to S211. If there is a detection category for the non-detection area stored in the detection area information storage unit 111 (YES in S1005), the subject detection unit 109 proceeds to S206.

[0098] In S1007, the subject detection unit 109 deletes the detection category of the non-detection area from the detection area information storage unit 111, resets the priority of the detection task to the original, and proceeds to S208.

[0099] If the detection category of the detection area of ​​the input image is the same as the detection category of the non-detection area in S1009 (YES in S1009), the detection task priority setting unit 912 determines the detection area of ​​the input image as a non-detection area in S1010.

[0100] If the detection category of the detection area of ​​the input image is not the same as the detection category of the non-detection area (NO in S1009), the detection task priority setting unit 912 advances the process to S211.

[0101] According to the fourth embodiment, by lowering the priority of the detection task corresponding to the detection category of the non-detection area or excluding the detection task, it is possible to suppress erroneous detection of a subject that is different from the user's intention. As a result, it becomes possible to detect the subject that the user intended, and the shutter chance will not be missed.

[0102] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0103] The disclosure of this specification includes the following image processing device, imaging device, image processing method, and program. (Item 1) a detection means for detecting a subject in an image; a display control means for displaying the image and the subject detection area on a display means; a designation means for designating the detection area displayed by the display control means as a non-detection area; a storage means for storing information about the non-detection area designated by the designation means; and a determination means for determining a non-detection area in another image to be displayed on the display means after the image based on information about the non-detection area. Image processing device. (Item 2) an acquisition means for acquiring image feature amounts of the non-detection area and image feature amounts of a detection area of ​​a subject detected in the other image; the determining means determines, as the non-detection area, a detection area of ​​the subject detected in the other image that has a high correlation with the non-detection area, based on image feature amounts of the non-detection area and image feature amounts of a detection area of ​​the subject detected in the other image. Item 1. An image processing device according to item 1. (Item 3) the acquisition means acquires an image feature amount of a non-detection area that is wider than the non-detection area. Item 3. The image processing device according to item 2. (Item 4) The wide non-detection region is a region including the non-detection region and a peripheral region of the non-detection region. Item 3. The image processing device according to item 3. (Item 5) the acquiring means acquires image feature amounts of a peripheral area of ​​the non-detection area; 5. The image processing device according to any one of items 2 to 4. (Item 6) a learning means for learning a detector that does not detect a subject in the non-detection area based on an image feature amount of the non-detection area; The detecting means comprises a trained detector. 6. The image processing device according to any one of items 2 to 5. (Item 7) the determining means determines a detection region of the subject detected in the other image as the non-detection region based on a reliability threshold determined using the reliability of the non-detection region; 7. The image processing device according to any one of items 1 to 6. (Item 8) the determining means determines a detection area of ​​the subject detected in the other image as the non-detection area by setting an area in the other image based on a center position and a size of the non-detection area on the image. 8. The image processing device according to any one of items 1 to 7. (Item 9) a setting unit for setting a priority of a detection task executed by the detection unit for the non-detection area to be lower than that of other detection tasks; the determining means determines a detection area of ​​the subject detected in the other image as the non-detection area based on the setting; 9. The image processing device according to any one of items 1 to 8. (Item 10) the setting means performs a setting to exclude a detection task of the detection means executed on the non-detection area from detection tasks executed by the detection means. Item 10. The image processing device according to item 9. (Item 11) a canceling means for canceling the designation of the non-detection area; 11. The image processing device according to any one of items 1 to 10. (Item 12) An image processing device according to any one of items 1 to 11, An imaging means; and an imaging control means for controlling the imaging means so that the imaging means does not focus on the non-detection area. Imaging device. (Item 13) An image processing method executed by an image processing device, a detection step of detecting an object in the image; a display control step of displaying the image and the subject detection area on a display unit; a designation step of designating the detection area displayed by the display control step as a non-detection area; a storage step of storing information about the non-detection region designated by the designation step; and a determination step of determining a non-detection area in another image to be displayed on the display unit after the image based on information about the non-detection area. Image processing methods. (Item 14) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0104] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0105] 100 Image processing device 101 Imaging lens 102 Image sensor 103 Analog signal processing section 104 A / D conversion section 105 Shooting control unit 106 Image control unit 107 Image display unit 108 Recording Media 109 Subject detection unit 110 Area specification part 111 Detection area information storage unit 112 Feature comparison unit

Claims

1. a detection means for detecting a subject in an image; a display control means for displaying the image and the subject detection area on a display means; a designation means for designating the detection area displayed by the display control means as a non-detection area; a storage means for storing information about the non-detection area designated by the designation means; and a determination means for determining a non-detection area in another image to be displayed on the display means after the image based on information about the non-detection area. Image processing device.

2. an acquisition means for acquiring image feature amounts of the non-detection area and image feature amounts of a detection area of ​​a subject detected in the other image; the determining means determines, as the non-detection area, a detection area of ​​the subject detected in the other image that has a high correlation with the non-detection area, based on an image feature amount of the non-detection area and an image feature amount of a detection area of ​​the subject detected in the other image. The image processing device according to claim 1 .

3. the acquisition means acquires an image feature amount of a non-detection area that is wider than the non-detection area. The image processing device according to claim 2 .

4. The wide non-detection region is a region including the non-detection region and a peripheral region of the non-detection region. The image processing device according to claim 3 .

5. the acquiring means acquires image feature amounts of a peripheral area of ​​the non-detection area; The image processing device according to claim 2 .

6. a learning means for learning a detector that does not detect a subject in the non-detection area based on an image feature amount of the non-detection area; The detecting means comprises a trained detector. The image processing device according to claim 2 .

7. the determining means determines the detection region of the subject detected in the other image as the non-detection region based on a reliability threshold determined using the reliability of the non-detection region; The image processing device according to claim 1 .

8. the determining means determines, in the other image, a region based on a center position and a size of the non-detection region on the image, so as to determine, as the non-detection region, a detection region of the subject detected in the other image. The image processing device according to claim 1 .

9. a setting unit for setting a priority of a detection task executed by the detection unit for the non-detection area to be lower than that of other detection tasks; the determining means determines a detection area of ​​the subject detected in the other image as the non-detection area based on the setting; The image processing device according to claim 1 .

10. the setting means performs a setting to exclude a detection task of the detection means executed on the non-detection area from detection tasks executed by the detection means. The image processing device according to claim 9 .

11. a canceling means for canceling the designation of the non-detection area; The image processing device according to claim 1 .

12. An image processing device according to any one of claims 1 to 11; An imaging means; and an imaging control means for controlling the imaging means so that the imaging means does not focus on the non-detection area. Imaging device.

13. An image processing method executed by an image processing device, a detection step of detecting an object in the image; a display control step of displaying the image and the subject detection area on a display unit; a designation step of designating the detection area displayed by the display control step as a non-detection area; a storage step of storing information about the non-detection region designated by the designation step; and a determination step of determining a non-detection region in another image to be displayed on the display unit after the image based on information about the non-detection region. Image processing methods.

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images