Image processing apparatus, image processing method, and storage medium

Abstract

To provide a technique capable of evaluating a type of a detected subject when different detection results are obtained for a region of the subject in an image.SOLUTION: An image processing device according to the present disclosure includes image acquisition means for acquiring an image, detection means capable of detecting a plurality of types of subjects included in the image, evaluation value acquisition means for acquiring an evaluation value indicating a probability of a type of a subject with respect to a region of the detected subject, and evaluation means for evaluating the type of the subject with respect to the region of the subject detected by the detection means. The evaluation unit acquires an evaluation value using the evaluation value acquisition unit and evaluates the type of the first object based on the evaluation value when a region of an object of a different type overlaps with the detected region of the first object.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] A subject detection technology that automatically detects a specific subject pattern from an image has been known (Patent Document 1). Patent Document 1 discloses an imaging device that detects an area corresponding to a specific subject pattern, such as a human face, from a captured image and optimizes focus and exposure for the detected area. Also known is a technology that uses a technique called deep learning to train a model consisting of a neural network to learn the characteristics of a subject (e.g., a dog) in an image, and after learning, recognizes a subject (e.g., a dog) included in an image different from the image used for learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-318554 Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting subjects from a captured image, it is possible to detect various types of subjects. However, when various types of subjects are detected, there may be multiple detection results of different types in the same area. In such a case, all but one of the multiple detection results may be a false detection.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can evaluate the type of a detected subject when different detection results are obtained for the subject area within an image. [Means for solving the problem]

[0006] In order to solve this problem, for example, an image processing device of the present invention has the following configuration: That is, it has an image acquisition means for acquiring an image, a detection means capable of detecting multiple types of subjects included in the image, an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood of the type of subject for an area of ​​the detected subject, and an evaluation means for evaluating the type of subject for the area of ​​the subject detected by the detection means, wherein when an area of ​​a different type of subject overlaps with the area of ​​a detected first subject, the evaluation means acquires the evaluation value using the evaluation value acquisition means and evaluates the type of the first subject based on the evaluation value. [Effects of the Invention]

[0007] According to the present invention, when different detection results are obtained for regions of the object in an image, it is possible to evaluate the type of the detected object. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 2] 1 is a flowchart showing a series of operations in subject detection processing according to the first embodiment. [Figure 3] 10 is a flowchart showing a series of operations in subject detection processing in the second embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a crop region to be input to a type estimation CNN according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating control according to subject priority setting in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment 1) Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following description, a digital camera capable of detecting a subject in an image is used as an example of an image processing device. However, this embodiment is not limited to digital cameras and can be applied to other devices capable of detecting a subject in an image. These devices may include, for example, smartphones, game consoles, tablet terminals, medical devices, and devices for monitoring systems.

[0011] <Configuration of image processing device> The configuration of an image processing device in this embodiment will be described with reference to Fig. 1. Fig. 1 shows an example of the functional configuration of an image processing device 100. The image processing device 100 is, for example, a digital still camera or video camera that can capture an image of a subject and record moving or still image data on various media (for example, tape, optical disk, magnetic disk, solid-state memory, etc.). Each unit within the image processing device 100 is connected via a bus 160. Each unit is controlled by a CPU 151 (central processing unit) described below.

[0012] The lens unit 101 includes, for example, a fixed first-group lens 102, a zoom lens 111, an aperture 103, a fixed third-group lens 121, and a focus lens 131. The lens unit 101 also includes a zoom motor (ZM) 112, an aperture motor (AM) 104, and a focus motor (FM) 132. In accordance with commands from the CPU 151, the aperture control unit 105 drives the aperture 103 via the aperture motor 104 to adjust the aperture diameter of the aperture 103 and thereby adjust the amount of light during shooting. The zoom control unit 113 changes the focal length by driving the zoom lens 111 via the zoom motor 112. The focus control unit 133 determines the drive amount for driving the focus motor 132 based on the amount of deviation in the focus direction (defocus amount) of the lens unit 101. In addition, the focus control unit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132. AF (autofocus) control is realized by controlling the movement of focus lens 131 by focus control unit 133 and focus motor 132. Focus lens 131 is a lens for adjusting focus, and although it is simply shown as a single lens in FIG. 1, it is usually made up of multiple lenses.

[0013] The subject image formed on the image sensor 141 via the lens unit 101 is converted into an electrical signal by the image sensor 141. The image sensor 141 includes a photoelectric conversion element that performs photoelectric conversion of the subject image (optical image) into an electrical signal. The image sensor 141 has light receiving elements arranged with m pixels in the horizontal direction and n pixels in the vertical direction. The image formed on the image sensor 141 and photoelectrically converted is arranged as an image signal (image data) by the image signal processing unit 142, and an image on the imaging surface can be acquired.

[0014] Image data output from the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in a RAM (random access memory) 154. The image data stored in the RAM 154 is compressed by an image compression / decompression unit 153 and then recorded on an image recording medium 157, which is, for example, a non-volatile memory. Concurrently, the image data stored in the RAM 154 is sent to the image processing unit 152. The image processing unit 152 processes the image signal, performs processes such as reducing or enlarging the image data to an optimal size and calculating the similarity between image data. The image data processed to the optimal size can be appropriately sent to the monitor display 150 for display, thereby enabling preview image display and through-image display. The object detection results of the object detection unit 162 can also be superimposed on the image data. Furthermore, the RAM 154 functions as a ring buffer, buffering multiple image data captured within a predetermined period, the detection results of the object detection unit 162 corresponding to each image data, and the position and orientation changes of the image processing device acquired by the position and orientation change acquisition unit 161.

[0015] The operation switch 156 is an input interface including, for example, a touch panel, buttons, etc. The user can perform various operations on the image processing device 100 by selecting and operating various function icons displayed on the monitor display 150, etc.

[0016] The CPU 151 includes one or more processors. The CPU 151 controls each unit of the image processing device 100 by executing programs stored in the flash memory 155 or the RAM 154. The CPU 151 determines the accumulation time of the image sensor 141 and the setting value of the gain when outputting from the image sensor 141 to the image signal processing unit 142, based on, for example, a user instruction from the operation switch 156 or the magnitude of the pixel signal of the image data accumulated in the RAM 154. The imaging control unit 143 receives instructions on the accumulation time and the gain setting value from the CPU 151 and controls the image sensor 141.

[0017] The object detection unit 162 determines an area where a predetermined object exists using image data (input image). As will be described in detail later, the object detection unit 162 may include, for example, an image acquisition function for acquiring an input image, a detection function capable of detecting an object included in the input image, and an evaluation value acquisition function for acquiring an evaluation value of the type of object for the area of ​​the detected object. In this embodiment, a machine learning model such as a convolutional neural network (CNN) that is trained in advance using training data can be used for detecting the object and acquiring the evaluation value of the type of object. CNN is known as a typical neural network model used in deep learning. Note that, although this embodiment will be described taking an example in which a CNN is used as a detector, other machine learning models (for example, a model such as a transformer that includes an attention mechanism) may also be used.

[0018] The focus control unit 133 can realize AF control for the subject area obtained by the object detection unit 162. Furthermore, the aperture control unit 105 can perform exposure control using the brightness value of the subject area obtained by the object detection unit 162. The image processing unit 152 performs gamma correction, white balance processing, and the like based on the subject area obtained by the object detection unit 162.

[0019] The monitor display 150 displays the image output from the image processing unit 152, the object detection results (for example, rectangles) from the object detection unit 162, and the like.

[0020] In the description of this embodiment, the object detection unit 162 and the CPU 151 are separate units, but the operation of the object detection unit 162 may be performed by the CPU 151 executing a program stored in the flash memory 155. Alternatively, the object detection unit 162 may include a predetermined processing circuit, and the operation of the object detection unit 162 may be realized by the processing circuit executing a program.

[0021] The battery 159 supplies power to each unit of the image processing device 100. The battery 159 is appropriately managed by the power management unit 158, and can provide a stable power supply to the entire image processing device 100.

[0022] The flash memory 155 includes a non-volatile memory and stores a control program required for the operation of the image processing device 100, parameters used for the operation of each unit of the image processing device 100, and the like. When the image processing device 100 is started up by a user operation (when the image processing device 100 transitions from a power-off state to a power-on state), the control program and parameters stored in the flash memory 155 are loaded into a part of the RAM 154. The CPU 151 controls the operation of the image processing device 100 in accordance with the control program and parameters loaded into the RAM 154. The control program may include a program for operating a machine learning model of the object detection unit 162. The parameters may also include trained weight parameters that constitute the machine learning model.

[0023] The position and orientation change acquisition unit 161 includes a position and orientation sensor such as a gyro, acceleration sensor, or electronic compass, and measures a change in the position and orientation of the image processing device 100 relative to the captured scene. The acquired position and orientation changes are stored in the RAM 154.

[0024] The defocus calculation unit 163 calculates the defocus amount from the image processing device for any object in the image. The generated defocus information is stored in the RAM 154 and is referenced by the image processing unit 152.

[0025] <Example of machine learning model in object detection unit 162> The object detection unit 162 in this embodiment uses a machine learning model (e.g., CNN) as a detector that detects objects in an image. The detector in this embodiment outputs, for example, a rectangular area on the image that corresponds to the area of ​​the object and a confidence level for the detection result. The confidence level for the detection result is output as an integer value from 0 to 255, and for example, the larger the value, the more likely the detection result is. The object detection unit 162 in this embodiment uses multiple CNNs according to the type of object that have been trained in advance using separate training data for each type of object, such as a dog-detection CNN trained using images of dogs and a bird-detection CNN trained using images of birds.

[0026] Furthermore, in this embodiment, the object detection unit 162 uses a machine learning model (e.g., a CNN) that estimates the type of object present in an input image, in addition to a CNN that detects a specific type of object. The CNN that estimates the type of object (type estimation CNN) functions as a classifier that classifies the type of object in the input image. The output of the type estimation CNN includes likelihoods corresponding to each predetermined type of object. The likelihoods corresponding to each type are, for example, integer values ​​from 0 to 255 for each type. When the type estimation CNN classifies the objects in an image into, for example, people, dogs, birds, and horses (i.e., four types), the type estimation CNN outputs a vector consisting of four integer values. For example, the type estimation CNN is trained to output a person likelihood of 255 for an image of a person and a likelihood of 0 for other types. Note that when an image containing no types of objects is input, the type estimation CNN is trained to set all four output values ​​to 0. During inference, the object detection unit 162 uses the largest output likelihood corresponding to each type output from the type estimation CNN. However, if the maximum value of the output likelihood is less than a predetermined threshold, the object detection unit 162 can determine that no type of object is included in the image. In this way, the type estimation CNN outputs an evaluation value indicating the likelihood of the type of object for the area of ​​the detected object. By using the evaluation value from the type estimation CNN, the detection results of the CNN (i.e., the detector) according to the type of object can be evaluated. In this embodiment, when the detection results of multiple CNNs according to the type of object overlap, the type of object based on the CNN that detected a specific object, such as the main subject, is compared (evaluated) with the type of object based on the evaluation value from the type estimation CNN. In this way, it is possible to evaluate detection results that may include erroneous detections.

[0027] Furthermore, as will be described later, in this embodiment, the type of object in the area of ​​the object can be determined. For example, when the types of the comparison objects are different, the type of object in the area of ​​the detected object can be determined using the evaluation value obtained by the type estimation CNN. In other words, the appropriate type of object can be determined using the evaluation value obtained by the type estimation CNN. This will be described in more detail below.

[0028] <Subject detection process sequence> Next, a series of operations in the subject detection process in this embodiment will be described with reference to Fig. 2. The subject detection process described below is realized by CPU 151 executing a program stored in flash memory 155. The operation of object detection unit 162 may also be realized by CPU 151 executing a program stored in flash memory 155. Alternatively, the operation of object detection unit 162 may be realized by a predetermined processing circuit of object detection unit 162 executing a program.

[0029] In S200, the object detection unit 162 acquires image data (input image) supplied from the imaging control unit 143, for example, via the RAM 154. In S201, the object detection unit 162 detects a subject in the input image. As described above, the object detection unit 162 uses independent detectors prepared in advance according to the type of subject. For example, four types of CNNs, for people, dogs, birds, and horses, are prepared in a trained state, and the object detection unit 162 executes the four CNNs in parallel for each frame. The areas where a subject is detected by each CNN become the subject areas.

[0030] In S202, the CPU 151 selects, from among the multiple detected subjects, a subject corresponding to an area that is most likely to be the main subject as the main subject. The main subject is a subject that is the target of processes such as AF and frame display. If multiple subjects exist in an image, only one subject becomes the main subject. The main subject may be selected using any method. For example, the CPU 151 may select the main subject based on at least one of the size of the area, the distance from the center of the image, and the distance from the ranging area set by the user. Furthermore, the CPU 151 may preferentially select a subject of a specific type as the main subject upon detection of the specific type of subject. For example, if a subject of a person type is detected, the CPU 151 may preferentially select the person subject as the main subject. If a unique main subject cannot be determined even in this manner, the CPU 151 may preferentially select, for example, a detection result with a larger size as the main subject.

[0031] In S203, CPU 151 determines whether a subject area (another subject area) different from the selected main subject area (area of ​​the main subject) has been detected overlapping the main subject area. In this embodiment, if it is possible to determine whether the other subject area overlaps the main subject area, CPU 151 can make this determination using any method or any threshold. For example, if IOU (the ratio of the area of ​​the intersection of the main subject area and the other subject area to the area of ​​the union of the main subject area and the other subject area) is 0.3 or greater, CPU 151 can determine that the main subject area overlaps the other subject area. If CPU 151 determines that the other subject area overlaps the main subject area, it proceeds to S204; otherwise, it terminates the subject detection process. Note that, upon terminating the subject detection process in S203, CPU 151 determines the type detected in the main subject area selected in S202 as the type of the main subject, maintains the main subject area, and can perform any process, such as AF or frame display.

[0032] In S204, CPU 151 performs crop processing on each of the main subject region and the region of another subject determined to overlap with the main subject region in S203. For example, when a certain subject is detected as a "person" and a "dog" by two detectors as shown in FIG. 4(a), CPU 151 generates two cropped images: an image of the person detection region shown in FIG. 4(b) and an image of the dog detection region shown in FIG. 4(c). Note that CPU 151 may crop the subject region as is, or may crop a region obtained by scaling (enlarging or reducing) the subject region at a predetermined magnification. Alternatively, CPU 151 may crop a region adjusted vertically or horizontally to match the aspect ratio of the input image of the type estimation CNN.

[0033] In S205, CPU 151 inputs each of the cropped images to the type estimation CNN and executes the type estimation CNN. Object detection unit 162 executes inference processing by the type estimation CNN to estimate (classify) the type of subject for each of the cropped images. At this time, CPU 151 may resize the cropped images as needed to match the size of the input image for the type estimation CNN. As described above, type estimation CNN outputs a vector consisting of four integer values ​​indicating the likelihood (i.e., an evaluation value indicating the likelihood of the type of subject) of each type (for example, four types: person, dog, bird, and horse). In the following description, the likelihood corresponding to the jth type of the ith cropped image is referred to as p i j For example, in the examples shown in Figures 4(b) and 4(c), i = 0, 1 (i = 0 is the person detection region, and i = 1 is the dog detection region). If there are two subject regions determined to overlap with the main subject region in S203, then i = 0, 1, 2. In the example of this embodiment, there are four types of classifications performed by the type estimation CNN, and the index j corresponds to a person with j = 0, a dog with j = 1, a bird with j = 2, and a horse with j = 3.

[0034] For example, the species estimation CNN outputs the species estimation result (e.g., {0,255,0,0}) of the image obtained by cropping the person detection area at i=0, and then outputs the species estimation result (e.g., {0,255,0,0}) of the image obtained by cropping the dog detection area at i=1.

[0035] In S206, the CPU 151 calculates the average value of the likelihood corresponding to each CNN output for each type of object. For example, the CPU 151 calculates the average value of the likelihood of the j-th type according to equation (1).

[0036]

number

[0037] In S207, the CPU 151 selects j that gives the maximum value of equation (1), which is the average value of the likelihood of the j-th type, according to equation (2). In the example of the result of estimating the type of the person detection area for i=0 (e.g., {0,255,0,0}) and the result of estimating the type of the dog detection area for i=1 (e.g., {0,255,0,0}), the average value of the likelihood is, for example, {0,255,0,0}. In this case, j max The type indicated by j=1 (i.e., dog).

[0038]

number

[0039] In S208, the CPU 151 compares the current type of the main subject (based on the main subject selected in S202) with the j calculated by Equation 2. max The CPU 151 determines whether the type indicated by the main subject is different from the type indicated by j. max If it is determined that the type indicated by j matches the type indicated by j, the object detection process is terminated. This is because, for example, the main object selected in S202 is an object detected by the person detection CNN, and j calculated by Equation 2 matches the type indicated by j. maxis 0 (index indicating a person). In this case, it can be determined that the type of the subject (i=0) detected by the person detection CNN is likely. In this case, the CPU 151 maintains the type and area of ​​the main subject selected in S202, and can perform any processing such as AF or frame display. On the other hand, the CPU 151 can determine the type of the current main subject and the area j calculated by Equation 2. max If it is determined that the type indicated by the parameter is different (does not match), the process proceeds to S209.

[0040] In S209, the CPU 151 determines whether the type of any of the other object areas overlapping with the main object area is j max If the type indicated by j matches, the subject (area or type) is reselected as the main subject. For example, in the example shown in FIG. 4(a), it is assumed that the main subject area selected in S202 is the person detection area, and that the dog detection area is determined to overlap with the main subject area in S203. Then, j max If it is equal to 1 (dog), CPU 151 changes the main subject region to a dog detection region and also changes the type of the main subject to dog.

[0041] The CPU 151 then ends the subject detection process. Note that any type of subject area overlapping with the main subject area is max If they do not match or if the likelihood calculated by equation (1) is less than the threshold, CPU 151 ends the subject detection process without reselecting the main subject. Note that, when the subject detection process ends, CPU 151 may discard information about other subjects that overlap with the subject that ultimately became the main subject.

[0042] As described above, in the above-described embodiment, the object detection unit 162, which can detect multiple types of objects included in an acquired image, detects the area of ​​the object. Then, when the area of ​​the detected object overlaps with the area of ​​a different type of object, the type estimation CNN is used to acquire a likelihood (evaluation value) for the detected object area, and the type of the object is evaluated based on the evaluation value. In this way, even when the area of ​​the detected object overlaps with the area of ​​a different type of object, the type of the object area detected by the detector of the object detection unit 162 can be appropriately evaluated. Furthermore, an appropriate object type for the object area is determined based on the evaluation value obtained by the type estimation CNN. In this way, even when the detection results of different types of objects overlap, it is possible to obtain an appropriate type for the object area.

[0043] In the above embodiment, CPU 151 applies the type estimation CNN to each of the overlapping object regions (i=0, 1) and evaluates the type of the main object based on the likelihood (evaluation value) for each of the overlapping object regions. However, this embodiment is not limited to this example. For example, CPU 151 may apply the type estimation CNN to only one object region (e.g., the main object) and evaluate the type of the object based on the likelihood for that object region. In this case, if the type of the main object matches the type of object indicated by the likelihood by the type estimation CNN, the type of the main object can be maintained. On the other hand, if the type of the main object does not match the type of object indicated by the likelihood by the type estimation CNN, type estimation CNN is applied to each of the overlapping object regions and likelihoods for multiple cropped images are calculated to determine the type of the main object. max Alternatively, the type indicated by may be selected.

[0044] (Embodiment 2) Next, a second embodiment will be described. The second embodiment relates to a process for re-estimating the type of a main subject while tracking the main subject. Therefore, although the subject detection process is partially different from that of the first embodiment described above, the configuration of the image processing device 100 is the same. Therefore, the same or substantially the same configuration or process will be assigned the same reference numerals and their description will be omitted, and the description will focus on the differences.

[0045] <Subject detection process sequence> A series of operations in the subject detection process in the second embodiment will be described with reference to Fig. 3. Note that in this embodiment, a case is considered in which, when the main subject is determined, only one type (e.g., a person) is detected as a detection result, and another type (e.g., a dog) is detected midway. In the subject detection process in this embodiment, when the correct subject is a dog, it is possible to correct the type of the main subject from a person to a dog. A small, distant subject is detected as the wrong type and made the main subject, but as the subject gets closer (gets larger), it can be detected as the correct type.

[0046] The following subject detection processing is realized by CPU 151 executing a program stored in flash memory 155. Furthermore, the operation of object detection unit 162 may be realized by CPU 151 executing a program stored in flash memory 155. Alternatively, the operation of object detection unit 162 may be realized by a predetermined processing circuit of object detection unit 162 executing a program. Below, the processing within one frame after the main subject has been determined will be described.

[0047] In the processes from S200 to S201, similarly to the first embodiment, the object detection unit 162 acquires an input image and detects a subject in the input image.

[0048] In S301, CPU 151 updates the region of the main subject. For example, CPU 151 identifies and updates the region of the main subject in the current frame by performing template matching using the main subject region of the previous frame as a template. If there is a region of the same type of subject as the main subject among the detection results obtained in S201, CPU 151 may use that region to update the region of the main subject.

[0049] In S203, CPU 151 determines whether or not an area of ​​a subject other than the selected main subject has been detected overlapping with the main subject area, as in embodiment 1. If CPU 151 determines that an area of ​​another subject overlaps with the main subject area, it proceeds to S302; otherwise, it ends the operation of the subject detection process.

[0050] In S302, CPU 151 adds (increments) 1 to the value of a counter corresponding to the type of detection result determined in S203 to overlap with the main subject area. This counter is prepared for each type, and in this embodiment, for example, there are four integer values ​​for person, dog, bird, and horse. It is assumed that all of the counters are initialized to 0 when the main subject is determined.

[0051] In S303, the CPU 151 determines whether any counter has reached a predetermined threshold value or more. In this embodiment, the threshold value is set to, for example, 10. If the CPU 151 determines that any counter has reached a predetermined threshold value or more, the process proceeds to S304; otherwise, the CPU 151 ends the operation of the subject detection process.

[0052] In S304, the CPU 151 performs cropping on each of the main subject region and the region of the detection result corresponding to the type for which the counter value is equal to or greater than the threshold value. As described below, the counters corresponding to all types are initialized to 0 each time the type estimation CNN is executed. Therefore, in S303 and S304, the CPU 151 executes evaluation using the type estimation CNN only when the counter value reaches a predetermined value. That is, when an evaluation value is obtained using the type estimation CNN, the CPU 151 controls the type estimation CNN so that evaluation values ​​are not obtained using the type estimation CNN for at least a predetermined period of time (i.e., evaluation of the type of subject is not performed for the predetermined period of time). This reduces the adverse effects of applying the type estimation CNN to all frames and improves processing speed. Furthermore, in S303 and S304, the types of subject regions to be cropped can be limited (narrowed) to those having counter values ​​that satisfy the threshold value. For example, even when detecting subjects for each of four types, if only people and dogs have counter values ​​that satisfy the threshold value, only the regions of the people and dogs may be cropped. Note that the cropping can be performed in the same manner as in the first embodiment.

[0053] Thereafter, CPU 151 performs the processes from S205 to S209 in the same manner as in embodiment 1. That is, CPU 151 changes the main subject (area or type) to another subject based on the evaluation value by the type estimation CNN, or continues the current main subject (area or type). Note that CPU 151 may initialize counters corresponding to all types to 0 every time it executes S205 (executes the type estimation CNN).

[0054] (Embodiment 3) Next, a third embodiment will be described. The above-described embodiments are based on the assumption that subjects of different types do not exist simultaneously in the same area. However, there are cases where subjects of different types actually exist in the same area, such as a scene in which a person is holding a dog. If the type is changed in the above-described subject detection process for such a scene, the main subject may be unintentionally changed to a person even when the intended main subject is a dog. In this case, since the scene actually contains both a person and a dog, it can be said that the type estimation CNN outputs a correct answer regardless of whether it outputs a higher likelihood for a person or a dog.

[0055] Therefore, in this embodiment, when there are different types of subjects, if the user sets the type of subject to be prioritized as the main subject (hereinafter referred to as subject priority setting), the subject types to be targeted for subject detection processing are limited based on the subject priority setting.

[0056] For example, as shown in Figure 5, in a scene where a person is holding a dog, there are correct detection results for the person and the dog (by the person detection CNN and the dog detection CNN), and an incorrect detection result for the bird (by the bird detection CNN).

[0057] When the subject priority setting is set to people priority, the CPU 151 determines a person as the main subject in the example of FIG. 5 without implementing the type estimation CNN. When the subject priority setting is set to animals priority, the type of the main subject is determined to be a dog or a bird in accordance with the subject detection process of embodiment 1 or embodiment 2 (the area is based on the output of the detection process corresponding to the determined type). That is, the CPU 151 performs cropping based on the dog detection result and the bird detection result, and executes the type estimation CNN on each cropped image. That is, the CPU 151 limits the type of subject for which an evaluation value is obtained using the type estimation CNN to the type of subject set in the subject priority setting. As a result, if the likelihood of either dog or bird is highest, the CPU 151 determines that type of subject as the main subject.

[0058] Furthermore, there are cases where the subject priority setting is set to something other than people, dogs, and birds (for example, vehicle priority), and people, dogs, and birds have equal priority. In this case, the CPU 151 determines the type of person, dog, or bird that has the highest likelihood output by the type estimation CNN as the type of the main subject, according to the subject detection processing of embodiment 1 or embodiment 2.

[0059] In this way, by providing the subject priority setting, even if different types of subjects actually exist in the same area, it is possible to determine the area of ​​the subject that matches the intention as the area of ​​the main subject.

[0060] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0061] (Disclosure of the present specification) The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) image acquisition means for acquiring an image; a detection means capable of detecting a plurality of types of subjects included in the image; an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood of the type of the object for the detected object area; evaluation means for evaluating the type of subject for the area of ​​the subject detected by the detection means; The image processing device is characterized in that, when an area of ​​a detected first subject overlaps with an area of ​​a subject of a different type, the evaluation means acquires the evaluation value using the evaluation value acquisition means and evaluates the type of the first subject based on the evaluation value. (Item 2) 2. The image processing device according to item 1, further comprising a determination means for determining a type of the first subject based on the evaluation value for the detected area of ​​the first subject. (Item 3) 3. The image processing device according to item 1 or 2, wherein the evaluation means evaluates the type of the first subject based on the evaluation value for the area of ​​the first subject acquired by the evaluation value acquisition means. (Item 4) 3. The image processing device according to item 1 or 2, characterized in that the evaluation means evaluates the type of the first subject based on the evaluation value for each of the overlapping subject areas acquired by the evaluation value acquisition means. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the evaluation means controls the evaluation value acquisition means so that, when the evaluation value is acquired using the evaluation value acquisition means, the evaluation value acquisition means is not used to acquire the evaluation value for a predetermined period of time. (Item 6) The camera further includes a setting unit for setting a type of subject to be preferentially designated as a main subject, 6. The image processing device according to any one of items 1 to 5, wherein the evaluation means limits the type of subject for which the evaluation value is acquired using the evaluation value acquisition means to the type of subject set by the setting means. (Item 7) 3. The image processing device according to item 2, characterized in that, when the area of ​​the detected first subject does not overlap with the area of ​​a different type of subject, the determination means determines the type of subject detected by the detection means in the area of ​​the first subject as the type of the first subject. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the evaluation means evaluates the type of the first subject by comparing the type of the first subject based on the detection means that detected the first subject with the type of the first subject based on the evaluation value. (Item 9) the evaluation means evaluates the type of the first subject by comparing the type of the first subject based on the detection means that detected the first subject with the type of the first subject based on the evaluation value; 3. The image processing device according to item 2, characterized in that, when the type of the first subject based on the detection means that detected the first subject does not match the type of the first subject based on the evaluation value, the determination means determines the type of the first subject based on the evaluation value for the area of ​​the detected first subject. (Item 10) the evaluation value acquisition means acquires the evaluation value for each predetermined type of subject for the detected subject area; 10. The image processing device according to any one of items 1 to 9, wherein the evaluation means evaluates the type of the subject of the first subject using the type of the subject with the highest evaluation value. (Item 11) Item 11. The image processing device according to item 10, wherein the evaluation value acquisition means includes a machine learning model that inputs the area of ​​the detected subject and outputs the evaluation value for the area of ​​the subject for each predetermined type of subject. (Item 12) the detection means includes a machine learning model trained to detect a particular type of object; 12. The image processing device according to any one of items 1 to 11, wherein the evaluation means evaluates a region of a subject of a specific type as the type of the first subject when the evaluation value obtained by the evaluation value obtaining means for the region of the subject of the specific type is the highest for the specific type. (Item 13) The image capturing device further includes a selection means for selecting a main subject area from the detected plurality of subject areas, 13. The image processing device according to any one of items 1 to 12, wherein the region of the first object is a region of a main object. (Item 14) 14. The image processing device according to any one of items 1 to 13, further comprising a cropping unit that crops an area in the image as an area of ​​the detected subject based on the detection result by the detection unit. (Item 15) an image acquisition step of acquiring an image; a detection step capable of detecting a plurality of types of subjects included in the image; an evaluation value acquisition step of acquiring an evaluation value indicating the likelihood of the type of the object for the detected object area; an evaluation step of evaluating the type of the object for the area of ​​the object detected in the detection step, In the evaluation step, when an area of ​​a detected first subject overlaps with an area of ​​a subject of a different type, the evaluation value acquisition step is executed to acquire the evaluation value, and the type of the first subject is evaluated based on the evaluation value. (Item 16) 15. A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 14.

[0062] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0063] 100...image processing device, 151...CPU, 162...object detection unit

Claims

1. image acquisition means for acquiring an image; a detection means capable of detecting a plurality of types of subjects included in the image; an evaluation value acquisition means for acquiring an evaluation value indicating the likelihood of the type of the object for the detected object area; evaluation means for evaluating the type of subject for the area of ​​the subject detected by the detection means; The image processing device is characterized in that, when an area of ​​a detected first subject overlaps with an area of ​​a subject of a different type, the evaluation means acquires the evaluation value using the evaluation value acquisition means and evaluates the type of the first subject based on the evaluation value.

2. 2. The image processing apparatus according to claim 1, further comprising a determination unit that determines the type of the first subject based on the evaluation value for the detected area of ​​the first subject.

3. 2 . The image processing device according to claim 1 , wherein the evaluation means evaluates the type of the first subject based on the evaluation value for the region of the first subject acquired by the evaluation value acquisition means.

4. 2. The image processing device according to claim 1, wherein the evaluation means evaluates the type of the first subject based on the evaluation value for each of the overlapping subject regions acquired by the evaluation value acquisition means.

5. 2. The image processing device according to claim 1, wherein, when the evaluation means acquires the evaluation value using the evaluation value acquisition means, the evaluation means controls the evaluation means so that the evaluation value acquisition means does not acquire the evaluation value using the evaluation value acquisition means for a predetermined period of time.

6. The camera further includes a setting unit for setting a type of subject to be preferentially designated as a main subject, 2. The image processing device according to claim 1, wherein the evaluation means limits the types of subjects for which the evaluation value is to be acquired using the evaluation value acquisition means to the types of subjects set by the setting means.

7. 3. The image processing device according to claim 2, wherein, when an area of ​​a detected first subject does not overlap with an area of ​​a different type of subject, the determining means determines the type of subject detected by the detecting means in the area of ​​the first subject as the type of the first subject.

8. 2. The image processing device according to claim 1, wherein the evaluation means evaluates the type of the first subject by comparing the type of the first subject based on the detection means that detected the first subject with the type of the first subject based on the evaluation value.

9. the evaluation means evaluates the type of the first subject by comparing the type of the first subject based on the detection means that detected the first subject with the type of the first subject based on the evaluation value; 3. The image processing device according to claim 2, wherein the determination means determines the type of the first subject based on the evaluation value for the area of ​​the detected first subject when the type of the first subject based on the detection means that detected the first subject does not match the type of the first subject based on the evaluation value.

10. the evaluation value acquisition means acquires the evaluation value for each predetermined type of subject for the detected subject area; 2. The image processing apparatus according to claim 1, wherein the evaluation means evaluates the type of the first subject using the type of the subject with the largest evaluation value.

11. 11. The image processing device according to claim 10, wherein the evaluation value acquisition means includes a machine learning model that receives input of a detected object area and outputs the evaluation value for the object area for each predetermined object type.

12. the detection means includes a machine learning model trained to detect a particular type of object; 2. The image processing device according to claim 1, wherein the evaluation means evaluates a specific type of subject as the first type of subject when the evaluation value obtained by the evaluation value obtaining means for an area of ​​the subject of the specific type is the highest for the specific type.

13. The image capturing device further includes a selection means for selecting a main subject area from the detected plurality of subject areas, 2. The image processing device according to claim 1, wherein the first object area is a main object area.

14. 2. The image processing apparatus according to claim 1, further comprising a cropping unit that crops an area in the image as the area of ​​the detected subject based on the result of detection by said detecting unit.

15. an image acquisition step of acquiring an image; a detection step capable of detecting a plurality of types of subjects included in the image; an evaluation value acquisition step of acquiring an evaluation value indicating the likelihood of the type of the object for the detected object area; an evaluation step of evaluating the type of the object for the area of ​​the object detected in the detection step, In the evaluation step, when an area of ​​a detected first subject overlaps with an area of ​​a subject of a different type, the evaluation value acquisition step is executed to acquire the evaluation value, and the type of the first subject is evaluated based on the evaluation value.

16. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 14.

Images