Image processing apparatus, control method of image processing apparatus, image capturing apparatus, and control method of image capturing apparatus

Abstract

In a case where the first setting is set, the determination unit determines a main subject area while setting a subject in which the first detection unit detects the first subject as a main subject in priority over a subject in which the first detection unit does not detect the first subject; and in a case where the second setting is set, the determination unit determines a main subject area while setting a subject in which the first detection unit does not detect the first subject and the second detection unit detects the second subject as a main subject in priority over a subject in which the first subject and the second subject are not detected.

Description

Technical Field

[0001] This invention relates to an image processing device for detecting a predefined subject and setting a main subject area for focusing and exposure adjustment, a control method for the image processing device, a camera device, and a control method for the camera device. Background Technology

[0002] Image processing methods that enable more stable detection of specific subjects from images include methods for detecting the upper body or head of humans.

[0003] Existing technical documents

[0004] Patent documents

[0005] PTL 1: Japanese Patent Publication No. 2022-51280 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] PTL 1 discloses a method for detecting and tracking the head portion of a subject, including the face, even when the subject is facing forward, using the aforementioned technique for detecting the upper body or head portion of a human, even when the subject is facing backward. In PTL 1, even when multiple humans are present, a primary subject can be appropriately selected.

[0008] However, PTL 1 did not initially anticipate the entry of undetectable human faces, such as those wearing goggles or masks, into the field of view.

[0009] The present invention was designed in view of the above-mentioned problems and aims to provide an image processing device, an image processing device control method, a camera device, and a camera device control method that can appropriately select a main subject that reflects the user's intention.

[0010] Solution for solving the problem

[0011] To address the aforementioned problems, an image processing apparatus according to the present invention includes: a first detection unit for detecting a first subject from an image; a second detection unit for detecting a second subject different from the first subject from the image; a determining unit for determining a main subject region as a main subject region based on at least one of the detection results obtained by the first detection unit and the second detection unit; and a setting unit for receiving a switch between a first setting and a second setting based on an operation from a user. When the first setting is set by the setting unit, the determining unit determines the main subject region while prioritizing subjects whose first subject is detected by the first detection unit over subjects whose first subject is not detected by the first detection unit. When the second setting is set by the setting unit, the determining unit determines the main subject region while prioritizing subjects whose first subject is not detected by the first detection unit and whose second subject is detected by the second detection unit over subjects whose first and second subjects are not detected by the second detection unit.

[0012] Advantages of the invention

[0013] According to the present invention, a main subject that better reflects the user's intent can be appropriately selected. Attached Figure Description

[0014] [ Figure 1 ] Figure 1 This is a block diagram illustrating the configuration of a camera device including an image processing apparatus according to a first embodiment.

[0015] [ Figure 2 ] Figure 2 This is a flowchart illustrating the main subject determination process according to the first embodiment.

[0016] [ Figure 3 ] Figure 3 This is a flowchart illustrating the main subject candidate determination process according to the first embodiment.

[0017] [ Figure 4 ] Figure 4 This is a flowchart illustrating the main subject locking determination process according to the first embodiment.

[0018] [ Figure 5 ] Figure 5 This is a flowchart illustrating the main subject determination process according to the first embodiment.

[0019] [ Figure 6 ] Figure 6 This is a flowchart illustrating the detection weight setting process according to the first embodiment.

[0020] [ Figure 7A ] Figure 7AThis is a diagram illustrating the priority weights related to position and size according to the first embodiment.

[0021] [ Figure 7B ] Figure 7B This is a diagram illustrating the priority weights related to position and size according to the first embodiment.

[0022] [ Figure 8A ] Figure 8A This is a diagram illustrating the detection type weights according to the second embodiment.

[0023] [ Figure 8B ] Figure 8B This is a diagram illustrating the detection type weights according to the second embodiment.

[0024] [ Figure 8C ] Figure 8C This is a diagram illustrating the detection type weights according to the second embodiment.

[0025] [ Figure 9A ] Figure 9A This is a diagram illustrating the effects according to the first embodiment.

[0026] [ Figure 9B ] Figure 9B This is a diagram illustrating the effects according to the first embodiment.

[0027] [ Figure 9C ] Figure 9C This is a diagram illustrating the effects according to the first embodiment.

[0028] [ Figure 9D ] Figure 9D This is a diagram illustrating the effects according to the first embodiment.

[0029] [ Figure 9E ] Figure 9E This is a diagram illustrating the effects according to the first embodiment.

[0030] [ Figure 9F ] Figure 9F This is a diagram illustrating the effects according to the first embodiment.

[0031] [ Figure 9G ] Figure 9G This is a diagram illustrating the effects according to the first embodiment.

[0032] [ Figure 10 ] Figure 10 This is a flowchart illustrating the main subject candidate determination process according to the second embodiment.

[0033] [ Figure 11 ] Figure 11 This is a flowchart illustrating the detection weight setting process according to the second embodiment.

[0034] [ Figure 12A ] Figure 12A This is a diagram illustrating the effects according to the second embodiment.

[0035] [ Figure 12B ] Figure 12B This is a diagram illustrating the effects according to the second embodiment.

[0036] [ Figure 12C ] Figure 12C This is a diagram illustrating the effects according to the second embodiment.

[0037] [ Figure 12D ] Figure 12D This is a diagram illustrating the effects according to the second embodiment.

[0038] [ Figure 12E ] Figure 12E This is a diagram illustrating the effects according to the second embodiment.

[0039] [ Figure 12F ] Figure 12F This is a diagram illustrating the effects according to the second embodiment.

[0040] [ Figure 13 ] Figure 13 This is a block diagram illustrating the configuration of an image processing apparatus including an autofocus device according to a second embodiment of the present invention.

[0041] [ Figure 14 ] Figure 14 This is a flowchart illustrating the main subject determination process according to the second embodiment. Detailed Implementation

[0042] The best mode for carrying out the present invention will now be described in detail with reference to the accompanying drawings.

[0043] As stated above, PTL 1 initially did not anticipate the possibility of undetectable human faces, such as those wearing goggles or masks, entering the field of view. Given this, it was considered to designate a subject whose face was never detected as the primary subject, as long as the subject's head was detected, thus allowing even undetectable human faces, such as those wearing goggles or masks, to be designated as primary subjects. However, in the aforementioned case, it is uncertain whether the subject is facing backward. Therefore, as in PTL 1, when excluding subjects whose heads are only detected from the primary subject candidates, even if the undetectable human face is facing forward, that human is also excluded from the primary subject candidates.

[0044] In view of the foregoing, the present invention is characterized by solving the above problems by allowing the user to set the subject to be set as a primary subject candidate.

[0045] <First Embodiment>

[0046] Figure 1 The illustration shows a configuration of a camera device, such as a camcorder, including an image processing apparatus according to a first embodiment. In this embodiment, a camcorder will be used as an example for explanation; however, the invention can be applied to other camera devices, such as digital still cameras.

[0047] exist Figure 1 In this embodiment, the imaging optical system for imaging light from a subject includes a first fixed lens 101, a zoom lens 102 that changes magnification by moving along the optical axis, an aperture 103, a second fixed lens 104, and a focusing lens 105. The focusing lens 105 has both the function of correcting the movement of the focal plane caused by the magnification change and the focusing function. An image sensor 106 is a component used as an image sensor and includes a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor. The image sensor 106 includes a plurality of pixel units arranged in a matrix. A light beam passing through the imaging optical system forms an image on the light-receiving surface of the image sensor 106 and is converted into a signal charge corresponding to the amount of incident light by photodiodes (photoelectric conversion units) included in each pixel unit. When capturing moving images, electrical signals for each frame are periodically output. The image sensor 106 according to this embodiment holds a plurality of photodiodes (described in this embodiment as two photodiodes) in one pixel to perform known autofocus (AF) based on phase difference of the imaging plane. By separating the light beam with a microlens and forming an image using two photodiodes (photodiodes A and B), two signals can be acquired for imaging and phase difference detection. In this embodiment, the signal obtained by adding the signals from the two photodiodes (image signal A+B) is the imaging signal, and the signals from each photodiode (image signal A, image signal B) are two image signals used for phase difference detection (AF).

[0048] Because photodiodes receive light beams passing through different regions of the exit pupil of the camera optical system, image signal B has parallax relative to image signal A. By calculating the correlation between the two image signals used for AF and performing focus detection using a phase difference detection method in the AF signal processing unit 113, which will be described below, image offset and various types of reliability information are calculated. The configuration of the image sensor supporting camera plane phase difference AF is not limited to the configuration of multiple photodiodes in a single pixel as in this embodiment. For example, multiple types of focus detection pixels (one photodiode per pixel) that receive light beams passing through different regions of the exit pupil of the camera optical system can be provided in the image sensor.

[0049] The Correlated Dual Sampling / Automatic Gain Control (CDS / AGC) circuit 107 samples the output of the image sensor 106 and adjusts the gain. The camera signal processing circuit 108 performs various types of image processing on the output signal from the CDS / AGC circuit 107 and generates a video signal. The monitor 109 includes a liquid crystal display (LCD) and displays the video signal from the camera signal processing circuit 108. The recording unit 115 records the video signal from the camera signal processing circuit 108 onto a recording medium such as magnetic tape, optical disc, or semiconductor memory.

[0050] Zoom drive source 110 moves zoom lens 102, and focus drive source 111 moves focus lens 105. Zoom drive source 110 and focus drive source 111 each include an actuator, such as a stepper motor, a DC motor, a vibration motor, and a voice coil motor.

[0051] AF gate 112 allows signals from the output signals of all pixels from CDS / AGC circuit 107, specifically those from the region to be used for focus detection (focus detection region), to pass through. AF signal processing circuit 113 generates an AF evaluation value by extracting high-frequency components from the signal passing through AF gate 112. The generated AF evaluation value is output to control unit 114. The AF evaluation value indicates the sharpness (contrast state) of the video generated based on the output signal from image sensor 106, and since sharpness varies according to the focus state (focus level) of the camera optics, the AF evaluation value ultimately becomes a signal indicating the focus state of the camera optics. Alternatively, AF signal processing unit 113 can use a known phase difference focus detection method to calculate the correlation between two image signals output from CDS / AGC circuit 107 for AF to calculate the amount of defocus, and output the amount of defocus to control unit 114.

[0052] The control unit 114 includes a processor such as a central processing unit (CPU) or a microprocessor unit (MPU) and storage components such as memory. The control unit 114 may include computing circuitry and causes the computing circuitry to perform some of the computational functions to be performed by the processor. The control unit 14 controls the operation of the entire camera and also performs AF control for focusing by controlling the focus drive source 111 and moving the focus lens 105 based on AF evaluation values, defocus amount, etc.

[0053] The face region detection unit 116 performs known face detection processing on the image signal and detects the human face region in the captured image. That is, the face region detection unit 116 detects a predefined subject based on electrical signals. The detection result is sent to the control unit 114. Examples of face region detection processing include methods that extract skin color regions from the grayscale colors of pixels represented by image data and detect faces based on their matching degree with a pre-prepared facial contour plate. In addition, there are methods for face detection that extract facial feature points such as eyes, nose, and mouth using known pattern recognition techniques, but the present invention is not limited to face detection processing methods, and any method can be used. This embodiment also selects the face as the first part of the subject to be detected, but the first part only needs to be a part of the subject and can be other parts such as arms or legs. Known methods similar to those described for faces can also be used for the detection of these other parts.

[0054] The head region detection unit 117 detects a predetermined region from the image, wherein the target subject region is defined as the head region. That is, the head region detection unit 117 detects a predefined subject (here, the head region) based on electrical signals. In the head region detection according to this embodiment, the head region is detected based on image data using learning data, and the detection result is sent to the control unit 114. In this embodiment, the head region is also described as a block to be detected, but this block only needs to be a block used for detecting human regions, such as the head region other than the face, the upper body region, the torso region, or the whole body region. In this embodiment, the head region is selected as the second part, but the second part can also be any other part used as part of the subject, as long as that part is other than the first part.

[0055] Based on the detection results of the face region detection unit 116 and the head region detection unit 117, the control unit 114 sends information to the AF gate 112 to set the focus detection area to the position of the focus detection area in the camera frame, including the face region and the head region. In the first embodiment, when a face is detected, the focus detection area is set based on the position / size of the face region. In contrast, when no face is detected and only the head region is detected, the focus detection area is set based on the position / size of the head region. When only the head region is detected, the position / size of the subject's face region can be estimated based on the head region detection results, and the focus detection area can also be set based on the estimated position / size.

[0056] The control unit 114 includes a main subject candidate determination unit 120, a main subject determination unit 121, and a main subject candidate reset determination unit 123. The main subject candidate determination unit 120 determines whether a subject can be a main subject candidate based on the detection results of the face region detected by the face region detection unit 116 and the detection results of the head region detected by the head region detection unit 117. Then, through the main subject determination process described below, a main subject is selected from the subjects determined as main subject candidates, and a process is performed to overlay a frame indicating the face region and head region onto the captured image, and the frame and the captured image are displayed on the monitor device 109.

[0057] In this embodiment, the control unit 114 performs focus control based on focus detection results (such as AF evaluation values ​​and defocusing amounts) obtained from focus detection areas of the face and head regions of the main subject selected by the main subject determination unit 121. Alternatively, focus control can be performed using focus detection results of the face and head regions of subjects other than the main subject, which are determined as main subject candidates by the main subject candidate determination unit 120. Focus control is a known technique, therefore a description of the control details will be omitted. Focus control is performed according to a computer program stored in the control unit 114.

[0058] Aperture drive source 118 includes an actuator and a driver for driving aperture 103. To acquire the brightness value of the metering frame in the image, the brightness information detection / calculation circuit 119 acquires the metering value from the signal read from the CDS / AGC circuit 107 and standardizes the measured metering value through calculation. Control unit 114 then calculates the difference between the metering value and the target value set to obtain appropriate exposure. Subsequently, the aperture correction drive amount is calculated based on the calculated difference, and control unit 114 controls the driving of aperture drive source 118. In this embodiment, it will be described assuming that the metering frame set in the image is fixed, but the metering frame can also be set to track the face or head area of ​​the main subject.

[0059] The operation unit 122 comprises various operating components used as an input unit for receiving operations from a user. The operation unit 122 includes at least a portion of the following operation units: shutter button, main electronic dial, power switch, secondary electronic dial, cross key, SET button, video button, AF lock button, zoom button, replay button, menu button, touch bar, and touch panel.

[0060] Next, we will refer to Figures 2 to 5 This describes the process of determining the main subject according to this embodiment.

[0061] First, refer to Figure 2The overall flow of the main subject determination process is described. This process is executed according to a computer program stored in the camera microcomputer 114. A system is assumed to execute a series of processes according to this embodiment in synchronization with the camera recording cycle, but the system is not limited to this.

[0062] First, in step S200, the control unit 114 retrieves setting information about the user-defined method for determining the main subject candidate from its memory. In this embodiment, the configurable main subject candidate determination method includes the following two methods.

[0063] The first determination method is based on the idea that a subject whose first part (face) has never been detected since it entered the frame cannot be the primary subject. That is, the first determination method is a method of setting only subjects whose first part (face) is detected at least once in multiple frames within a predetermined time period as primary subject candidates (hereinafter referred to as the first setting).

[0064] The second determination method is based on the idea that even if a subject's first part (face) has never been detected in multiple frames within a predetermined time period from the moment it enters the frame, it can still be considered the main subject, so as to capture images of subjects whose first part (face) cannot be detected due to goggles, etc. That is, the second determination method is a method of setting subjects with detected first parts (face) or second parts (head) that are different from the first part as main subject candidates (hereinafter referred to as the second setting).

[0065] Next, in step S201, the control unit 114 obtains the face detection results and head detection results from the face region detection unit 116 and the head region detection unit 117. In this embodiment, the position information, size information, gradient information, or detection reliability information of the face or head region are obtained as the face or head detection results, but the face or head detection results are not limited to these.

[0066] The control unit 114 also determines which face and which head belong to the same person based on the acquired face and head detection results. The control unit 114 further determines, based on position or size, which humans in previous frames have the same face and head detection results acquired in the current frame. Humans identified as having the same detection results as those in previous frames are assigned the same ID as the same human in the previous frame, and subsequent processing is performed.

[0067] Next, in step S202, the main subject candidate determination unit 120 performs a main subject candidate determination process. In the main subject candidate determination process, the main subject candidate determination unit 120 determines from the detected people whether each person is a candidate subject that can be used as a main subject, on which the focus is to be placed and the brightness is to be controlled.

[0068] The following will refer to Figure 3 Describe the details of the main subject candidate determination process.

[0069] Next, in step S203, the control unit 114 determines whether the user has issued an instruction to specify a subject via the operation unit 122. If it is determined that an instruction to specify a subject has been issued, the process proceeds to step S204. If it is determined that no instruction to specify a subject has been issued, the process proceeds to step S205. The operation unit 122 can specify the subject using a touch panel, or it can specify the subject by detecting direction or pressure, such as using a crosshair.

[0070] Next, in step S204, the main subject locking unit 121 performs main subject locking processing. The main subject locking processing is a process for determining the area of ​​the main subject based on the detection results obtained in step S201 and the information about the subject designation indication obtained in step S203, regardless of the result of the main subject candidate determined in step S202.

[0071] The following will refer to the appendix. Figure 4 Describe the details of the main subject locking process.

[0072] Next, in step S205, a main subject determination process is performed to determine the main subject from the main subject candidates determined by the main subject candidate determination unit.

[0073] The following will refer to Figure 5 Describe the details of the main subject determination process.

[0074] Next, in step S206, the control unit 114 performs frame display control processing on the monitor device 109. Specifically, the control unit 114 displays a frame indicating that the subject is the main subject in the area of ​​the main subject determined in steps S204 and S205, superimposed on the face or head portion. The control unit 114 also displays a frame indicating that the subject is a secondary subject on the face or head portion of subjects that have been identified as primary subject candidates by the primary subject candidate determination unit 120 but have not yet been identified as primary subjects (subjects identified as secondary subjects). In this embodiment, when the primary subject is locked in step S204, the frame indicating that the primary subject has been locked is displayed for the user, and the frame is not displayed on the face or head portion of the human who has been identified as a primary subject candidate but has not yet been identified as a primary subject.

[0075] Next, in step S207, the control unit 114 sets the subject area of ​​the main subject in the AF gate 112 and performs AF control based on the AF evaluation value obtained from the AF signal processing unit 113. The method of using a contrast evaluation value indicating the contrast or sharpness of the subject as the AF evaluation value can be adopted, or the method of using the amount of defocus calculated based on the phase difference up to the focus position as the AF evaluation value can also be adopted.

[0076] Next, we will refer to Figure 3 Describe the details of the subject candidate determination process.

[0077] First, in step S300, the subject candidate determination unit 120 of the control unit 114 obtains the detection result of the first person (hereinafter referred to as the target human) from the detection result obtained in step S201 to determine whether a face has been detected. In this embodiment, the detection result includes the position and size information of the detected face, head, and other regions in the image, as well as an ID used to distinguish them from other parts, and determines whether a face has been detected based on the ID.

[0078] If it is determined that a face has been detected, the process proceeds to step S301. If it is determined that a face has not been detected, the process proceeds to step S302.

[0079] Next, in step S301, the control unit 114 stores information about the detected facial region (e.g., location, size) as subject region data of the target human and proceeds to step S310.

[0080] Next, in step S302, the control unit 114 refers to the setting information of the main subject candidate determination method already acquired in step S200. If the setting information indicates a first setting, the process proceeds to step S303. If the setting information indicates a second setting, the process proceeds to step S304.

[0081] Next, in step S303, the control unit 114 determines whether the subject has already been identified as a primary subject candidate in a previous primary subject candidate determination process based on the primary subject candidate flag. If it is determined that the subject has already been identified as a primary subject candidate in a previous primary subject candidate determination process (the primary subject candidate flag is set to ON), the process proceeds to step S304. In contrast, if it is determined that the subject has never been identified as a primary subject candidate (never within a predetermined time period), including previous frames (the primary subject candidate flag is set to OFF), the process proceeds to step S312.

[0082] Next, in step S304, it is determined whether the detection results for the target human include head detection results. If head detection results are included, the process proceeds to step S305. If head detection results are not included, the process proceeds to step S311.

[0083] Next, in step S305, information about the detected head region (e.g., location, size) is stored as subject region data of the target human, and the process proceeds to step S310.

[0084] When the head detection area is stored as subject area data of the target human in step S305, it is effective to slightly shift the position of the subject area downwards (relative to the body direction of the head) relative to the stored position of the head detection area. This is because the head area detection unit 117 can detect humans even when the human is facing backwards or to the side. The back and side areas of the human head tend to have low contrast compared to the face and are difficult to focus on. Furthermore, controlling brightness based on the hair portion at the back of the head may result in an overly bright setting for the entire subject. Specifically, by shifting the position of the subject area in the body direction by approximately 1 / 4 of the head detection size, not only the hair portion at the back of the head is included, but also the collar portion. This increases the likelihood of capturing the contrast of the neck and collar, and can be expected to improve focusing accuracy and exposure control accuracy.

[0085] Next, in step S310, since the face detection area or head detection area was substituted into the subject area in step S301 or S305, and the subject has become the main subject candidate, the main subject candidate flag is set to ON, and the process proceeds to step S312.

[0086] In contrast, in step S311, since the region identified as the primary subject candidate region in the previous frame does not include either face detection results or head detection results, the subject is determined not to be a primary subject candidate, the primary subject candidate flag is set to OFF, and the process proceeds to step S312. The timing for setting the primary subject candidate flag to OFF can be the timing when face detection or head detection results have become absent once, or the timing when face detection or head detection results have been absent for several consecutive times (a predetermined number of frames).

[0087] Next, in step S312, it is determined whether the inspection of all detection results obtained in step S201 has been completed. If the inspection of all detection results has been completed, the process ends. If the inspection of all detection results has not been completed, the process returns to step S300, and the next detection result is set as the target human.

[0088] When the setting information indicates the first setting, the subject is identified as a primary subject candidate only if it is determined that a face has been detected in the process of step S300, or only if the primary subject candidate flag is set to ON in the process of step S303 and a head has been detected in step S304. These conditions implement the idea that a subject whose face has never been detected since the subject entered the frame cannot be a primary subject. This is because a subject whose face has never been detected is not looking at the camera and is unlikely to be the primary subject in the shot, on which the focus would be placed and whose brightness would be controlled.

[0089] Furthermore, when the setting information indicates the second setting, the subject is identified as a primary subject candidate only if it is determined that a face has been detected in the processing of step S300, or only if it is determined that a head has been detected in step S304. This condition realizes the idea that a subject whose face has never been detected since it entered the frame can be a primary subject. Because the second setting is intended for use when it is desired to capture an image of a subject whose face is undetectable due to goggles or a mask, it is independent of whether a face has already been detected. This is also because a subject where only the head is detected may be the primary subject in the image, a subject on which the focus is desired and whose brightness is desired to be controlled.

[0090] Through the above processing, when the setting information indicates the first setting, it is possible to prevent unnecessary humans whose faces or heads are detected in the image and would not be identified as primary subjects from being identified as primary subject candidates. Furthermore, while preventing focusing and brightness control on unexpected humans, when the setting information indicates the second setting, subjects whose faces are undetectable due to goggles can also become primary subject candidates. This configuration increases the possibility of placing focus on and controlling brightness for humans that the user intends to see.

[0091] Next, we will refer to Figure 4 The main subject locking determination method is described and will be performed by the main subject determination unit 121 of the control unit 114.

[0092] First, in step S401, it is determined whether the position of the face detection result or head detection result obtained in step S201 within the image of the subject received by the operation unit 122 (within the frame, XY plane) (touch position in the case of a touch panel) is within a predetermined range. This predetermined range can be within the size of the detected face or head, or it can include areas outside the size of the detected face or head, such as a range within twice the size of the face or head. Furthermore, it can also be within a predetermined distance relative to the depth direction of the current subject (the Z direction perpendicular to the XY plane). If the face detection result or head detection result is within the predetermined range, the process proceeds to step S402. If the face detection result or head detection result is not within the predetermined range, the process proceeds to step S409.

[0093] Next, in step S402, it is determined whether there are multiple face detection results and head detection results that are determined to exist within a predetermined range from the touch position. If only one subject is determined, the process proceeds to step S403. If multiple subjects are determined to exist, the process proceeds to step S404.

[0094] Next, in step S403, it is determined whether the reliability of the face detection results and head detection results identified as existing within a predetermined range from the touch position is at or greater than a threshold. If the reliability is determined to be at or greater than the threshold, the process proceeds to step S405. If the reliability is determined to be less than the threshold, the process proceeds to step S406.

[0095] Next, in step S404, it is determined whether the size of all face and head detection results identified as existing within a predetermined range from the touch position is equal to or less than a threshold. If the size is determined to be equal to or less than the threshold, the process proceeds to step S407. If the size is determined to be greater than or equal to the threshold (no in step S404), the process proceeds to step S408.

[0096] Next, in steps S405 and S408, the face detection result or head detection result closest to the touch position within the range of the touch position is locked. If the face detection result is included, the face detection result is used; if the face detection result is not included but the head detection result is included, the head detection result is used.

[0097] Next, in steps S406, S407 and S409, objects other than human faces or heads are tracked by extracting features of the subject based on the touch location (hereinafter referred to as object tracking).

[0098] Next, in step S410, the locked subject is determined as the main subject, and the main subject candidate flag of the locked human is set to ON.

[0099] Next, in step S411, the main subject candidate flag of objects other than locked humans is set to OFF to prevent objects other than locked subjects from becoming candidates, and the process ends.

[0100] The reason for performing object tracking in step S406 when the reliability is determined to be at or below the threshold in step SS403 is that, in cases where the reliability of head detection is low, such as when part of the head goes out of frame, the detection position and size become unstable, and the AF may become unstable.

[0101] Therefore, when the reliability of face or head detection is low, area designation is performed by tracking color and brightness information obtained at the time of touch, and the subject present at the user-specified location is temporarily tracked as the object. When a face or head with high reliability is detected nearby during tracking, the possibility of unstable AF can also be reduced by switching the tracking target to that face or head.

[0102] Furthermore, the reason for performing object tracking in step S404 when multiple small-sized faces or heads are detected within a predetermined range from the touch position is that, due to the offset of the touch position from the user-specified position, unintended humans are very likely to be set as tracking targets. Therefore, the subject present at the user-specified position is temporarily tracked as an object, and then, when a face or head of a predetermined size or larger is detected within the predetermined range, the tracking target is switched to that face or head. Using this configuration, focusing on unintended humans can be avoided.

[0103] Next, we will refer to Figure 5 , Figure 7A and Figure 7B The description describes the main subject determination process performed by the main subject determination unit 121 for the main subject candidate region in step S205.

[0104] First, in step S501, the main subject determination unit 121 refers to the setting information of the main subject candidate determination method already obtained in step S200, and if the setting information indicates a first setting, the process proceeds to step S502. If the setting information indicates a second setting, the process proceeds to step S503.

[0105] Next, in step S502, position priority weights are set based on position weights 1 and 2 corresponding to the distance from the center of the image to the candidate area of ​​the main subject, and the process proceeds to step S504.

[0106] In contrast, in step S503, position priority weights are set based on position weights 3 and 4 corresponding to the distance from the center of the image to the candidate area of ​​the main subject, and the process proceeds to step S505.

[0107] Here, we will refer to Figure 7A Describe the position weights 1, 2, 3, and 4 mentioned above.

[0108] Figure 7A This demonstrates how to calculate position weights based on distance from the center position.

[0109] The solid line in the figure represents the curve used to calculate the priority weight to be set in step S502, and the dashed line represents the curve used to calculate the priority weight to be set in step S503.

[0110] When the setting information of the main subject candidate determination method indicates the first setting, the weight of position becomes position weight 1 until the distance from the center of the frame becomes distance 1, and the position weight 1 linearly decreases to position weight 2 until distance 1 becomes distance 2. When the distance from the center of the frame is distance 2 or greater, the weight of position is fixed at position weight 2.

[0111] When the setting information of the main subject candidate determination method indicates the second setting, the weight of position becomes position weight 3 until the distance from the center of the frame becomes distance 3, and the position weight 3 linearly decreases to position weight 4 until distance 3 becomes distance 4. When the distance from the center of the frame is distance 4 or greater, the weight of position is fixed at position weight 4.

[0112] Next, in step S504, a weight corresponding to the size of the candidate region of the main subject is set, and the process proceeds to step S506.

[0113] Reference Figure 7B Describe the above size weights 1, 2, 3, and 4.

[0114] Figure 7B This demonstrates how to calculate size weights based on the size of the candidate region of the main subject.

[0115] The solid line in the figure represents the curve used to calculate the priority weight to be set in step S504, and the dashed line represents the curve used to calculate the priority weight to be set in step S505.

[0116] When the setting information of the main subject candidate determination method indicates the first setting, if the size of the detection area is size 1 or less than size 1, the weight of size becomes size weight 1, and size weight 1 increases linearly to size weight 2 until size 1 becomes size 2. If the size is size 2 or greater than size 2, the weight of size is fixed at size weight 2.

[0117] When the setting information of the main subject candidate determination method indicates the second setting, if the size of the detection area is size 3 or less than size 3, the size weight becomes size weight 3, and size weight 3 increases linearly to size weight 4, until size 3 becomes size 4. If the size is size 4 or greater than size 4, the weight of size is fixed at size weight 4.

[0118] When the setting information of the main subject candidate determination method indicates a second setting, only the head is detected as a main subject candidate. Therefore, the number of main subject candidate regions can become greater than the number when the setting information of the main subject candidate determination method indicates a first setting. Consequently, the likelihood of identifying an unexpected subject as the main subject also increases. Considering that the image of the main subject is captured at a relatively large size and at a relatively central location, such as... Figure 7A and 7B As shown, compared to the first setting indicated by the setting information in the main subject candidate determination method, the photographer assigns higher weight to larger subjects located closer to the center. This configuration avoids identifying unintended subjects as the main subject.

[0119] Regarding the position weights 1 to 4, distance weights 1 to 4, size weights 1 to 4, and size weights 1 to 4 in the attached figures, when the setting information indicates the second setting, the weight of the subject closer to the center of the frame becomes greater than the weight set when the setting information indicates the first setting. Furthermore, as long as a larger subject has a larger weight, the shape of the graph shown in the attached figures, the number of change points, and the position are not limited. In addition, in this embodiment, linearly varying weights as described above and weights cropped at the minimum and maximum points are used; however, any weight setting method can be used, as long as the idea remains the same: the smaller the distance from the center, the greater the weight; and the larger the size, the greater the weight.

[0120] Next, in step S506, weights corresponding to the detection state are set, and the process proceeds to step S507.

[0121] The following will combine Figure 6 Describe the processing of weights corresponding to the detection state.

[0122] Next, in step S507, the priority of the target region is calculated based on the weights set in steps S502 to S506.

[0123] As an example of a priority calculation method, the priority is calculated using the following formula, which adds the above weights together at a predetermined ratio.

[0124] Priority = α × (position weight) + β × (size weight) + γ × (detection weight)

[0125] α, β, and γ are the coefficients by which the weights are multiplied, and they can be set freely.

[0126] Priority calculation is not limited to the calculations mentioned above.

[0127] Next, in step S508, it is determined whether the inspection of all acquired detection results has been completed. If the inspection of all acquired detection results has been completed, the process ends. If not all acquired detection results have been inspected, the process returns to step S501, and the next detection result is set as the target human.

[0128] Next, in step S509, the region with the highest priority is determined from the regions whose priorities have been calculated in step S507, and it is the main subject region.

[0129] When determining the main subject region, the candidate region with the highest priority can be determined as the main subject region during the processing of each frame, or a specific candidate region can be determined as the main subject region if it has the highest priority within a predetermined time period.

[0130] Alternatively, the main subject candidate flag of a main subject candidate region with a priority lower than a predetermined threshold can be set to OFF, so that the main subject candidate region does not become a main subject candidate region until it is determined again to include the face detection result in a subsequent frame (when the processing result in step S300 becomes yes).

[0131] In this embodiment, weights regarding position and size are used as parameters for determining the main subject. However, the parameters are not limited to this; the main subject can be determined based on the reliability and orientation of face and head detection results, or whether both face and head are detected simultaneously or only one of them. If the subject is locked as the main subject in step S204, it can always be determined as the main subject.

[0132] Next, we will refer to Figure 6 The description is based on the weight settings of the detection state in step S506.

[0133] First, in step S601, the control unit 114 refers to the setting information of the main subject candidate determination method already acquired in step S200. If the setting information indicates a first setting, the process proceeds to step S602. If the setting information indicates a second setting, the process proceeds to step S603.

[0134] Next, in step S602, the subject candidate determination unit 120 of the control unit 114 acquires the target human based on the detection results obtained in step S201, and determines whether a face has been detected.

[0135] In step S602, the determination can be based on whether the face region detection unit 116 detects a face, or even if a face has not been detected, it can be based on whether the pupil region detection unit (not shown) has detected a pupil within the face (not shown). For example, when a person is wearing sunglasses, even if the face region detection unit 116 cannot detect a face, the pupil region detection unit can still detect the pupil based on the area detected by the head region detection unit 117. This is because this situation can be considered equivalent to detecting a face.

[0136] If it is determined that a face has been detected, the process proceeds to step S603. If it is determined that a face has not been detected, the process proceeds to step S605.

[0137] Next, in step S603, a timer is reset to maintain the time period indicating how long the head-only state without detected face lasts. This timer is the one that will be compared with the times TH1 and TH2 in steps S605 and S607, which will be described below.

[0138] Next, in step S604, the detection weight for the face / head is set to Pr1, and the process ends.

[0139] Next, in step S605, it is determined whether the time for the head-only state where no face was detected is less than time TH1. If the time is less than time TH1, the process proceeds to step S606. If time TH1 or more has elapsed, the process proceeds to step S607.

[0140] Next, in step S606, the detection weight for the face / head is set to Pr2, and the process ends.

[0141] Next, in step S607, it is determined whether the time for the head-only state where no face was detected is less than time TH2. If the time is less than time TH2, the process proceeds to step S608. If time TH2 or more has elapsed, the process proceeds to step S609.

[0142] Next, in step S608, the detection weight for the face / head is set to Pr3, and the process ends.

[0143] Next, in step S609, the detection weight for the face / head is set to Pr4, and the process ends.

[0144] The aforementioned detection weights Pr1 to Pr4 satisfy the relationship Pr1≥Pr2≥Pr3≥Pr4, and when the main subject candidate determination method has a first setting, the priority of the detection weights is set to a lower priority as time progresses.

[0145] As an example, when Pr1>Pr2>Pr3>Pr4 and TH2>TH1≠0, the detection weight Pr1, set when a face is detected, becomes the highest, and subjects with detected faces are given priority. Since the detection weight decreases to detection weight Pr2 when a face becomes undetectable and only a head is detected, if other subjects with detected faces exist, a higher detection weight is set for those subjects, increasing the likelihood of increased priority. Furthermore, if the state of no face detection results persists for a predetermined time greater than or equal to time TH1, the detection weight further decreases to detection weight Pr3 after the predetermined time. By providing a time threshold TH2 and a detection weight Pr3, when a face temporarily becomes undetectable and its priority decreases, the detection weight becomes detection weight Pr2. In contrast, when the undetectable state of a face persists for a predetermined time period and its priority decreases, the detection weight becomes detection weight Pr3. Therefore, a distinction can be made between these two values. Additionally, when the state of no face detection results persists for a time greater than or equal to time TH2, the detection weight becomes the lowest detection weight Pr4.

[0146] In step S509, the detection weight Pr4 can be set to a value such that the priority always becomes less than a predetermined threshold. By making the priority less than the threshold, it is possible to prevent a subject facing backwards for an extended period from becoming the main subject, regardless of the location or size of the area, until the face is detected again. This configuration prevents unintended subjects from being designated as the main subject.

[0147] As another example, when Pr1 = Pr2 > Pr3 ≥ Pr4, during the time period from the moment immediately following the state where no face is detected and only the head is detected until time TH1, a detection weight equal to the detection weight of a subject whose face has already been detected is calculated. This is to perform control in such a way that subjects whose heads are detected are not unintentionally excluded from the main subject if it is expected that a face will be detected again immediately after causing a head-only state.

[0148] Furthermore, in the above example, the detection weight Pr1 remains unchanged regardless of whether the main subject candidate determination method is in the first setting or the second setting. However, it is also possible to consider setting the detection weight Pr1 to a larger value when the main subject candidate determination method is in the second setting than the value set when the main subject candidate determination method is in the first setting. This is because since the user intentionally sets the settings in a way that they expect to select subjects from the head as the main subject, they may expect subjects that are not detectable on the face to be prioritized over subjects that are detectable on the face.

[0149] Please refer to the appendix Figures 9A to 9G Describe the above process.

[0150] Figures 9A to 9D This illustrates the case where the settings for the main subject candidate determination method are the first settings, and... Figures 9E to 9G This illustrates the case where the main subject candidate determination method is set to the second setting.

[0151] Figure 9A The image shows a scene where a human 901, facing backward, has entered the frame. In this case, it is unlikely that the image of human 901 will be captured as the main subject, therefore no region setting and frame display are performed for the main subject candidate region.

[0152] Figure 9B The image shows human 901 facing forward. In this case, the human is looking at the camera, so the image of the human is considered the main subject, and a region setting for the main subject candidate area is performed for human 901, and a display box 902 is generated. Furthermore, since there is only one main subject candidate, human 901 is automatically selected as the main subject.

[0153] Figure 9C It shows that 901 humans from Figure 9B The image shows a situation where the subject is facing forward but then facing backward. In this case, Human 901 has already faced forward once and is considered likely to face forward again, so Human 901 continues to be set as the primary subject candidate area. By continuing to set Human 901 as the primary subject candidate area, focusing remains on Human 901, and it is possible to avoid placing focus on unintended areas and control brightness for unintended areas.

[0154] Figure 9DThe example shows a scenario where human 901 remains facing backward. In this case, human 901 is considered unlikely to be the primary subject, and therefore, the weight corresponding to the detection state is set to detection weights Pr2, Pr3, or Pr4 via determination in step S605 or S607. This weight is a lower value than the value set when a face has already been detected (when the human is facing forward). Using the weights set in the above process, a priority is calculated in step S507. However, as a result of the priority calculation, the primary subject candidate flag for a primary subject candidate region that has a priority less than a predetermined threshold can be set to OFF, preventing that primary subject candidate region from becoming a primary subject candidate region until it is determined again to include a face detection result in a subsequent frame (until it is determined in the process of step S300 that a face has been detected (when the processing result in step S300 becomes yes)).

[0155] The above processing can prevent a subject that is always facing backward from becoming the main subject candidate when the user has already set the second setting. Therefore, it can avoid focusing on the area of ​​the unintended subject and control the brightness of the unintended subject area.

[0156] Figure 9E The scene depicts a human with a detectable head but an undetectable face wearing goggles, and a subject with a detectable face in the background.

[0157] Although the face of human 903 was undetectable, the head was detected. Furthermore, the faces and heads of two other humans (human 904 and human 905) in the background were detected.

[0158] In the above scenario, equal detection weights are assigned to subjects that only detect heads and subjects that only detect faces, thus determining the primary subject based solely on position and size. Therefore, the largest human (903) located closer to the center is selected as the primary subject, and the focus can be placed on the user-expected subject, with brightness controlled accordingly. Because the priority does not decrease even when only the head is detected, tracking using head detection can be prevented from ceasing after a predetermined time.

[0159] As described above, when the second setting is set, by setting the detection weight Pr1 to a larger value than the value to be set when the first setting is set, the probability that human 904 or human 905 will become the main subject becomes lower, and the probability that human 903 can continue to be the main subject becomes higher.

[0160] in addition, Figure 9FImages of humans 906 and 907 are shown, with both humans having undetectable faces and only their heads detected. Figure 9G It shows that 906 humans from Figure 9F The state in the frame goes out of frame, and only Human 907 exists, and the image of Human 907 is taken as the main subject.

[0161] exist Figure 9F In this process, by using the determination in step S601, head tracking of both humans 907 and 908 is not stopped at a specific time. Therefore, even when human 906 goes to... Figure 9G After the out-of-frame transition, the main subject can be immediately switched to Human 907, and it becomes possible to record natural video with the focus on Human 907 and the brightness controlled for Human 907.

[0162] As mentioned above, the likelihood of the intended subject being set as the main subject can be increased and camera errors can be reduced based on the user's settings.

[0163] <Second Embodiment>

[0164] Next, a second embodiment of the present invention will be described. In the first embodiment described above, the weights are changed based on the detection status of a first body part (facial region) and parts other than the first body part (e.g., head region, upper body region, torso region, and whole body region). Furthermore, the detection weights to be used in priority calculation are changed according to the detection body part of the human body.

[0165] In recent years, not only humans but also various types of subjects such as animals and vehicles have become detectable. A technique exists in which, when a subject that the user has set to be prioritized (hereinafter referred to as the priority subject) and other subjects (hereinafter referred to as non-priority subjects) are detected, the focus can be placed on the user-expected subject by setting the priority subject as the primary subject, and the brightness can be controlled for the user-expected subject.

[0166] However, although there are priority subjects and non-priority subjects in the frame, if the non-priority subject is clearly suitable as the main subject according to the scene, the settings need to be changed every time.

[0167] The feature of this embodiment is that, even under the above circumstances, while setting the expected subject as the main subject through the subject set by the priority user, a non-priority subject that may be the main subject is set as the main subject according to the scenario, without having to painstakingly change the settings.

[0168] Figure 13The configuration of a camera device, such as a video camera, including an autofocus device, according to a second embodiment is shown. Because it is assigned to... Figure 1 Blocks with the same number in Figure 1 The blocks in the text are the same, so the description will be omitted.

[0169] Human region detection unit 131 performs known human detection processing on the image signal and detects human regions (face region, head region, upper body region, torso region, full body region, etc.) within the camera frame. The detection results are sent to control unit 114. Examples of human region detection processing include, for example, methods that extract skin color regions from the grayscale colors of pixels represented by image data and detect human regions based on their matching degree with a pre-prepared human profile template. Another example is a method that detects human regions based on image data using learned data. Yet another example is a method that performs face detection by extracting facial feature points such as eyes, nose, and mouth using known pattern recognition techniques, but the present invention is not limited to human detection processing methods, and any method can be used.

[0170] Animal region detection unit 132 performs known animal detection processing on the image signal and detects animal regions (face region, head region, upper body region, torso region, whole body region, etc.) within the camera frame. The detection results are sent to control unit 114. Examples of animal region detection processing include methods that detect animal regions based on the degree of matching with a pre-prepared animal profile template. Another example is a method that detects animal regions based on image data using learning data. Yet another example is a method that performs animal facial region detection by extracting facial feature points such as eyes, nose, and mouth using known pattern recognition techniques, etc. However, the present invention is not limited to human detection processing methods, and any method can be used.

[0171] The vehicle region detection unit 133 performs known vehicle detection processing on the image signal and detects vehicle regions (the entire vehicle, driver, wheels, windshield, etc.) within the camera frame. The detection results are sent to the control unit 114. Examples of vehicle region detection processing include methods for detecting vehicle regions based on matching degrees with a pre-prepared vehicle outline. Another example is a method for detecting vehicle regions based on image data using learning data. Yet another example is a method for performing vehicle region detection by extracting feature points of the vehicle using known pattern recognition techniques, etc. However, the present invention is not limited to vehicle detection processing methods, and any method can be used.

[0172] Next, although according to the first embodiment Figure 2In step S200 of the main subject determination process, a first setting and a second setting are provided, but in this embodiment, the following three settings are provided:

[0173] The first setting (referred to as the third setting to distinguish it from the first setting in the first embodiment) is the setting that prioritizes human areas (hereinafter referred to as the human priority setting);

[0174] The second setting (fourth setting) is the setting that prioritizes areas for animals such as dogs, cats, or birds (hereinafter referred to as animal priority setting); and

[0175] The third setting (fifth setting) is the setting of areas that prioritize vehicles such as cars, bicycles or trains (hereinafter referred to as vehicle priority setting).

[0176] In this embodiment, the above-described types are detected; however, when a setting is provided to switch the type of subject to be detected, the method is not limited to the above description. Different types of animals can also be set, such that the first setting is for dogs, the second setting is for cats, and the third setting is for birds.

[0177] In this embodiment, all types can be detected even when any of the three settings are set, but for example, the highest priority is given to humans only when the human priority setting is set, and thus processing can be performed to not detect subjects other than humans.

[0178] In the first embodiment, in step S201, the control unit 114 performs the process of acquiring face detection results and head detection results from the face region detection unit 116 and the head region detection unit 117. However, in this embodiment, the control unit 114 performs the process of acquiring human detection results, animal detection results, and vehicle detection results from the human region detection unit 131, the animal region detection unit 132, and the vehicle region detection unit 133, respectively.

[0179] In the following description, the processes described in the first embodiment are assigned the same number, and the description will be omitted.

[0180] Next, we will refer to Figure 10 The process of determining the main subject candidate, which is a feature of this embodiment, is described.

[0181] The processing is performed according to a computer program stored in the camera microcomputer 114. This embodiment assumes a system that performs a series of processes synchronously with the camera's recording cycle, but the system is not limited to this.

[0182] First, in step S1000, the subject candidate determination unit 120 of the control unit obtains a first detection result (hereinafter referred to as the target subject) from the detection result obtained in step S201, and determines whether a human has been detected.

[0183] If it is determined that a human has been detected, the process proceeds to step S1001. If it is determined that no human has been detected, the process proceeds to step S1002.

[0184] Next, in step S1001, the detected human region is stored as subject region data of the target subject (e.g., location, size), and the processing proceeds to step S310.

[0185] Next, in step S1002, it is determined whether the detection results of the target subject include animal detection results. If animal detection results are included, the process proceeds to step S1003. If animal detection results are not included, the process proceeds to step S1004.

[0186] Next, in step S1003, the detected animal region is stored as subject region data of the target subject, and the processing proceeds to step S310.

[0187] Next, in step S1004, it is determined whether the detection results of the target subject include the vehicle detection results. If the vehicle detection results are included, the process proceeds to step S1005. If the vehicle detection results are not included, it is determined that nothing was detected in the target area, and the process proceeds to step S311.

[0188] Next, in step S1005, the detected area of ​​the vehicle is stored as the subject area data of the target subject, and the processing proceeds to step S310.

[0189] Next, we will refer to Figure 14 Flowcharts and Figures 8A to 8C The figure in the diagram is used to describe the main subject determination process according to the second embodiment.

[0190] First, in step S1401, it is determined whether the setting information of the main subject candidate determination method indicates human priority. If it is determined that the setting information indicates human priority, the process proceeds to step S1402. If it is determined that the setting information indicates a setting other than human priority, the process proceeds to step S1403.

[0191] Next, in step S1402, the detection type weights are set to satisfy Pr_H>Pr_A and Pr_H>Pr_V, and the process proceeds to step S1406.

[0192] Next, in step S1403, it is determined whether the setting information of the main subject candidate determination method indicates animal priority. If it is determined that the setting information indicates animal priority, the process proceeds to step S1404. If it is determined that the setting information indicates a setting other than animal priority (in this embodiment, if it is determined that the setting information indicates vehicle priority), the process proceeds to step S1405.

[0193] Next, in step S1404, the detection type weights are set to satisfy Pr_A>Pr_H>Pr_V, and the process proceeds to step S1406.

[0194] Next, in step S1405, the detection type weights are set to satisfy Pr_V>Pr_H>Pr_A, and the process proceeds to step S1406.

[0195] Next, in step S1406, weights corresponding to the detection type are set, and the process proceeds to step S507.

[0196] The following will combine Figure 11 The processing method is explained.

[0197] Next, refer to the appendix Figures 8A to 8C Details are described regarding the priorities Pr_H, Pr_A, and Pr_V set in steps S1402, S1404, and S1405.

[0198] Figure 8A This diagram illustrates the priorities to be set when the setting information of the main subject candidate determination method indicates a human-priority setting. When a human-priority setting is set, the human priority weight Pr_H is set to its maximum value, and the animal and vehicle priorities Pr_A and Pr_V are set to values ​​lower than the priority weight Pr_H. In this embodiment, because a human-priority setting is set, the animal and vehicle priorities Pr_A and Pr_V are set to the same value, but they do not always need to be set to the same value; they only need to be set to values ​​lower than the priority weight Pr_H.

[0199] The priorities Pr_A and Pr_V for animals and vehicles need to be set so that when animals and vehicles are detected at the center of the frame and have the largest size, their priority is higher than that of humans, which is calculated based on the weights corresponding to the distance from the center of the frame and the size of the subject.

[0200] By configuring the settings as described above, while prioritizing the user-defined subject as the primary subject, a non-priority subject that might be the primary subject can be set as the primary subject depending on the scene, without the need to expend effort changing the settings. Therefore, even when a non-priority subject is clearly suitable as the primary subject according to the scene, it is not necessary to change the settings every time, and the focus can be placed on the expected subject to be used as the primary subject, and the brightness can be controlled for that expected subject.

[0201] Figure 8B This diagram illustrates the priorities set when the setting information for the primary subject candidate determination method indicates an animal-first setting. When an animal-first setting is set, the animal priority weight Pr_A is set to its maximum value, and the priorities Pr_H and Pr_V for humans and vehicles are set to values ​​lower than the priority weight Pr_A. The human priority Pr_H also has a higher value than the vehicle priority weight Pr_V, which is used for non-priority subjects other than humans. This is based on the idea that even with an animal-first setting, humans can often become the primary subject among both non-priority subjects and vehicles. Therefore, setting the human priority weight higher than the vehicle priority weight is effective even if they are both non-priority subjects.

[0202] When an animal-priority setting or a vehicle-priority setting is set, the priority weight Pr_H for humans is also set to a higher value than the priority weights Pr_A and Pr_V for non-priority subjects (animals and vehicles) when a human-priority setting is set. This is based on the idea that humans are more likely to be the primary subject when an animal or vehicle-priority setting is set than when an animal or vehicle-priority setting is set.

[0203] Figure 8C The exchange was shown Figure 8B The settings for animals and vehicles in the game will be omitted from the description.

[0204] Next, we will refer to Figure 11 Describe the setting of weights corresponding to the detection type in step S1406.

[0205] Next, in step S1101, the subject candidate determination unit 120 of the control unit obtains the target human based on the detection results obtained in step S201, and determines whether the detected subject type is human.

[0206] If it is determined that a human has been detected, the process proceeds to step S1102. If it is determined that an object other than a human has been detected, the process proceeds to step S1103.

[0207] Next, in step S1102, the priority weight Pr_H of the human is set to the priority weight corresponding to the priority type setting, and the process ends.

[0208] Next, in step S1103, the subject candidate determination unit 120 obtains the target human based on the detection results obtained in step S201, and determines whether the detected subject type is an animal.

[0209] If it is determined that an animal has been detected, the process proceeds to step S1104. If it is determined that an object other than an animal has been detected (in this embodiment, if it is determined that a vehicle has been detected), the process proceeds to step S1105.

[0210] Next, in step S1104, the animal's priority weight Pr_A is set to the priority weight corresponding to the priority type setting, and the process ends.

[0211] Next, in step S1105, the priority weight Pr_V of the vehicle is set to the priority weight corresponding to the priority type setting, and the process ends.

[0212] The following section describes the priority values ​​for humans, animals, and vehicles, and therefore which type of subject is prioritized, when the settings information for the primary subject candidate determination method indicates each setting. Specifically, refer to the appendix... Figures 12A to 12F Examples describing the actual settings for position weights 1 and 2, size weights 1 and 2, and priorities Pr_A, Pr_V, and Pr_H.

[0213] It is also assumed that the possible values ​​of distance from the center of the image, size, position weight, size weight, and type weight are standardized with the minimum value set to 0 and the maximum value set to 255.

[0214] The value to be set is not limited to the following description, and only needs to fall within the range described above.

[0215] In this embodiment, the values ​​are set as follows: distance 1 = 20, distance 2 = 200, position weight 1 = 200, position weight 2 = 20, size 1 = 20, size 2 = 200, size weight 1 = 20, and size weight 2 = 200. When human priority is set, Pr_H = 255, Pr_A = 100, and Pr_V = 100. When animal priority is set, Pr_H = 150, Pr_A = 255, and Pr_V = 100. When vehicle priority is set, Pr_H = 150, Pr_A = 100, and Pr_V = 255. The coefficients α, β, and γ to be used in the priority calculation are set to α = 1, β = 1, and γ = 2.

[0216] First, assuming human priority is set, Figure 12A This shows a situation where human 1200 appears approximately in the center, and animal 1201 appears at the edge of the frame.

[0217] When the distance from the center to Human 1200 is 0, the size of Human 1200 is 50, the distance from the center to Animal 1201 is 200, and the size of Animal 1201 is 30, the priority of Human 1200 is 750 and the priority of Animal 1201 is 250. Therefore, Human 1200 is selected as the main subject.

[0218] Next, Figure 12B This shows a situation where animal 1201 appears near the center and human 1200 appears at the edge of the frame.

[0219] Assuming the distance from the center to human 1200 is 150, the size of human 1200 is 40, the distance from the center to animal 1201 is 0, and the size of animal 1201 is 50, then the priority of human 1200 becomes 620 and the priority of animal 1201 becomes 450. Therefore, human 1200 is selected as the main subject.

[0220] Next, Figure 12C The image shows an animal 1201 appearing in a large size near the center, while a human 1200 appears in a small size at the edge of the image.

[0221] Assuming the distance from the center to human 1200 is 200, the size of human 1200 is 20, the distance from the center to animal 1201 is 0, and the size of animal 1201 is 180, then the priority of human 1200 is 550, and the priority of animal 1201 is 580. Therefore, animal 1201 is selected as the main subject.

[0222] Next, with animals given priority, Figure 12DThis shows a situation where animal 1201 appears near the center and human 1200 appears at the edge of the frame.

[0223] Assuming the distance from the center to human 1200 is 200, the size of human 1200 is 30, the distance from the center to animal 1201 is 0, and the size of animal 1201 is 50, then the priority of human 1200 is 350, and the priority of animal 1201 is 760. Therefore, animal 1201 is selected as the main subject.

[0224] Next, Figure 12E This shows a situation where human 1200 appears approximately in the center and animal 1201 appears at the edge of the frame.

[0225] Assuming the distance from the center to human 1200 is 0, the size of human 1200 is 50, the distance from the center to animal 1201 is 150, and the size of animal 1201 is 40, then the priority of human 1200 is 550, and the priority of animal 1201 is 620. Therefore, animal 1201 is selected as the main subject.

[0226] Next, Figure 12F This shows a situation where human 1200 appears in a large size near the center, while animal 1201 appears in a small size at the edge of the image.

[0227] Assuming the distance from the center to Human 1200 is 0, the size of Human 1200 is 180, the distance from the center to Animal 1201 is 200, and the size of Animal 1201 is 20, then the priority of Human 1200 is 680, and the priority of Animal 1201 is 550. Therefore, Human 1200 is selected as the main subject.

[0228] Through the above processing, while setting the expected subject as the main subject by prioritizing the subject set by the user, non-priority subjects that may be the main subject can be set as the main subject according to the scene, without having to painstakingly change the settings.

[0229] Preferred embodiments of the present invention have been described to date, but the invention is not limited to the above embodiments, and various modifications and changes can be made within its spirit. For example, in this embodiment, detection results of human facial and head regions have been described, but combinations of human facial and body regions, or combinations of animal facial and head or body regions, can also be used.

[0230] While the invention has been described in detail so far based on preferred embodiments, it is not limited to these specific embodiments, and various configurations are included without departing from the spirit of the invention. Some of the above embodiments may also be suitably combined.

[0231] The software program that implements the functions of the above embodiments is provided to a system or device including a computer that can execute the program directly from a recording medium or using wired / wireless communication, and the execution of the program is also included in this invention.

[0232] Accordingly, program code that is provided to and installed on a computer to implement the functional processing of the present invention also implements the present invention. That is, computer programs for implementing the functional processing of the present invention are also included in the present invention.

[0233] In this case, the format of the program, including its functionality such as object code, programs to be executed by the interpreter, or script data to be provided to the operating system (OS), is not restricted.

[0234] The recording medium used to provide the program can be, for example, a magnetic recording medium such as a hard disk or magnetic tape, an optical / magnetic optical storage medium, or a non-volatile semiconductor memory.

[0235] The method of providing the program can also be a method of storing the computer program that forms the present invention on a server on a computer network, and a method of downloading and programming the computer program on a client computer connected to the server.

[0236] This invention is not limited to the embodiments described above, and various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

[0237] This application is based on and claims priority from Japanese Patent Application No. 2023-053893, filed on March 29, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus, comprising: A first detection component is used to detect a first subject from an image; The second detection component is used to detect a second subject that is different from the first subject from the image; The determining component is configured to determine a main subject region as the main subject region based on at least one of the detection results obtained by the first detection component and the second detection component; as well as A settings component is used to receive a switch between a first setting and a second setting based on user actions. Wherein, when the setting component sets the first setting, the determining component, while prioritizing subjects detected by the first detection component as the first subject over subjects not detected by the first detection component, determines the main subject area, and When the setting component sets the second setting, the determining component sets the main subject area while prioritizing subjects where the first detection component does not detect the first subject and the second detection component detects the second subject over subjects where neither the first nor the second subject is detected.

2. The image processing apparatus according to claim 1, wherein, In the first setting, the determining component sets the subject detected by the first detection component as the main subject, and prioritizes multiple frames acquired within a predetermined time period to determine the main subject region.

3. The image processing apparatus according to claim 2, wherein, In the second setting, the determining component sets the subject as the main subject, regardless of whether the subject is the same subject detected by the first detection component in multiple frames acquired within a predetermined time period, and determines the main subject region.

4. The image processing apparatus according to claim 1, wherein, In the second setting, the determining component calculates the priority of the subject while assigning a greater weight to at least one of the subject's position and size than in the first setting, and determines the main subject based on the priority.

5. The image processing apparatus according to claim 4, wherein, In the second setting, the determining component reduces the weight to be added to the priority for a subject in which the first detection component has not detected the first subject for a predetermined time, before the predetermined time has elapsed.

6. The image processing apparatus according to claim 4, wherein, In the second setting, the determining component is given a greater weight than in the first setting when the first detection component does not detect the first subject.

7. The image processing apparatus according to claims 1 to 6, The first subject is the face of the subject, and in, The second subject is a part of the body that is different from the face, including the head, torso, and whole body of the subject.

8. The image processing apparatus according to any one of claims 1 to 6, The first subject is at least a part of a human being, and The second subject is at least a part of a subject that is different from the human being, including animals and vehicles.

9. A control method for an image processing device, the control method comprising: The first subject is detected from the image as the first detection; A second subject, different from the first subject, is detected from the image as a second detection; Based on at least one of the detection results obtained in the first detection and the second detection, a main subject region is determined as the region of the main subject. as well as The setup process involves receiving a switch between a first setting and a second setting based on user actions. Wherein, in the setting step, when the first setting is set, the determination of the main subject area, while prioritizing subjects that detect the first subject in the first detection over subjects that do not detect the first subject in the first detection, and the determination of the main subject area, and In the setting step, when the second setting is set, the main subject area is determined while prioritizing the subject that was not detected in the first detection but was detected in the second detection over the subject that was not detected in either the first or the second detection.

Images