Image processing apparatus, imaging apparatus, and control method of the same

Abstract

To provide an imaging apparatus which when a large number of subjects are projected, preferentially detects a subject attracting keen interest from a user and selects the subject as a main subject.SOLUTION: An imaging apparatus comprises: detection means for detecting a plurality of subjects from a captured image; area determination means for determining a candidate area for determining a main subject from the captured image; and main subject determination means for determining the main subject from the subjects detected by the detection means in the candidate area determined by the area determination means. The area determination means determines the candidate area by a method different corresponding to imaging setting for capturing the captured image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a detection device for detecting a subject. [Background technology]

[0002] Conventionally, imaging devices such as digital cameras have been commercialized that feature a tracking AF (autofocus) mode, which detects subjects such as people, animals, and vehicles from images continuously output from the imaging element and continuously optimizes the focus state for the detected subject.

[0003] When multiple subjects are detected, it is necessary to select from among them a subject for which the focus state and exposure state are actually optimized (hereinafter also referred to as the main subject).

[0004] As a method for selecting a main subject, a method has been disclosed in which a plurality of detected subjects are evaluated in some way and the main subject is determined based on the evaluation result.

[0005] For example, Patent Document 1 discloses a method for recognizing facial expressions and selecting main subject candidates according to the degree of smiling. Patent Document 2 discloses a configuration for performing individual recognition and controlling focus and exposure using information on whether it was determined that the subject is the same as a previously registered subject. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-156704 [Patent Document 2] Japanese Patent Application Laid-Open No. 2013-232060 Summary of the Invention [Problem to be solved by the invention]

[0007] However, evaluating all detected subjects poses challenges in terms of processing speed and power consumption. In recent years, it has become common to use deep learning algorithms to improve accuracy in evaluations of personal recognition and pose estimation. In devices such as digital cameras that have limited resources and use embedded software, it can be difficult to perform authentication processing, which requires a large processing load, on a large number of subjects within these limited resources.

[0008] Therefore, when there are a large number of subjects, it is desirable to select the subjects to be evaluated before evaluating each subject, and to evaluate only a number of subjects that meet the requirements for processing speed, power consumption, etc.

[0009] The present invention has been made in view of the above-mentioned problems, and its object is to provide an image processing device, an imaging device, and a control method therefor that can determine a main subject that meets the user's wishes from among a large number of subjects, while achieving faster processing speed or reduced power consumption. [Means for solving the problem]

[0010] The image processing device of the present invention comprises a detection means for detecting a plurality of subjects from a captured image, an area determination means for determining a candidate area for determining a main subject from the captured image, and a main subject determination means for determining a main subject from the subjects detected by the detection means within the candidate area determined by the area determination means, and is characterized in that the area determination means determines the candidate area in different ways depending on the imaging settings used to capture the captured image. [Effects of the Invention]

[0011] According to the present invention, it is possible to determine a main subject that meets the user's wishes from among a large number of subjects, while achieving an increase in processing speed or a reduction in power consumption. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a diagram showing the configuration of a digital single-lens camera as an embodiment of an imaging apparatus of the present invention. [Figure 2] Block diagram of the camera's control system. [Figure 3] FIG. 2 is a configuration diagram of a neural network used in this embodiment. [Figure 4] 10 is a flowchart showing a process of selecting one of a plurality of selection methods. [Figure 5] FIG. 10 is a diagram illustrating a specific example of determining a tracking target. [Figure 6] FIG. 4 is a diagram illustrating priority selection within the AF area. [Figure 7] FIG. 10 is a diagram illustrating tracking target neighborhood priority selection. [Figure 8] FIG. 10 is a diagram illustrating priority selection near the focal plane. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] FIG. 1 is a diagram showing the configuration of a digital single-lens camera (hereinafter also simply referred to as a camera) 100, which is one embodiment of the imaging device of the present invention. FIG. 2 is a diagram showing the configuration related to control of the digital single-lens camera 100. In the embodiment described below, an example in which the present invention is applied to an imaging device, as an example of an image processing device, capable of capturing images under different shooting conditions from images obtained by shooting. However, the present invention can be applied to any device that can generate images under different shooting conditions from images obtained by shooting.

[0015] In camera 100 of this embodiment, as shown in Fig. 1, a detachable, interchangeable lens unit 120 is attached to the front side (subject side) of camera body 101. Lens unit 120 has a focus lens 121, an aperture 122, etc., and is electrically connected to camera body 101 via mount contacts 123. Via this electrical connection, control unit 201 (see Fig. 2) of camera body 101 controls lens unit 120, and is able to adjust the amount of light taken into camera body 101 and the focal position. Note that focus lens 121 can also be manually adjusted (manual focus) by the user.

[0016] The image sensor 104, which captures a subject image, is composed of a CCD or CMOS sensor or the like, and includes an infrared cut filter, a low-pass filter, etc. The image sensor 104 receives light from the subject via an imaging optical system including a lens unit 120, photoelectrically converts the subject image formed on the imaging surface, and transmits signal information for generating a captured image to the arithmetic unit 102. The arithmetic unit 102 generates a captured image from the received signal information, stores it in an external storage device 107 (see FIG. 2), and displays it on a display unit 105 such as an LCD. The shutter 103 shields the image sensor 104 from light when not capturing an image, and opens it when capturing an image to expose the image sensor 104 to light.

[0017] Next, the configuration relating to the control of the camera 100 will be described with reference to FIG.

[0018] The arithmetic unit 102 includes a multi-core CPU capable of parallel processing of multiple tasks, RAM, ROM, and a dedicated circuit for executing specific arithmetic processing at high speed. The arithmetic unit 102 includes a control unit 201, a main subject calculation unit 202 for detecting a subject, a tracking calculation unit 203, a focus calculation unit 204, an exposure calculation unit 205, etc. The control unit 201 controls each unit of the camera body 101 and the lens unit 120.

[0019] 2 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. Alternatively, one or more of the functional blocks shown in FIG. 2 may be implemented by a combination of software and hardware. Therefore, even when different functional blocks are described as being responsible for the operations in the following description, they may be implemented by the same hardware.

[0020] The operation unit 106 has multiple input devices (buttons, switches, dials, etc.) that can be operated by the user. Some of the input devices in the operation unit 106 have names corresponding to the functions assigned to them. For example, these include a shutter button, a mode change switch, a power switch, etc. If the display unit 105 is a touch display, the touch panel is also included in the operation unit 106. The control unit 201 monitors the operation of the input devices included in the operation unit 106. When the control unit 201 detects an operation of an input device, it executes processing corresponding to the detected operation.

[0021] The shutter button has a first shutter switch (SW1) that turns on when pressed halfway, and a second shutter switch (SW2) that turns on when pressed all the way. When the control unit 201 detects that SW1 is turned on, it executes preparatory operations for still image shooting. The preparatory operations include AE ​​processing and AF processing. When the control unit 201 detects that SW2 is turned on, it executes still image shooting and recording operations according to the shooting conditions determined by the AE processing.

[0022] The mode changeover switch is an operation unit for switching between various shooting modes, playback modes, etc. However, the method of switching modes is not limited to operating this switch.

[0023] Main subject calculation unit 202 includes a subject detector 211 that detects subjects, a selection unit 212 that selects some or all of the subjects detected by subject detector 211, and a detection result output unit 214 that outputs the detection results of the subjects selected by selection unit 212. Main subject calculation unit 202 also includes an evaluation unit 215 that evaluates each of the subjects output by detection result output unit 214, and a main subject determination unit 216 that determines the main subject based on the output subjects and the evaluation result by evaluation unit 215.

[0024] The object detector 211 sequentially receives successive images acquired from the image sensor 104 and performs processing to detect objects such as people, animals, and vehicles from each image. Any known method such as AdaBoost or a convolutional neural network (CNN) can be used as the detection method. The implementation form may be a program running on a CPU, dedicated hardware, or a combination of these.

[0025] The configuration of the neural network used in this embodiment is shown in Figure 3. In this network, intermediate features are output by inputting an image into a network called the backbone. The features obtained through the backbone are input into separate networks for each task, which estimate the object position and subject frame of an object (such as a vehicle or animal). The network shown in Figure 3 obtains a "center map" that indicates the center position of the subject, and two "size maps" that indicate the width and height of the frame surrounding the subject (subject frame). Each map is a two-dimensional array, represented by a grid. The center map infers the likelihood that the array is the center position of the subject.

[0026] The center map indicates that the closer to the center of the black circle, the higher the likelihood of the corresponding subject. The size map consists of two maps, one for width and one for height, and infers the width and height of the object when that position is considered to be the center of the subject. The size map represents the magnitude of the value with the length of the double arrow, indicating that values ​​indicating width and height are inferred at the center position of the subject.

[0027] The object detector 211 is also capable of switching between different networks for each type of object to be detected. For example, the networks may be divided into categories such as people, animals, and vehicles, or more specifically into categories such as human faces, human heads, and human upper bodies. The object detector 211 selects some or all of these models for each input image and performs object detection processing. The network configuration may be different for each type of object, or a common backbone may be used with separate networks at the subsequent stages. Furthermore, networks with the same configuration may be used with different training data, and each type may be configured with parameters (weights) obtained for each type.

[0028] The selection unit 212 receives the center map and size map from the object detector 211, and selects (determines) a predetermined number of object regions (center coordinates, width, height) for each model. A specific selection method will be described later.

[0029] The selection unit 212 also integrates the inference results of multiple models. That is, it performs correlation between the multiple object regions selected by the multiple networks and then selects a predetermined number of object regions. For example, when a detection process is performed on an input image using a human face network and a human head network, both networks may output object regions for the same object. To prevent such overlap, the selection unit 212 determines that the object regions inferred by multiple models are the same object if their IoU (Intersection over Union) is equal to or greater than a predetermined threshold. In this case, the inference result of one model may be ignored, or the inference results of both models may be averaged. If the models are based on different parts of the same object, it is remembered that they are the same object. In this embodiment, this process based on IoU is called correlation (connection). After correlation is complete, a predetermined number of object regions are selected from all object regions remaining as object detection results. The selection method here can be the same as the selection performed for each model. Specific selection methods will be described later.

[0030] Once the selection is complete, a predetermined number of subject regions are output as the detection results as candidate regions for the main subject by the detection result output unit 214. Examples of output methods include outputting to a storage medium such as a volatile memory in the arithmetic device 102, or communicating with the CPU via I2C communication or the like. In this embodiment, each model outputs the detection results to a predetermined area in the volatile memory.

[0031] The evaluation unit 215 performs additional evaluation on each of the subject regions in the output detection results. Examples of evaluation include personal recognition to identify an individual and pose estimation to estimate the pose of the subject. In this embodiment, both personal recognition and pose estimation are performed, but the present invention may perform at least one or other evaluations.

[0032] In personal recognition, the facial region of the subject is cropped, and the cropped image is input into a recognition model to compare the similarity with previously registered (trained) people and evaluate whether the person is the person in question. Personal recognition also generally uses a neural network, and the processing time increases, especially when a network with high recognition performance is used. When an image contains multiple subjects, recognition processing must be performed for each of the subjects, and the processing time increases as the number of subjects increases. By selecting subjects in advance using the selection unit 212, as in this embodiment, it is possible to prevent the processing time from increasing too much.

[0033] In posture estimation, the evaluation unit 215 detects the joint points of the subject using a machine learning model for the input image, and estimates the position and posture of the subject by connecting joint points that are estimated to be joint points of the same subject. In this embodiment, the joint points are the top of the head, neck, both elbows, both wrists, both knees, and both ankles, but the present invention is not limited to this.

[0034] Similarly, in the above-mentioned personal recognition, pose estimation, and other evaluations using neural networks, an increase in processing speed as the number of subjects increases is an issue, and selecting subjects in advance, as in this embodiment, is effective in reducing the processing speed.

[0035] Main subject determination section 216 receives the detection result from detection result output section 214 and the evaluation result from evaluation section 215, and determines (decides) the main subject based on these results.

[0036] Tracking calculation unit 203 calculates an AF area and an AE (auto exposure) area on an LV (live view) image (corresponding to the imaging surface of image sensor 104) so ​​as to track the main subject determined by main subject determination unit 212. Specifically, the tracking calculation unit 203 determines the main subject or a peripheral area of ​​the main subject including the main subject as the target area for AF and AE.

[0037] The focus calculation unit 204 acquires focus information in the AF area (the contrast evaluation value of the LV image and the defocus amount of the imaging optical system). The control unit 201 transmits a focus instruction to the lens unit 120 to control the position of the focus lens 121 based on the focus information. The lens unit 120 drives the focus lens 121 in accordance with the focus instruction. This performs tracking AF as focus control for the main subject.

[0038] Furthermore, the exposure calculation unit 205 acquires brightness information in the AE region. The control unit 201 transmits an aperture instruction to the lens unit 120 to control the opening amount of the aperture 122 based on the brightness information. The lens unit 120 drives the aperture 122 in accordance with the aperture instruction. This performs tracking AE as exposure control for the main subject.

[0039] Next, the selection method in selection unit 212 will be described in detail with reference to Figures 4 to 8. Figure 4 is a flowchart showing the process of selecting one of a plurality of selection methods, and Figures 5 to 8 are diagrams explaining the screens displayed on display unit 105 for the operation of each processing step.

[0040] In step S401, control unit 201 determines whether the user is manually controlling the focus state. More specifically, this is determined when the focus ring arranged annularly around the optical axis of lens unit 120 is operated, or when a button on lens unit 120, a button on camera body 101, or a menu operation via display unit 105 is performed. If the focus control setting is set to manual focus, it is determined that the focus is being manually controlled. If the focus is being manually controlled, the process proceeds to step S405; if not, the process proceeds to step S402.

[0041] In step S402, control unit 201 refers to the confirmation state of the main subject and determines whether the main subject has been confirmed as a tracking target. If it has been confirmed as a tracking target, the process proceeds to step S404; if not, the process proceeds to step S403.

[0042] Here, a state in which the main subject is confirmed as the tracking target is a state in which the main subject is not switched even if another subject with a more favorable evaluation value that should be determined to be the main subject is detected. Operations for confirming a subject as the tracking target include half-pressing the release button or touching the subject displayed on display unit 105. When these operations are performed, the selected subject is confirmed as the tracking target, and main subject determination unit 216 continues to select the confirmed subject as the main subject until the confirmation is released (by releasing the half-press, touching a different subject, the passage of a predetermined time, etc.).

[0043] FIG. 5 is a diagram illustrating a specific example of determining a tracking target. As shown in FIG. 5(a), there are multiple subjects in the input image. When the computing device 102 has not determined a subject as a tracking target, the main subject determination unit 216 selects the main subject based on the coordinates in the image, the size of the subject, and the like. Typically, a subject 501 that is closer to the center of the image and larger (closer) is determined to have a higher priority as the main subject, and a frame is displayed around the main subject. Priorities are also predefined for subject types, such as animals over vehicles, people over animals, and so on. Subjects of higher priority types are evaluated more highly and are more likely to be selected as the main subject according to the priority. Touching another person 502 that is not the main subject changes the main subject, as shown in FIG. 5(b). Even though the other person 501 is more suitable as the main subject in terms of coordinates and size, this person 502 is subsequently tracked as the main subject. This state is the state in which the subject has been determined as the tracking target. It is desirable to change the design of the frame attached to the subject to be tracked displayed on display unit 105 so that it is easy to distinguish whether or not the subject has been confirmed as a tracking subject. In this embodiment, subjects that have not been confirmed as a tracking subject are displayed with a single frame, and subjects that have been confirmed as a tracking subject are displayed with a double frame.

[0044] Furthermore, even after the tracking target has been confirmed, it is possible to switch the tracking target in response to user operation. For example, by inputting up, down, left, or right using the direction indicating member of the operation unit 106, the tracking target can be moved to another adjacent subject detected in that direction, and the main subject can be switched sequentially in that direction. In this embodiment, subjects can be switched left and right (in the X-axis direction).

[0045] The control unit 201 selects one of the sorting methods in steps S403 to S405 based on the state determinations in steps S401 and S402, and executes sorting.

[0046] Next, each selection method will be explained using Figures 6 to 8. Here, the upper limit of the number of subjects that can be left after selection is set to six, and all of the remaining subjects are displayed with a dashed frame. The dashed frame is not necessarily displayed on the display unit 105, but is used for explanation purposes. Furthermore, among the selected subjects, the one that is particularly selected as the main subject is displayed with a solid frame.

[0047] FIG. 6 is a diagram explaining the AF area priority selection as the second selection method shown in step S403, and shows how the image obtained from the image sensor 104 is displayed on the display unit 105 during shooting, although as mentioned above, the dashed frame does not necessarily need to be displayed. The user sets the AF area as a detection area in advance. The AF area is an area setting that searches for the area determined to be most suitable for focusing within the area and performs focus detection within that area. The AF area can be selected to cover the entire image, or a smaller area can be specified. It can also be set to any size and position to suit the user's shooting environment.

[0048] When AF area priority selection is selected, the selection unit 212 sets the AF area as a candidate area and selects subjects detected within the candidate area as candidates for the main subject. If there are more subjects within the AF area than the upper limit (a predetermined number), subjects closer to the center of the AF area are detected with higher priority. Then, each of the detected subjects is evaluated using the aforementioned personal recognition and pose estimation, and a subject determined to be suitable as a main subject as a result of the evaluation (for example, a person registered in advance) is set as the main subject.

[0049] In this way, by selecting subjects based on the AF area, which is the user's area of ​​interest, it is possible to select a subject that is desirable for the user as the main subject when the tracking target has not been determined and the user wants to evaluate the detected subjects and select a suitable main subject.

[0050] FIG. 7 is a diagram illustrating the tracking target proximity priority selection as the first selection method shown in step S404. The selection unit 212 performs tracking target proximity priority selection when the imaging setting is set to perform tracking processing, tracking has been performed up to the previous frame, or a subject area has been specified by a user operation such as touch. In tracking target proximity priority selection, the distances between the tracking target 701 and other subjects are calculated, and areas within a predetermined distance range from the main subject are designated as candidate areas. Subjects present in these candidate areas are selected as candidates for the main subject. The distance is measured using Euclidean distance based on XY coordinates or the length of a vector obtained by orthogonally projecting a vector connecting two subjects onto the X or Y axis. In this embodiment, since the main subject can be switched to subjects located to the left or right of the main subject, the distance is calculated using the length of the vector orthogonally projected onto the X axis. However, when calculating the distance, the distances to both the left and right of the tracking target 701 are calculated, and the smaller of the two distances is used. Taking subject 702 as an example, the distance between tracking target 701 and subject 702 can be expressed as L3 or L1 + L2, with the shorter length L1 + L2 being the distance to subject 702. As a result, if a subject at the edge of the image is the tracking target, the subject at the other edge on the opposite side will be detected. As a result, it is possible to switch the tracking target from a subject at the edge of the image to a subject at the other edge. In addition, because the subject next to the tracking target is always detected, the user can select all detectable subjects as the main subject by repeating subject switching in one direction.

[0051] FIG. 8 is a diagram illustrating the focal plane vicinity priority selection as the third selection method shown in step S405. The selection unit 212 performs focal plane vicinity priority selection when the imaging settings allow the user to manually adjust the focus. FIG. 8(a) shows the initial state in which subject 801 is the main subject and the focus is adjusted to be in focus. That is, subject 801 is in focus, and a plane at the same depth as this subject is the focal plane. In the selection, the defocus amount of the imaging optical system is referenced as depth information, and an area where the defocus amount is smaller than a predetermined value is set as a candidate area, and subjects present in the candidate area are selected as candidates for the main subject. FIG. 8(b) shows the main subject detection result when the user manually operates the focus using the focus ring or the operation unit 106 to shift the focal plane toward the near side. As the main subject switches from subject 801 to subject 802, the selection result also changes, such that subject 803 is selected.

[0052] In this way, by preferentially adopting a subject that is closest to the focal plane and setting it as the main subject, it becomes possible to realize, for example, an auxiliary function for adjusting the focus position to the main subject.

[0053] The near-focus plane priority selection can be used in combination with other selection methods. For example, when AF area priority selection is selected, selection may be performed so that subjects that are extremely far from the focus plane are not detected.

[0054] As described above, in this embodiment, the subject detection system selects at least one from a plurality of selection methods depending on the camera state, such as the subject tracking state and the user's focus operation state, and detects subjects up to a predetermined number. This makes it possible to select a subject that matches the user's intention as the main subject while suppressing processing time, even when additional evaluation is performed on the detected subjects, particularly when the evaluation processing load is large.

[0055] (Other embodiments) The object of the present invention can also be achieved as follows: A storage medium storing software program code describing procedures for realizing the functions of each of the above-described embodiments is supplied to a system or device, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the program code stored in the storage medium.

[0056] In this case, the program code itself read from the storage medium will realize the novel functions of the present invention, and the storage medium storing the program code and the program will constitute the present invention.

[0057] Furthermore, examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, etc. Also usable are CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0058] The functions of the above-described embodiments are realized by making the computer executable the read program code. Furthermore, the functions of the above-described embodiments may be realized by an operating system (OS) or the like running on the computer performing some or all of the actual processing based on the instructions of the program code.

[0059] The following case is also included: First, program code is read from a storage medium and written into memory on an expansion board inserted into a computer or on an expansion unit connected to the computer. Then, based on the instructions of the program code, a CPU or other device on the expansion board or unit performs some or all of the actual processing.

[0060] The disclosure of this embodiment includes the following configurations and methods.

[0061] (Configuration 1) a detection means for detecting a plurality of subjects from a captured image; an area determining means for determining a candidate area for determining a main subject from the captured image; a main subject determining means for determining a main subject from the subjects detected by the detection means within the candidate region determined by the region determining means, The image processing device is characterized in that the area determining means determines the candidate area by different methods depending on an imaging setting for capturing the captured image.

[0062] (Configuration 2) a tracking means for tracking the main subject determined by the main subject determining means, The image processing device according to configuration 1, characterized in that, when the imaging setting is a setting for tracking a main subject determined by the main subject determination means, the area determination means determines an area within a predetermined distance range from the main subject as the candidate area.

[0063] (Configuration 3) 3. The image processing device according to configuration 1 or 2, further comprising a switching means for sequentially switching the main subject to a subject adjacent to the predetermined direction among the subjects detected by the detection means in response to a user operation instructing movement in the predetermined direction.

[0064] (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the switching means switches the main subject to the subject closest to the end of the image opposite the predetermined direction when the user operation instructs movement in the predetermined direction and there are no other subjects in the predetermined direction.

[0065] (Configuration 5) The image processing device according to any one of configurations 1 to 4, characterized in that when an edge of the captured image is within a predetermined distance range from the main subject, the area determination means determines an area within the predetermined distance range as a candidate area by combining the distance from the edge on the opposite side to the edge.

[0066] (Configuration 6) 6. The image processing device according to any one of configurations 1 to 5, wherein, when the imaging setting is a setting to autofocus on a subject within an area specified by a user operation that specifies an area, the area determination means sets the area specified by the user operation as a candidate area.

[0067] (Configuration 7) The image processing device according to any one of configurations 1 to 6, characterized in that, when the imaging setting is a setting in which the focus position can be manually changed by a user operation, the area determination means determines, as a candidate area, an area corresponding to a focus plane that is at a predetermined image plane distance from the focus plane of the main subject.

[0068] (Configuration 8) 8. The image processing device according to claim 1, wherein the main subject determining means evaluates a plurality of subjects within the candidate area, and determines the subject with the highest evaluation as the main subject.

[0069] (Configuration 9) 9. The image processing device according to configuration 8, wherein the main subject determining means assigns a higher evaluation to a subject that is closer to the center, closer to the subject, and of a type with a higher priority in the captured image.

[0070] (Configuration 10) 8. The image processing device according to any one of configurations 1 to 7, further comprising an imaging means for receiving a light beam via an imaging optical system and outputting the captured image.

[0071] (Configuration 11) 11. The image processing device according to configuration 10, further comprising a tracking means for tracking the main subject determined by the main subject determining means across a plurality of captured images.

[0072] (Method 1) a detection step of detecting a plurality of subjects from a captured image; a region determining step of determining a candidate region for determining a main subject from the captured image; a main subject determination step of determining a main subject from the subjects detected by the detection means within the candidate region determined in the region determination step, The method for controlling an image processing device, wherein the region determining step determines the candidate region by different methods depending on an imaging setting for capturing the captured image.

[0073] (Configuration 12) A computer-executable program that describes the steps of the method for controlling an image processing device according to Method 1.

[0074] (Configuration 13) A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an image processing device according to Method 1. [Explanation of symbols]

[0075] 100 digital single-lens camera 102 Arithmetic equipment 104 Image sensor 201 Control Unit 202 Main subject calculation section 211 Object Detector 212 Sorting Department 214 Detection result output unit 215 Evaluation Department 216 Main subject determination section

Claims

1. a detection means for detecting a plurality of subjects from a captured image; an area determining means for determining a candidate area for determining a main subject from the captured image; a main subject determining means for determining a main subject from the subjects detected by the detection means within the candidate region determined by the region determining means, The image processing device is characterized in that the area determining means determines the candidate area by different methods depending on an imaging setting for capturing the captured image.

2. a tracking means for tracking the main subject determined by the main subject determining means, 2. The image processing device according to claim 1, wherein when the imaging setting is a setting for tracking a main subject determined by the main subject determination means, the area determination means determines an area within a predetermined distance range from the main subject as the candidate area.

3. 3. The image processing device according to claim 2, further comprising a switching unit that can sequentially switch the main subject to a subject adjacent to the subject in a predetermined direction among the subjects detected by the detection unit in response to a user operation that instructs movement in the predetermined direction.

4. 4. The image processing device according to claim 3, wherein the switching means switches the main subject to the subject closest to the end of the image opposite the predetermined direction when the user operation instructs movement in the predetermined direction and there are no other subjects in the predetermined direction.

5. 5. The image processing device according to claim 4, wherein, when an edge of the captured image is within a predetermined distance range from the main subject, the area determination means determines an area within the predetermined distance range as a candidate area by combining the distance from the edge on the opposite side to the edge in question.

6. 2. The image processing device according to claim 1, wherein, when the imaging setting is a setting for autofocusing on a subject within an area specified by a user operation that specifies an area, the area determination means sets the area specified by the user operation as a candidate area.

7. 2. The image processing device according to claim 1, wherein, when the imaging setting is a setting in which the focus position can be manually changed by a user operation, the area determination means determines, as the candidate area, an area corresponding to a focus plane that is at a predetermined image plane distance from the focus plane of the main subject.

8. 8. The image processing apparatus according to claim 1, wherein the main subject determining means evaluates a plurality of subjects within the candidate area, and determines the subject with the highest evaluation as the main subject.

9. 9. The image processing device according to claim 8, wherein the main subject determining means assigns a higher evaluation to a subject that is closer to the center of the captured image, closer to the subject, and has a higher priority.

10. 8. The image processing apparatus according to claim 1, further comprising an imaging unit that receives a light beam through an imaging optical system and outputs the captured image.

11. 11. The image processing apparatus according to claim 10, further comprising a tracking unit that tracks the main subject determined by the main subject determining unit across a plurality of captured images.

12. a detection step of detecting a plurality of subjects from a captured image; a region determining step of determining a candidate region for determining a main subject from the captured image; a main subject determination step of determining a main subject from the subjects detected by the detection means within the candidate region determined in the region determination step, The method for controlling an image processing device, wherein the region determining step determines the candidate region by different methods depending on an imaging setting for capturing the captured image.

13. A computer-executable program that describes the steps of the method for controlling an image processing apparatus according to claim 12.

14. A computer-readable storage medium storing a program for causing a computer to execute each step of the method for controlling an image processing apparatus according to claim 12.

Images