Main subject determination device, imaging device, main subject determination method, and program

Abstract

To determine a main object that is highly likely to match the intention of a user in an image where a plurality of objects is present.SOLUTION: A main object determination device includes: image acquisition means that acquires images captured at different timings; selection means that selects main object candidate(s) from objects using information on feature points of the objects in the images; determination means that determines whether or not the main object candidate(s) each selected at different timings are the same using information on feature amounts calculated from the feature points; and input means that receives operation from a user. In a case where the determination means determines that the main object candidate(s) selected by the selection means in an image of interest and one or more images captured within a predetermined time before the image of interest is captured are the same, the determination means determines that the candidate is a main object. In a case where the input means receives an instruction to specify a new main object from the user, the determination means switches the main object according to the instruction.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technique for estimating a subject and determining a main subject based on the estimation result. [Background technology]

[0002] 2. Description of the Related Art Conventionally, various techniques have been proposed for detecting a subject to be controlled in order to perform imaging control such as autofocus (AF) in imaging devices such as digital cameras.

[0003] Patent Document 1 discloses a technology for recognizing actions of multiple people, in which multiple people are tracked simultaneously and the time-series data is input into a recurrent neural network to simultaneously estimate the type of action and the position of each person. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2018-538631 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 requires simultaneous tracking of multiple subjects and a recurrent neural network, which imposes a high processing load when implemented in hardware such as an imaging device.

[0006] The present invention has been made in consideration of this situation, and aims to provide a technology that can accurately determine a main subject that is likely to meet the user's intentions in an image containing multiple subjects, while reducing the processing load. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides: Photography Statue 1st image a subject detection means for detecting a person from the subject; a calculation means for acquiring posture information of the person estimated from the positional relationship of joints for each of the detected persons and calculating a reliability corresponding to the degree of possibility that the person is the main subject of the first image based on the posture information; a determination means for determining whether or not a candidate for the main subject exists in the first image based on the reliability; and when the determination means determines that a candidate for the main subject exists in the first image, determining whether or not the candidate for the main subject of the first image and the candidate for the main subject in a plurality of images captured within a predetermined number of frames before the first image are the same subject, and when they are determined to be the same subject in images of a predetermined number of frames or more, The relevant Main subject The image capturing apparatus has a determining means for determining the candidate as the main subject, and a switching means for switching the main subject in response to an instruction from a user to designate a new main subject when the instruction is received. The determining means does not determine whether the first image is the same subject or not when there is no candidate for the main subject in the first image. The present invention provides a main subject determination device characterized by the above.

[0009] Further, according to the present invention, Photography Statue 1st image a subject detection step of detecting a person from the image; a calculation step of acquiring posture information of the person estimated from the positional relationship of joints for each of the detected persons and calculating a reliability corresponding to the degree of possibility that the person is the main subject of the first image based on the posture information; a determination step of determining whether or not a candidate for the main subject in the first image exists based on the reliability; when it is determined by the determination step that a candidate for the main subject exists in the first image, determining whether or not the candidate for the main subject in the first image and the candidate for the main subject in a plurality of images captured within a predetermined number of frames before the first image are the same subject, and when it is determined that they are the same subject in images of a predetermined number of frames or more, The method has a determining step of determining the candidate as the main subject, and a switching step of switching the main subject in response to an instruction from a user to designate a new main subject when the instruction is received. In the determining step, if there is no candidate for the main subject in the first image, the determination of whether or not the subject is the same is not performed. The present invention provides a method for controlling a main subject determination device. [Effects of the Invention]

[0011] According to the present invention, it is possible to accurately determine a main subject that is likely to meet the user's intention in an image containing multiple subjects. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the configuration of an imaging apparatus 100 including a main subject determination device. [Figure 2] FIG. 2 is a block diagram showing a detailed configuration of an image processing unit 152 according to the first embodiment. [Figure 3] 6 is a flowchart of main subject determination processing according to the first embodiment. [Figure 4] 3 is a conceptual diagram of information acquired by the posture acquisition unit 203. [Figure 5] Examples in different frames of the image being processed. [Figure 6] 10 is a flowchart of a main subject determination process according to the third embodiment. [Figure 7] 10 shows examples of main subject candidates according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0014] [First embodiment] <<Overall Configuration of Imaging Device 100>> FIG. 1 is a block diagram showing the configuration of an imaging device 100 including a main subject determination device. The imaging device 100 may be, but is not limited to, a digital still camera or video camera that captures an image of a subject and records video or still image data on various media such as tape, solid-state memory, optical disk, or magnetic disk. For example, the imaging device 100 may be applied to any device that incorporates or externally connects an imaging device, such as a mobile phone (smartphone), a personal computer (laptop, desktop, tablet, etc.), a game console, an in-vehicle sensor, an FA (factory automation) device, a drone, or a medical device. Therefore, the term "imaging device" in this specification is intended to encompass any electronic device equipped with an imaging function. Furthermore, the term "main subject determination device" in this specification is intended to encompass any electronic device that determines a main subject based on an image captured by the imaging device.

[0015] In the following, an example will be described in which the subject is a person. The main subject refers to a subject that is the target of imaging control as intended by the user. Note that the configuration shown in FIG. 1 is merely one example of the configuration of imaging device 100.

[0016] Each unit in the imaging device 100 is connected via a bus 160. Furthermore, each unit is controlled by a main control unit 151.

[0017] The lens unit 101 is an imaging optical system including a fixed first-group lens 102, a zoom lens 111, an aperture 103, a fixed third-group lens 121, and a focus lens 131. The aperture control unit 105 adjusts the aperture diameter of the aperture 103 and controls the amount of light during imaging by driving the aperture 103 via an aperture motor 104 (AM) in accordance with commands from the main control unit 151. The zoom control unit 113 changes the focal length by driving the zoom lens 111 via a zoom motor 112 (ZM). The focus control unit 133 determines the drive amount for driving the focus motor 132 (FM) based on the amount of deviation in the focus direction of the lens unit 101. Additionally, the focus control unit 133 controls the focus adjustment state by driving the focus lens 131 via the focus motor 132 (FM). AF control is achieved by controlling the movement of the focus lens 131 by the focus control unit 133 and the focus motor 132. The focus lens 131 is a lens for adjusting focus, and although it is simply shown as a single lens in FIG. 1, it is usually made up of multiple lenses.

[0018] The subject image formed on the image sensor 141 via the lens unit 101 is converted into an electrical signal by the image sensor 141. The image sensor 141 is a photoelectric conversion element that photoelectrically converts the subject image (optical image) into an electrical signal. The image sensor 141 has m pixels in the horizontal direction and n pixels in the vertical direction light receiving elements arranged. The image formed on the image sensor 141 and photoelectrically converted is arranged as an image signal (image data) by the image signal processing unit 142. This makes it possible to acquire an image on the imaging surface.

[0019] The image data output from the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in a RAM (random access memory) 154. The image data stored in the RAM 154 is compressed by an image compression / decompression unit 153 and then recorded on an image recording medium 157. In parallel with this, the image data stored in the RAM 154 is sent to the image processing unit 152.

[0020] The image processing unit 152 applies predetermined image processing to the image data stored in the RAM 154. The image processing applied by the image processing unit 152 includes, but is not limited to, development processes such as white balance adjustment, color interpolation (demosaic), and gamma correction, as well as signal format conversion and scaling. In this embodiment, the image processing unit 152 selects a main subject candidate based on position information (e.g., joint position) of the subject. The image processing unit 152 may use the results of the main subject candidate selection process for other image processing (e.g., white balance adjustment). Furthermore, the image processing unit 152 determines whether main subject candidates selected at different times are the same subject. The image processing unit 152 stores the processed image data, the posture information of each subject, and position information of the center of gravity, face, and eye of the main subject candidate in the RAM 154. The image processing unit 152 also has a tracking unit (not shown) that can track a subject or a specific area between images, such as images during live view.

[0021] The operation unit 156 is an input interface including buttons, etc. By the user selecting various function icons displayed on the display unit 150, various operations can be performed on the imaging device 100, such as changing the shooting mode and switching the method of subject determination processing, which will be described later.

[0022] The main control unit 151 has one or more programmable processors, such as a CPU or MPU. The main control unit 151 controls each unit of the imaging device 100 and realizes the functions of the imaging device 100 by loading a program stored in, for example, a flash memory 155 into a RAM 154 and executing the program. The main control unit 151 also executes AE processing that automatically determines exposure conditions (shutter speed or accumulation time, aperture value, and sensitivity) based on information about the brightness of the subject. The information about the brightness of the subject can be acquired from, for example, the image processing unit 152. The main control unit 151 can also determine exposure conditions based on the area of ​​a specific subject, such as a person's face.

[0023] A focus control unit 133 performs AF control for the position of the main subject stored in RAM 154. An aperture control unit 105 performs exposure control using the brightness value of a specific subject area.

[0024] Display unit 150 displays images, main subject detection results, etc. Battery 159 is appropriately managed by power management unit 158 ​​and provides a stable power supply to the entire imaging device 100. Flash memory 155 stores control programs necessary for the operation of imaging device 100, parameters used for the operation of each unit, etc. When imaging device 100 is started up by a user operation (when the imaging device 100 transitions from a power-off state to a power-on state), the control programs and parameters stored in flash memory 155 are loaded into a part of RAM 154. Main control unit 151 controls the operation of imaging device 100 in accordance with the control programs and constants loaded into RAM 154.

[0025] <<Main subject detection process>> The main subject determination process executed by image processing unit 152 will be described with reference to Figures 2 and 3. Figure 2 is a block diagram showing a portion of the detailed configuration of image processing unit 152. Figure 3 is a flowchart of the main subject determination process. Unless otherwise specified, the processing of each step in this flowchart is realized by each unit of image processing unit 152 operating under the control of main control unit 151. In the following, a shooting scene in which multiple people play sports will be described as the target of the main subject determination process, but the shooting scene to which this embodiment can be applied is not limited to this.

[0026] In S301, the image acquisition unit 201 acquires, from the imaging control unit 143, the image captured in the Nth frame.

[0027] In S302, the posture acquisition unit 202 detects a subject (person) in the image acquired by the image acquisition unit 201, estimates the posture of the detected subject, and acquires posture information. The posture information is acquired by acquiring "joint positions" (described later) for the detected subject.

[0028] The method of detecting the subject and joint positions performed by the pose acquisition unit 202 may use a trained model such as a convolutional neural network (CNN) trained by machine learning. When detecting a subject using a trained model, the pose acquisition unit 202 can detect the subject using dictionary data for subject detection generated by machine learning. Different dictionary data may be used for each specific subject, such as dictionary data for "people" and dictionary data for "animals." The pose acquisition unit 202 detects the subject using the dictionary data, and changes the content of subsequent pose estimation depending on which dictionary data was used to detect the subject. For example, if subject detection is completed using dictionary data for "people," pose estimation is performed to correspond to "people."

[0029] When the posture acquisition unit 202 completes detection of the subject, the posture acquisition unit 202 starts estimating the posture of the subject depending on the type of the detected subject. Here, as an example, a case will be described where the detected subject is a person. The posture acquisition unit 202 first acquires the positions of multiple joints of the person as the subject as feature points. Then, the posture of the subject is estimated based on the acquired information on the joint positions. Any method may be used for posture estimation, such as a method that utilizes deep learning.

[0030] Note that any trained model other than trained CNN may be used for the subject detection method and joint position detection method performed by the pose acquisition unit 202. For example, a trained model generated by machine learning such as a support vector machine or a decision tree may be applied to the pose acquisition unit 202. Furthermore, the pose acquisition unit 202 does not need to use a trained model generated by machine learning. For example, any subject detection method and joint position detection method that do not use machine learning may be applied to the pose acquisition unit 202.

[0031] In S303, the selection unit 203 calculates the probability that each subject is the main subject based on the posture information.

[0032] In S304, the selection unit 203 determines whether there are any subjects that are candidates for the main subject, and if there are, proceeds to S305. A method for calculating the probability representing the likelihood of being the main subject and a specific method for selecting main subject candidates will be described later. If there are no subjects, proceeds to S310.

[0033] In S305, determination unit 204 refers to information in RAM 154 and determines whether or not there are any main subject candidates in the images of frames N through N-1, which were captured at a timing different from frame N. If there are any main subject candidates, the process proceeds to S306; if there are no main subject candidates, the process proceeds to S309. If N=1 (first frame), there are no previous frames, so the process proceeds to S309 after processing S305.

[0034] In S306, information on one or more main subject candidates is stored in RAM 154, and the process proceeds to S307.

[0035] In S307, determination unit 204 performs matching (identical subject determination) with the main subject candidate detected in some frames temporally close to frame N among the images of frames N to N-1. If the matching result indicates that the main subject is the same, proceed to S308; otherwise, proceed to S309.

[0036] Here, when the frame rate is f, M is preferably adjusted so that M / f is an appropriate time for the shooting scene and the subject. For example, when a subject of interest in a sports scene such as soccer makes a shooting motion, the time (grace period) from the preparatory movement to the moment of the shot (the shutter chance) is generally about 2 to 3 seconds. Therefore, it is preferable that the value of M is set so that the same subject determination is completed before the shutter chance, and M / f should be shorter than the grace period. Note that, since the grace period until the shutter chance differs depending on the shooting scene and the subject, the determination unit 204 may be configured to switch the value of M as needed. Alternatively, the value of M may be configured so that the user can determine it in advance. Furthermore, information on frames prior to the NM frame is not used in the processing of S307.

[0037] In S308, the subject that the determination unit 204 has determined to be the same subject is determined to be the main subject, and in S309, this is stored in the RAM 154 as history information of the main subject.

[0038] In S310, the image processing unit 152 updates frame N to N+1, and in S311 determines whether all frames have been processed, and if not, returns to S301.

[0039] <<Posture acquisition section and selection section>> 4A and 4B are conceptual diagrams of information acquired by the posture acquisition unit 202. Fig. 4A shows an image to be processed, in which a subject 401 is about to kick a ball 403. The subject 401 is an important subject in the shooting scene.

[0040] In this embodiment, the selection unit 203 uses the posture information of the subject acquired by the posture acquisition unit 202 to determine a subject (main subject) that is likely to be intended by the user as a target for imaging control, monitoring (gazing), etc. On the other hand, the subject 402 is a non-main subject. Here, the non-main subject refers to a subject other than the main subject.

[0041] FIG. 4(b) is a diagram showing an example of posture information of subjects 401 and 402. Joints 411 represent each joint of subject 401, and joints 412 represent each joint of subject 402. FIG. 4(b) shows an example in which positions corresponding to the top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles are acquired as joints (feature points), but some of these joint positions may be acquired, or other positions may be acquired. Furthermore, posture estimation may use not only joint positions but also information such as axes connecting joints.

[0042] In the following, a case where joint positions are acquired as posture information will be described.

[0043] In S302 of FIG. 3, the posture acquisition unit 202 acquires two-dimensional coordinates (x, y) of the joints 411 and 412 in the image. Here, the unit of (x, y) is pixels. The posture acquisition unit 202 estimates the posture of the subject based on the acquired coordinate information of the joints. Specifically, the positional relationship between the positions of the joints is grasped based on the acquired coordinate information of each joint, and posture information estimated from the positional relationship is acquired. Any trained model other than a trained CNN may be used to estimate the posture of the subject. For example, a trained model generated by machine learning such as a support vector machine or a decision tree may be applied to the posture acquisition unit 202. Furthermore, the posture acquisition unit 202 does not need to use a trained model generated by machine learning. For example, any posture estimation method that does not use machine learning may be applied to the posture acquisition unit 202.

[0044] Returning to the explanation of FIG. 3, in S303, the selection unit 203 calculates a reliability (probability) representing the likelihood of each subject being the main subject, based on the joint coordinates and posture information acquired by the posture acquisition unit 202. The probability can be calculated using machine learning techniques such as neural networks, support vector machines, and decision trees. In addition to machine learning, a function that outputs a reliability or probability value based on a certain model may also be constructed. The learned weights and bias values ​​are saved in advance in the flash memory 155, and are stored in the RAM 154 as needed.

[0045] The selection unit 203 may calculate the reliability using data obtained by performing a predetermined transformation such as a linear transformation on the coordinate data of the joints acquired by the posture acquisition unit 202. In this case, the posture acquisition unit 202 or the selection unit 203 may perform a predetermined transformation on the coordinate data of the joints.

[0046] In this embodiment, a case will be described in which the probability that a subject is the main subject of the image to be processed is used as the confidence level representing the likelihood of the subject being the main subject (confidence level corresponding to the degree of possibility that the subject is the main subject of the image to be processed), but values ​​other than probability may also be used. For example, the confidence level can be the inverse of the distance between the center of gravity of the subject and the center of gravity of an important object in the scene, such as a ball. For example, in a soccer shooting scene, the subject that the user is focusing on (the person taking the shot) is expected to be close to the soccer ball, which is an important object, so the inverse can be used to calculate the confidence level.

[0047] In S304, the selection unit 203 selects the subject with the highest probability from among the detected subjects (people) as a main subject candidate. Then, in S306, the selection unit 203 stores the coordinates of the joints of the main subject candidate and representative coordinates representing the main subject candidate (such as the center of gravity position or face position) in RAM 154. This completes the selection process. Note that the process of S304 may be performed on the determination unit 204 side.

[0048] In the above explanation, a case where a main subject candidate is selected using posture information of a single frame has been described. However, a configuration may be adopted in which successive frames or videos are read and the main subject is determined by calculating the probability using time-series posture information. When time-series posture information is used, information on the joint positions (feature points) at each time may be used, or joint position information at a certain time may be used in combination with information on the motion vectors of the joints or the subject (feature amounts calculated from feature points). Any other information may be used as long as it represents time-series information.

[0049] When the selection unit 203 is trained to calculate the reliability (probability), it can be trained to use the state (preparatory state) of the main subject before moving on to an important action (action related to an event to be recorded, detected, or monitored).

[0050] For example, in the case of kicking a ball, the state of swinging one's leg up to kick the ball can be learned as one state of the main subject. The reason for adopting this configuration is that when a subject that should actually be the main subject performs an important action, the subject must be accurately determined as the main subject and the control of the imaging device 100 must be executed in accordance with the main subject. Furthermore, when the reliability (probability value) corresponding to the main subject exceeds a preset threshold, control for automatically recording images or videos (recording control) is initiated, allowing the user to capture important moments (photo opportunities) without missing them. In this case, information on the typical time (leave time until the photo opportunity) from the state to be learned until the important action occurs may be used to control the imaging device 100. In other words, when the main control unit 151 detects an important action from the subject, control may be performed so that AF, exposure, etc. are completed and the actual shooting operation (release of the shutter) is performed after a typical time corresponding to the detected important action.

[0051] <<Judgment part>> FIG. 5 shows examples of different frames of the image to be processed. FIG. 5(a) shows an image (image of interest) of a frame of interest (frame of interest), and FIG. 5(b) shows an image M frames before the frame of interest. When the selection unit 203 selects subjects 501 and 503, the determination unit 204 calculates the distance between the center of gravity 505 of subject 501 and the center of gravity 506 of subject 503. If the distance is less than a predetermined threshold, the subjects are considered to be the same. This is because, if the matching time M / f [sec], where f is the frame rate [fps], is sufficiently short, the distance the subjects move during that time is expected to be limited. Note that in FIG. 5, the center of gravity positions 505 and 506 are shown as a figure combining a circle and intersecting line segments (the intersection of the line segments is considered to be the center of gravity) for ease of understanding. The actual center of gravity positions are calculated as points or areas on the coordinate system that can be calculated from the positions of the joints in the two-dimensional coordinate system described above.

[0052] Other methods that can determine whether a subject is the same include template matching using the color or brightness histogram of an image, or matching using partial joint information. Generally, a subject whose posture information has been detected is less likely to be occluded. Therefore, it is possible to achieve high matching accuracy with a simple method.

[0053] As described above, according to the first embodiment, image capture device 100 acquires posture information for each of a plurality of subjects detected from an image to be processed, and selects a main subject candidate from among the plurality of subjects based on the posture information for each of the plurality of subjects. Then, image capture device 100 determines the main subject by determining whether the main subject candidates detected in frames within a predetermined time period are identical.

[0054] This makes it possible to determine the main subject that is most likely to meet the user's intention in an image containing multiple subjects.

[0055] In addition, by performing matching only once in the main subject determination process, the processing load can be reduced, and by performing same subject determination using information from two or more frames in the main subject determination process, the accuracy of main subject detection can be further improved.

[0056] <<Display section>> The image displayed on display unit 150 and the detection result of the main subject will be described.

[0057] After the above-described main subject determination is performed, display unit 150 may display an image in which a marker, a frame, or the like is superimposed on the determined main subject based on instructions from main control unit 151. Note that the superimposed display of a marker, a frame, or the like may be performed not only on the main subject, but also on main subject candidates and tracking subjects described below. In this case, the color, thickness, shape, or the like of the markers or frames may be changed so that the main subject candidates can be distinguished from the determined main subject and tracking subject.

[0058] For example, a thick frame may be displayed superimposed on the main subject, and a thin frame may be displayed on the main subject candidate. The display method is not limited to this example, and any display that is distinguishable by the user may be used. In addition, to avoid cluttering the display, markers may be displayed only on those corresponding to the main subject, or if the main subject and the tracking subject are the same, a marker indicating either the main subject or the tracking subject may be superimposed.

[0059] The display of the marker or frame does not need to wait until the main subject determination is complete, but may start from the point when a main subject candidate is detected in the image. On the other hand, if there are no main subject candidates, main subject, or tracking subject in the image, the superimposed display may not be displayed. Also, the user may be able to turn the superimposed display ON / OFF as needed.

[0060] <<Tracking unit>> The tracking section of the image processing section 152 will now be described.

[0061] The tracking unit uses the subject detection results to identify the image area (subject area) to be tracked. For example, it extracts features from the subject area of ​​an image in a certain base frame, and uses the extracted features to search for an area that is highly similar to the subject area of ​​the base frame within each image that is supplied one after another, as the subject area. Note that the base frame may be replaced with a new frame in chronological order at any timing. In this case, the features extracted from the subject area searched for in the new base frame are used as the features to be used for tracking the next frame and thereafter.

[0062] Methods for searching for an area based on image features include template matching, histogram matching, and the Kanade-Lucas-Tomasi Feature Tracker (KLT). Note that other methods may be used as long as they can search for an object based on its features. In addition to the above, a CNN for object tracking may be trained, and different frames of images may be input to the CNN, which then outputs the image area to be tracked.

[0063] The tracking section of the image processing section 152 performs tracking processing on the images acquired by the image acquisition section 201 based on instructions from the main control section 151. Information about the subject detected by the orientation acquisition section 202 may be used to identify the image area to be tracked. Alternatively, the tracking section may detect the subject area. When tracking begins, the tracking section stores information about the subject being tracked in RAM 154 for each image (frame) supplied one after another as history information of the tracked subject.

[0064] As a means for determining the subject to be tracked, the user may set in advance the subject to be prioritized for tracking. For example, the mode for the subject to be prioritized for tracking can be set via the display unit 150. Alternatively, the result of the main subject determination can be used. For example, it is possible to set the subject to be tracked that has been determined to be the main subject.

[0065] [Second embodiment] In the second embodiment, a modification of the main subject determination process in the first embodiment will be described.

[0066] In the second embodiment, the basic configuration of the imaging device 100 is the same as that of the first embodiment (see FIG. 1). Below, differences from the first embodiment will be mainly described.

[0067] In S307 of FIG. 3, the determination unit 204 performs matching not only with the frame closest to N but also with all main subject candidates N-M to N-1 recorded in the RAM 154, and if it is determined that they are the same subject, the process proceeds to S308, and if not, the process proceeds to S309.

[0068] In this way, by performing matching with subject candidates in all of the past M frames, even if candidate B is detected during the detection of candidate A, as shown in FIG. 7, candidate A can be determined.

[0069] As described above, according to the second embodiment, even if a different person is detected as a main subject candidate, it is less likely that a past main subject candidate will be overlooked, and the accuracy of detecting the main subject can be further improved.

[0070] [Third embodiment] In the third embodiment, a case will be described in which the main subject determination process in the first and second embodiments and subject tracking process are performed simultaneously.

[0071] In the third embodiment, the basic configuration of the imaging device 100 is the same as in the first and second embodiments (see FIG. 1). Below, differences from the first embodiment will be mainly described.

[0072] 6 is a flowchart of this embodiment. In S601, the posture acquisition unit 202 detects, in the Nth frame, the same subject (tracked subject) that the tracking unit of the image processing unit 152 has been tracking up to the N-1th frame.

[0073] 3, and is the main subject determination process described in embodiment 1. In S610, it is determined whether or not a main subject has been determined in S602, and if a main subject has been determined, in S603 the tracking subject is changed to the main subject determined in S610. Note that if the main subject determined in S610 and the tracking subject are the same, the tracking subject is not changed.

[0074] As described above, according to the third embodiment, even while the tracking process is being used, it is possible to determine a main subject that is likely to meet the user's intention from among multiple subjects, and to track that subject.

[0075] In the present embodiment, the image sensor 141 of the image capturing device 100 is fixed to the main body, and an example has been shown in which the subject is tracked within the same angle of view. However, this is not limited to this, including the other embodiments, and the image capturing device 100 may have a pan / tilt / zoom drive mechanism and may track the subject while performing at least one of panning, tilting, and zooming in accordance with the movement of the subject.

[0076] [Fourth embodiment] In the fourth embodiment, a modification of the main subject determination process in the first to third embodiments will be described.

[0077] In the fourth embodiment, the basic configuration of the imaging device 100 is the same as in the first to third embodiments (see FIG. 1). Below, differences from the first embodiment will be mainly described.

[0078] When determination unit 204 determines the main subject in S308, the main subject may change to a different subject from the main subject determined in the immediately preceding frame. In this case, if the new main subject is different from the subject intended by the user, it is preferable to be able to cancel or reset the selection of the main subject. Therefore, in this embodiment, the user can cancel or reset the selection of the main subject by performing a predetermined operation on operation switch 156 (including a touch panel, gesture detector, etc.).

[0079] To cancel or redetermine the selection of the main subject, when the user performs a predetermined operation on operation switch 156, image processing unit 152 refers to the main subject history stored in RAM 154 in S309. The main subject history to be referred to may be the main subject history for the frame immediately before or most recent than the time the predetermined operation was performed (Nth frame), or all of the main subject history recorded from N L to N-1th frames. Here, L is a number determined independently of M described above.

[0080] Determination unit 204 matches all subject candidates in the current frame of interest with the main subject history in the frame before the main subject changed in the referenced main subject history. If determination unit 204 determines, as a result of matching, that the subject candidate is the same subject as the main subject in the main subject history, determination unit 204 determines the subject candidate in the frame of interest as the new main subject. This makes it possible to deselect and re-determine the main subject by referring to the previous main subject history and making an identity determination even in situations where the main subject has changed.

[0081] Furthermore, the tracking subject history can also be used to deselect or reset the main subject. When the user performs a predetermined operation on operation switch 156, image processing unit 152 references the tracking subject history stored in RAM 154. The tracking subject history to reference is preferably the tracking subject history for the most recent frame or the frame immediately before the predetermined operation was performed. This is because there is a high possibility that the subject that was being tracked before the main subject was switched is the subject that the user intended to photograph.

[0082] The determination unit 204 performs matching between the tracking subject history in the frame before the main subject changed and the subject candidate in the current frame of interest. If the determination unit 204 determines as a result of matching that the subject candidate is the same subject as the tracking subject in the tracking subject history, the determination unit 204 determines that subject candidate in the frame of interest as a new main subject. The tracking unit also determines that subject as the tracking target. This makes it possible to deselect and re-determine the main subject by referring to the previous tracking subject history and making an identity determination even in situations where the main subject has changed.

[0083] Note that the deselection and redetermination of the main subject using the main subject history and the deselection and redetermination of the main subject using the tracking subject history may be performed using either one of them, or the main subject may be determined using both, and the main subject may be determined by having image processing unit 152 evaluate which has higher reliability.

[0084] As a method of canceling the selection of the main subject, when the user performs a predetermined operation on operation switch 156, determination unit 204 may not perform main subject determination processing on the image captured at the time the predetermined operation was performed, but may proceed with processing on the next captured image. Alternatively, when the user performs a predetermined operation, all main subject history and tracking subject history recorded in RAM 154 up to that point may be erased, and new main subject determination may be started. In this case, the influence of past main subject and tracking subject history can be reduced.

[0085] As a predetermined operation of operation switch 156, the user may select (designate) the intended subject by touching a touch panel (attached to display unit 150) while checking display unit 150. In this case, the touched subject may be subjected to the above-described main subject redetermining process. Furthermore, if the touched subject is not included in the main subject history, determination unit 204 may determine that subject as the main subject. This allows a newly appearing main subject candidate to be immediately selected as the main subject.

[0086] In the photographic scene assumed in this embodiment, it is expected that the time it takes for the main subject candidate to transition from the preparatory operation to the moment of the photo opportunity will be short. In other words, the user may not always have enough time to select the subject by touch. In that case, operation switch 156 may be provided separately as a button for redeterminating the main subject. This may allow the user to quickly cancel the selection of the main subject candidate and issue a command to redeterminate it. In this case, the predetermined operation refers to operating the redetermination button. Alternatively, this may be linked to the operation of the shutter button (not shown) of imaging device 100. For example, the main subject may be canceled and redeterminated when the shutter button is released from its half-pressed state (photography preparation state).

[0087] Furthermore, when the above-mentioned re-determination or selection is performed, main control unit 151 may perform control so that the marker or frame that was superimposed and displayed as the main subject on display unit 150 is replaced with another subject.

[0088] Note that there may be cases where the subject the user did not want to be the main subject is not detected when the user performs a predetermined operation. This may occur when the subject moves off the screen or is hidden behind another subject or an obstacle. In such cases, the main subject determined in the frame when the user performed the predetermined operation is maintained, and processing proceeds to the next frame. In this case, the superimposed display on the main subject on display unit 150 will be maintained as the main subject determined in the frame when the user performed the predetermined operation.

[0089] As described above, according to the fourth embodiment, when a main subject different from the user's intention is determined, the user can cancel the determination and reset the main subject by performing an operation.

[0090] [Fifth embodiment] In the fifth embodiment, a modified example of the main subject determination process in the first to fourth embodiments will be described. In this embodiment, the main subject determination is performed by combining evaluation of the probability values ​​indicating the likelihood of a plurality of subjects being the main subject with matching across a plurality of frames. This makes it possible to increase the accuracy of main subject determination while reducing the number of missed main subject candidates when there are multiple subjects with a high probability of being the main subject, such as in a sports contest.

[0091] In the fifth embodiment, the basic configuration of the imaging device 100 is the same as that in the first embodiment (see FIG. 1). The following description will be mainly based on the first embodiment.

[0092] In this embodiment, the same processes as those in the first embodiment are performed for S301 to S303 in FIG.

[0093] In S304, the selection unit 204 selects, as main subject candidates, not only the subject having the highest probability of being the main subject, but also subjects having a difference from the maximum probability that is smaller than a predetermined value.

[0094] In S305, determination unit 204 refers to the information in RAM 154 and determines whether or not a main subject candidate exists in the images within frames NM to N-1. If a main subject candidate exists, the process proceeds to S306, and if not, the process proceeds to S309. In S306, the same process as in the first embodiment is performed.

[0095] In S307, matching is performed between all main subject candidates stored in RAM 154 and frames N-1. If it is determined that they are the same subject, the process proceeds to S308; if not, the process proceeds to S309. In addition, in the first to third embodiments, the condition for the same subject determination in S307 is that the same subject must be captured in the frame of interest and one other frame. However, the condition may also be that the same subject must be matched in the frame of interest and two or more other frames. The method of storing multiple main subject candidates and the method of tightening the conditions for same subject determination may be performed simultaneously, or either one may be performed. By tightening the conditions for same subject determination, it is possible to prevent a decrease in the accuracy of same subject determination even in situations where there are many main subject candidates.

[0096] As described above, according to the fifth embodiment, when there are multiple subjects in the screen that are likely to be the main subject, it is possible to improve the accuracy of subject selection while reducing the number of main subject candidates that are missed.

[0097] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0098] 100 Imaging device 141 Image sensor 151 Main control unit 152 Image processing section 201 Image acquisition unit 202 Attitude acquisition section 203 Selection Section 204 Judgment section

Claims

1. A subject detection means for detecting a person from a captured first image; a calculation means for acquiring posture information of each of the detected persons estimated from a positional relationship of joints, and calculating a reliability corresponding to a degree of possibility that the detected persons are a main subject of the first image based on the posture information; a determination means for determining whether or not a candidate for a main subject exists in the first image based on the reliability; a determining means for determining whether or not the candidate main subject of the first image and the candidate main subject of a plurality of images captured within a predetermined number of frames prior to the first image are the same subject when the determining means determines that a candidate main subject in the first image exists, and for determining that the candidate main subject is the main subject when the candidate main subject is determined to be the same subject in images of a predetermined number of frames or more; a switching means for switching the main subject in response to a user's instruction to specify a new main subject when the instruction is received, The main subject determination device according to claim 1, wherein the determining means does not determine whether or not the main subject is the same subject when there is no candidate for the main subject in the first image.

2. The main subject determination device of claim 1, characterized in that the determination means selects candidates for the main subject based on the reliability of each of the multiple persons as well as the distance between the person detected by the subject detection means and the ball.

3. A tracking means for tracking a subject, 3. The main subject determination device according to claim 1, wherein the tracking means switches a tracking target to the main subject when the determining means determines that the main subject candidates are the same.

4. 2. The main subject determination device according to claim 1, wherein when there is an input from a user specifying a main subject, the main subject specified by said input has priority over the main subject determined by said determination means.

5. 2. The main subject determination device according to claim 1, wherein when a cancellation input for canceling the main subject determined by said determination means is received from a user, said main subject is determined to be the subject that was the main subject before said determination means determined the main subject.

6. a display control means for displaying the image on a display unit; The main subject determination device according to claim 3, characterized in that the display control means superimposes a marker or a frame on the image for at least one of the main subject candidates, the main subject, or the tracking target detected in the image.

7. 7. The main subject determination device according to claim 6, wherein the display control means changes the color, thickness and shape of the marker or frame depending on the candidate for the main subject, the main subject or the target to be tracked in the image.

8. an imaging means for capturing an object image formed via an imaging optical system; An imaging apparatus comprising the main subject determination device according to claim 1 .

9. A subject detection step of detecting a person from a captured first image; a calculation step of acquiring posture information of each of the detected persons estimated from a positional relationship of joints, and calculating a reliability corresponding to a degree of possibility that the detected persons are a main subject of the first image based on the posture information; a determining step of determining whether or not a candidate for a main subject exists in the first image based on the reliability; a determination step of determining whether or not the candidate main subject in the first image and the candidate main subject in a plurality of images captured within a predetermined number of frames prior to the first image are the same subject when it is determined by the determination step that a candidate main subject in the first image exists, and determining that the candidate main subject is the main subject when it is determined that the candidate main subject is the same subject in images of a predetermined number of frames or more; a switching step of switching the main subject in response to a user's instruction to designate a new main subject when the instruction is received, A method for controlling a main subject determination device, wherein in the determining step, if there is no candidate for the main subject in the first image, a determination as to whether or not they are the same subject is not performed.

10. A program for causing a computer to function as each of the means of the main subject determination apparatus according to any one of claims 1 to 7.

Images