Electronic devices, image processing methods, programs, and storage media

Abstract

When multiple subjects are adopting poses that strongly suggest they are the main subject, the goal is to suppress the decrease in responsiveness to changes in the main subject while preventing frequent changes in the main subject. [Solution] The system comprises: a first detection means for detecting one or more subjects from an image; a first determination means for determining a main subject from the one or more subjects detected by the first detection means based on posture information; a second determination means different from the first determination means for determining a main subject from the one or more subjects detected by the first detection means; and a control means that, when the main subject is determined by the first determination means, restricts the change of the main subject to other subjects more than when the main subject is determined by the second determination means.

Description

Technical Field

[0001] The present invention relates to an electronic device, an image processing method, a program, and a storage medium, and particularly to a subject detection technique in an electronic device.

Background Art

[0002] Conventionally, in an imaging device such as a digital camera, a technique has been proposed for detecting a subject from a captured image and performing imaging control such as focus adjustment and exposure control on the detected subject. In an imaging device equipped with such a technique, when a plurality of subjects are detected from a captured image, there is a device that has a function of assisting the user's imaging operation by automatically selecting a main subject from the detected subjects.

[0003] For example, Patent Document 1 discloses a technique for performing main subject selection by acquiring the pose information of a plurality of subjects detected from a captured image and calculating the degree of main subject likeness based on the pose information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional technique disclosed in Patent Document 1, when a plurality of subjects are in a pose state with a high degree of main subject likeness, the main subject likeness changes according to the change in the pose, so there is a possibility that the main subject frequently selected among the plurality of subjects will change. As a result, the main subject in the shooting scene may not be determined, and stable shooting may become impossible. On the other hand, simply restricting the change of the main subject in order to avoid this will reduce the responsiveness of changing the main subject to a certain subject when that subject is in a state with a high degree of main subject likeness.

[0006] This invention was made in view of the above-mentioned problems, and aims to prevent frequent changes in the main subject while suppressing a decrease in responsiveness to changes in the main subject when multiple subjects are taking poses that make them appear to be the main subject. [Means for solving the problem]

[0007] To achieve the above objective, the electronic device of the present invention comprises: a first detection means for detecting one or more subjects from an image; a first determination means for determining a main subject from the one or more subjects detected by the first detection means based on posture information; a second determination means different from the first determination means for determining a main subject from the one or more subjects detected by the first detection means; and a control means that, when a main subject is determined by the first determination means, restricts the change of the main subject to other subjects more than when a main subject is determined by the second determination means. [Effects of the Invention]

[0008] According to the present invention, when multiple subjects are taking poses that strongly resemble the main subject, it is possible to suppress a decrease in responsiveness to changes in the main subject while preventing frequent changes in the main subject. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example of the functional configuration of a digital camera system according to the first embodiment of the present invention. [Figure 2] A diagram showing an example of the correspondence between the exit pupil and the photoelectric conversion unit according to the first embodiment. [Figure 3] A schematic diagram showing an example of the configuration of the gaze detection unit according to the first embodiment. [Figure 4] This figure shows the sensitivity setting screen for subject change based on action detection according to the first embodiment. [Figure 5] A flowchart illustrating the procedures for the shooting preparation operation and continuous shooting operation according to the embodiment. [Figure 6](a) is a block diagram showing the functional configuration of the image processing unit in the main subject area setting process according to the first embodiment, and (b) is a diagram showing an example of integrated detection information. [Figure 7] A flowchart showing the procedure for setting the main subject area according to the first embodiment. [Figure 8] This figure shows an example of the results obtained by performing the subject tracking process, specific object detection process, and subject detection process according to the first embodiment. [Figure 9] A diagram showing an example of the action determination process according to the first embodiment. [Figure 10] A diagram illustrating the priority of subject feature regions according to the first embodiment. [Figure 11] A flowchart showing the procedure for determining the main subject area according to the first embodiment. [Figure 12] A diagram showing a specific example of the process for determining the main subject area according to the first embodiment. [Figure 13] A flowchart showing the procedure for determining the main subject area according to the second embodiment. [Figure 14] A flowchart showing the procedure for determining the main subject area related to the modified form. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in the case of being implemented as an interchangeable-lens digital camera. However, the present invention is applicable to any electronic device capable of mounting a subject detection function and an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, drive recorders, and the like. Note that these are examples, and the present invention is also applicable to other electronic devices. Further, the present invention is also applicable to a configuration in which the subject detection function and the imaging function are provided in separate devices (for example, a main body and a remote controller) that can communicate with each other.

[0012] <First Embodiment> FIG. 1, FIG. 2, and FIG. 3 are block diagrams showing functional configuration examples of a digital camera system as an example of an electronic device according to an embodiment of the present invention. The digital camera system includes a main body 100 of an interchangeable-lens digital camera and a lens unit 150 that is detachable from the main body 100. Note that the main body 100 and the lens unit 150 may be integrally configured.

[0013] The lens unit 150 has a communication terminal 6 that contacts a communication terminal 10 provided on the main body 100 when mounted on the main body 100. Power is supplied from the main body 100 to the lens unit 150 through the communication terminal 10 and the communication terminal 6. Further, the lens system control circuit 4 and the system control unit 50 of the main body 100 can communicate bidirectionally through the communication terminal 10 and the communication terminal 6.

[0014] In the lens unit 150, the lens group 103 is an imaging optical system composed of a plurality of lenses including a movable lens. The movable lens includes at least a focus lens. Further, depending on the lens unit 150, one or more of a zoom lens and a shake correction lens may be further included. The AF drive circuit 3 includes a motor, an actuator, etc. for driving the focus lens. The focus lens is driven when the lens system control circuit 4 controls the AF drive circuit 3. The aperture drive circuit 2 includes a motor or an actuator, etc. for driving the aperture 102. The aperture amount of the aperture 102 is adjusted when the lens system control circuit 4 controls the aperture drive circuit 2.

[0015] The mechanical shutter 101 is driven by the system control unit 50 to adjust the exposure time of the imaging element 22. Note that the mechanical shutter 101 is held in the fully open state during video shooting.

[0016] The imaging element 22 is, for example, a CCD image sensor or a CMOS image sensor. A plurality of pixels are two-dimensionally arranged in the imaging element 22, and each pixel is provided with one microlens, one color filter, and one or more photoelectric conversion parts. In the present embodiment, a plurality of photoelectric conversion parts are provided for each pixel, and signals can be read out for each photoelectric conversion part. By configuring the pixels in such a manner, a captured image and a parallax image pair can be generated from the signals read out from the imaging element 22.

[0017] FIG. 2(a) is a diagram schematically showing the correspondence relationship between the exit pupil of the lens unit 150 and each photoelectric conversion part when the pixel constituting the imaging element 22 has two photoelectric conversion parts.

[0018] The two photoelectric conversion parts 201a and 201b provided in the pixel share one color filter 252 and one microlens 251. Light that has passed through the partial region 253a of the exit pupil enters the photoelectric conversion part 201a, and light that has passed through the partial region 253b of the exit pupil enters the photoelectric conversion part 201b.

[0019] Therefore, for pixels within a given pixel region, the image formed by the signal read from the photoelectric conversion unit 201a and the image formed by the signal read from the photoelectric conversion unit 201b constitute a disparity image pair. This disparity image pair can be used as image signals (A image signal and B image signal) for phase-difference autofocus, and a normal image signal (captured image) can be obtained by adding the signals read from the photoelectric conversion units 201a and 201b for each pixel.

[0020] In this embodiment, each pixel of the image sensor 22 can be used as a pixel for generating a signal for phase-detection AF (focus detection pixel) or as a pixel for generating a normal image signal (imaging image pixel). Basic ) also functions as a focus detection sensor. However, some pixels of the image sensor 22 may be dedicated to focus detection, while other pixels are used for imaging. Figure 2(b) shows an example of the configuration of focus detection pixels and the region 253 of the exit pupil through which incident light passes. The focus detection pixels in the configuration shown in Figure 2(b) function similarly to the photoelectric conversion unit 201b in Figure 2(a). In practice, by distributing the focus detection pixels in the configuration shown in Figure 2(b) and another type of focus detection pixels that function similarly to the photoelectric conversion unit 201a in Figure 2(a) across the entire image sensor 22, it becomes possible to set a focus detection region of virtually any location and size.

[0021] The pixel configuration shown in Figure 2 uses the image sensor for obtaining recording images as a sensor for phase-detection autofocus (AF). However, the present invention is independent of the AF method as long as a focus detection area of ​​indeterminate size and position is available. For example, the present invention can be implemented even with a configuration using contrast AF. When using only contrast AF, each pixel has only one photoelectric conversion unit.

[0022] Returning to Figure 1, the A / D converter 23 is used to convert the analog image signal output from the image sensor 22 into a digital image signal (image data). The A / D converter 23 may also be provided by the image sensor 22.

[0023] The image data (RAW image data) output by the A / D converter 23 is processed by the image processing unit 24 as needed, and then stored in the memory 32 via the memory control unit 15. The memory 32 is used as a buffer memory to temporarily store image data and audio data, or as video memory for the display unit 28.

[0024] The image processing unit 24 applies predetermined image processing to image data to generate signals and image data, and acquire and / or generate various types of information. The image processing unit 24 may be a dedicated hardware circuit, such as an ASIC designed to perform a specific function, or it may be a configuration in which a processor, such as a DSP, executes software to perform a specific function.

[0025] The image processing applied by the image processing unit 24 includes preprocessing, color interpolation, correction, detection, data processing, and evaluation value calculation. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is the process of interpolating the values ​​of color components not included in the image data, and is also called demosaicing. Correction processing includes white balance adjustment, image brightness correction, optical aberration correction of the lens unit 150, and color correction.

[0026] The detection process is the process of detecting feature regions in an image. Feature regions are subject regions that can be detected by known techniques, such as the eyes, face, head, torso, upper body, lower body, and whole body regions of people and animals, as well as vehicle, scene-specific objects, arbitrary object regions, and regions with high consistency in pattern matching. Alternatively, only candidate feature regions may be detected. The detection process may also include action determination processes, such as estimating the posture of a detected person and determining whether that person is performing a specific action in a sport, such as a soccer shot or a volleyball spike.

[0027] Data processing includes scaling, encoding and decoding, and header information generation. Evaluation value calculation processing includes calculating a pair of image signals for phase-detection AF, evaluation values ​​for contrast AF, and evaluation values ​​used for automatic exposure control. These are examples of image processing that the image processing unit 24 can perform and do not limit the image processing performed by the image processing unit 24. Furthermore, the evaluation value calculation processing may be performed by the system control unit 50.

[0028] The D / A converter 19 generates an analog signal suitable for display on the display unit 28 from the display image data stored in the memory 32 and supplies it to the display unit 28. The display unit 28, for example, has a liquid crystal display device and displays information based on the analog signal from the D / A converter 19.

[0029] By repeatedly taking pictures and sequentially displaying each image obtained, the display unit 28 can be made to function as an electronic viewfinder (EVF). The video displayed to enable the display unit 28 to function as an EVF is called a live view image. The display unit 28 may be provided inside the main body 100 for observation through the eyepiece, or it may be provided on the surface of the housing of the main body 100 so that it can be observed without using the eyepiece. Alternatively, the display unit 28 may be provided both inside the main body 100 and on the surface of the housing.

[0030] The system control unit 50 is, for example, a CPU (also called an MPU or microprocessor). The system control unit 50 controls the operation of the main unit 100 and the lens unit 150 and realizes the functions of the camera system by reading programs stored in the non-volatile memory 56 into the system memory 52 and executing them. The system control unit 50 controls the operation of the lens unit 150 by sending various commands to the lens system control circuit 4 through communication terminals 10 and 6.

[0031] The non-volatile memory 56 may be rewritable. The non-volatile memory 56 stores programs executed by the system control unit 50, various settings of the camera system, GUI (Graphical User Interface) image data, etc. The system memory 52 is the main memory used by the system control unit 50 when executing programs.

[0032] As part of its operation, the system control unit 50 performs automatic exposure control (AE) processing based on the image processing unit 24 or evaluation values ​​it generates, and determines the shooting conditions. These shooting conditions, for example, for still image shooting, are shutter speed, aperture value, and ISO sensitivity. The system control unit 50 determines one or more of the shutter speed, aperture value, and ISO sensitivity according to the set AE mode. The system control unit 50 controls the aperture value (aperture) of the aperture mechanism of the lens unit 150. The system control unit 50 also controls the operation of the mechanical shutter 101.

[0033] Furthermore, the system control unit 50 drives the focus lens of the lens unit 150 based on the evaluation value or defocus amount generated by the image processing unit 24 or itself, and performs autofocus (AF) processing to bring the lens group 103 into focus on a subject within the focus detection area. The system timer 53 is an internal clock and is used by the system control unit 50.

[0034] The operation unit 70 has multiple input devices (buttons, switches, dials, etc.) that the user can operate. Some of the input devices on the operation unit 70 have names corresponding to their assigned functions. Note that the shutter button 61, mode selector switch 60, and power switch 72 are shown separately from the operation unit 70 for convenience, but are included in the operation unit 70. If the display unit 28 is a touch display, the touch panel is also included in the operation unit 70. The operation of the input devices included in the operation unit 70 is monitored by the system control unit 50. When the system control unit 50 detects an operation of an input device, it executes processing corresponding to the detected operation.

[0035] The shutter button 61 has a first shutter switch 62 (SW1) that turns ON when half-pressed and a second shutter switch 64 (SW2) that turns ON when fully pressed. When the system control unit 50 detects that SW1 is ON, it performs preparatory operations for still image shooting. These preparatory operations include AE ​​processing and AF processing. When the system control unit 50 detects that SW2 is ON, it performs still image shooting and recording operations according to the shooting conditions determined by the AE processing.

[0036] The mode switch 60 is an operation unit for switching settings in subject selection. Figure 4 shows an example of a setting screen for switching the sensitivity setting of the main subject selection using the action judgment process described later, via the mode switch 60. Three sensitivity settings are available: high, standard, and low.

[0037] Furthermore, the operation unit 70 of this embodiment includes a gaze detection unit 71 that detects the user's gaze direction. Although the gaze detection unit 71 is not a component that the user directly operates, it is included in the operation unit 70 because the direction of gaze detected by the gaze detection unit 71 is treated as input.

[0038] Figure 3(a) is a schematic side view showing an example of the configuration of the gaze detection unit 71 installed in the viewfinder. The gaze detection unit 71 detects the rotation angle of the optical axis of the user's eyeball 301a as the direction of the gaze when the user is looking at the display unit 28 installed inside the main body 100 through the viewfinder's eyepiece. Based on the detected direction of the gaze, the unit can identify the position that the user is fixating on in the display unit 28 (the point of fixation in the displayed image).

[0039] For example, a live view image is displayed on the display unit 28, and the user can see the contents of the display unit 28 through the eyepiece 71d and the dike stomachObservation can be made through the lock mirror 71c. The light source 71e can emit infrared light in the direction of the eyepiece window (outward from the main body 100). When the user is looking through the finder, the infrared light emitted by the light source 71e is reflected by the eyeball 301a and returns to the finder. The infrared light that enters the finder is reflected by the dichroic mirror 71c towards the light-receiving lens 71b.

[0040] The light-receiving lens 71b forms an image of the eyeball using infrared light on the imaging surface of the image sensor 71a. The image sensor 71a is a two-dimensional image sensor having a filter for infrared light imaging. The number of pixels of the image sensor 71a for gaze detection may be less than the number of pixels of the image sensor 22 for imaging. The eyeball image captured by the image sensor 71a is transmitted to the system control unit 50. The system control unit 50 detects the position of the corneal reflection of infrared light and the position of the pupil from the eyeball image, and detects the direction of gaze from the positional relationship between the two. The system control unit 50 also detects the position of the display unit 28 that the user is fixated on (the point of fixation in the display image) based on the detected direction of gaze. Alternatively, the position of the corneal reflection and the position of the pupil may be detected by the image processing unit 24 from the eyeball image, and the system control unit 50 may obtain these positions from the image processing unit 24.

[0041] The present invention does not depend on the method of gaze detection or the configuration of the gaze detection unit. Therefore, the configuration of the gaze detection unit 71 is not limited to that shown in Figure 3(a). For example, as shown in Figure 3(b), gaze detection may be performed based on a captured image obtained by a camera 71f positioned near the display unit 28 located on the back of the main unit 100. The field of view of the camera 71f, shown by the dotted line, is set so that the face 300 of the user who is taking a picture while looking at the display unit 28 is captured. The direction of the gaze can be detected based on the images of the eye regions 301a and 301b detected from the image captured by the camera 71f. When using infrared light images, a light source 71e can be placed near the camera 71f, and infrared light can be projected onto the subject within the field of view to perform the shooting. The method for detecting the direction of the gaze from the obtained image may be the same as the configuration in Figure 3(a). Also, when using visible light images, it is not necessary to project light. When using visible light images, the direction of the gaze can be detected from the positional relationship between the inner corner of the eye and the iris in the eye region.

[0042] Returning to Figure 1, the power control unit 80 consists of a battery detection circuit, a DC-DC converter, a switch circuit for switching which blocks are energized, etc., and detects whether a battery is installed, the type of battery, and the remaining battery level. The power control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, supplying the necessary voltage to each part, including the recording medium 200, for the required period. The power supply unit 30 consists of a battery, an AC adapter, etc.

[0043] I / F18 is an interface to a recording medium 200 such as a memory card or hard disk. The recording medium 200 stores data files such as captured images and audio. The data files stored on the recording medium 200 can be read through I / F18 and played back through the image processing unit 24 and the system control unit 50.

[0044] The communication unit 54 enables communication with external devices via at least one of wireless and wired communication. Images captured by the image sensor 22 (including live view images) and images recorded on the recording medium 200 can be transmitted to external devices through the communication unit 54. In addition, image data and other various information can be received from external devices through the communication unit 54.

[0045] The attitude detection unit 55 detects the attitude of the main body 100 with respect to the direction of gravity. The attitude detection unit 55 may be an acceleration sensor or an angular velocity sensor. The system control unit 50 can record orientation information corresponding to the attitude detected by the attitude detection unit 55 during shooting in a data file that stores the image data obtained during shooting. The orientation information can be used, for example, to display a recorded image in the same orientation as when it was taken.

[0046] In the digital camera system having the above configuration, the image processing unit 24 detects areas in the image captured by the image sensor 22 that are determined to match predetermined features using a known method, and outputs detection information such as the position, size, and confidence level of each feature area to the system control unit 50.

[0047] The system control unit 50, having acquired the detection information, selects the feature region that is most suitable as the main subject in that shooting scene from among the various feature regions, and designates it as the main subject region. For example, if multiple feature regions are detected when shooting a particular sports competition, the system control unit 50 selects the region of the subject that is determined by the action determination process to be performing an action important to the shooting of that competition, from among the subjects corresponding to the feature regions, as the main subject region.

[0048] The system control unit 50 then performs various shooting controls to ensure that the main subject area produces an appropriate image. Such controls include the following: First, autofocus adjustment (AF) to focus on the main subject area; automatic exposure control (AE) to ensure the main subject area is properly exposed; automatic white balance control to ensure the white balance of the main subject area is appropriate; and automatic flash intensity adjustment to ensure the brightness of the main subject area is appropriate. However, it is not limited to these. Furthermore, the system control unit 50 displays a rectangular frame corresponding to the main subject area on the shooting screen displayed on the display unit 28.

[0049] The present invention is independent of the type of feature region and detection method, and since known methods can be used for detecting feature regions, a description of the feature region detection method will be omitted. Furthermore, feature regions can also be used to detect subject information. When the feature region is a face region, subject information may include, but is not limited to, whether or not red-eye is present, whether or not the eyes are closed, and facial expressions (e.g., a smile).

[0050] Next, referring to the flowchart in Figure 5, the processing procedure for performing shooting preparation and continuous shooting operations while repeatedly adjusting focus and controlling exposure in a digital camera system having the above configuration will be explained.

[0051] Figure 5 is a flowchart showing the steps for preparing for shooting and continuous shooting after SW1 is turned ON. The processes from S501 to S506 correspond to one frame in continuous shooting, and continuous shooting is performed by repeating this series of processes.

[0052] In step S501, the system control unit 50 generates a live view image signal by accumulating charge in the image sensor 22 and outputs the generated image signal to the image processing unit 24 through the A / D converter 23. The image processing unit 24 generates the captured image, the image signal for phase-detection AF, and the image signal for AE based on the input image signal and stores them in the memory 32 through the memory control unit 15. These data stored in the memory 32 are read out and used as appropriate in each subsequent step.

[0053] In S502, the image processing unit 24 performs feature region detection processing on the captured image generated in S501, and simultaneously performs action determination processing on the detected feature regions. Next, the image processing unit 24 outputs the number of detected feature regions, information on each feature region, and the action determination results to the system control unit 50. Subsequently, the system control unit 50 sets the main subject region in the captured image based on the acquired feature region and action determination result information. Details of the feature region detection processing, action determination processing, and main subject region setting processing performed in S502 will be described later.

[0054] In S503, the system control unit 50 performs focus detection based on the image signal for phase-detection AF corresponding to the main subject area set in S502, and drives the lens group 103 based on the detected defocus amount to perform AF processing to focus on the subject within the main subject area. The system control unit 50 also performs automatic exposure calculation using a known method based on the image signal for AE obtained from the pixels of the main subject area, and determines the aperture value (AV value), shutter speed (TV value), and ISO sensitivity (ISO value) using a pre-stored program diagram. If the main subject area is not set in S502, the focus position and exposure value of the previous frame will be used.

[0055] In S504, the system control unit 50 detects the state of SW2. If SW2 is ON, the process proceeds to S505; otherwise, it proceeds to S506.

[0056] In S505, the system control unit 50 adjusts the aperture amount of the aperture 102 based on the aperture value determined in S503, and then drives the shutter 101 based on the shutter speed also determined in S503 to expose the image sensor 22 and generate a still image signal. The generated still image signal is given a gain according to the ISO sensitivity and then transmitted to the image processing unit 24. The image processing unit 24 generates a still image based on the received still image signal and outputs it to the system control unit 50. Subsequently, the system control unit 50 saves the received still image to the recording medium 200 and displays the image on the display unit 28.

[0057] In S506, the system control unit 50 detects the status of SW1 and SW2. If either SW1 or SW2 is ON, the process proceeds to S501. If both are OFF, the shooting preparation operation and continuous shooting operation are stopped.

[0058] The above describes the shooting preparation operation and continuous shooting operation procedures in this embodiment. Note that during continuous shooting, live view imaging (S501) may be omitted, and the processing of S502 and S503 may be performed using the still image captured in the previous frame. Alternatively, the processing of S501 to S503 may be repeated multiple times between still image captures during continuous shooting.

[0059] Next, the process of setting the main subject area, which is performed in S502, will be explained in detail. Figure 6(a) is a functional block diagram of the image processing unit 24, and Figure 7 is a flowchart showing the procedure for feature area detection and main subject selection processing. In the following explanation, a ball game played by multiple people will be used as an example scene, but the scenes to which this embodiment can be applied are not limited to this.

[0060] First, in S701, the system control unit 50 reads the captured image generated in S501 or S505 of the previous frame, and outputs the image signal of the read-out captured image to the image acquisition unit 601. The image acquisition unit 601 outputs the acquired image signal to the subject detection unit 602, the joint point detection unit 603, the specific object detection unit 606, and the subject tracking unit 607.

[0061] In S702, the subject tracking unit 607 reads the tracking template 609 and performs subject tracking processing on the captured image acquired in S701. Subject tracking processing is the process of detecting regions in the image that have features that closely match the tracking template as feature regions. Since the tracking template 609 contains feature information of the feature region corresponding to the main subject in the previous frame, the subject tracking processing makes it possible to detect the position and size of the main subject in the current frame in the previous frame. Note that any method may be used for subject tracking processing, and a known method is used here.

[0062] After subject tracking processing, the subject tracking unit 607 outputs the position, size, and tracking confidence of the detected feature region to the detection information integration unit 605. If the subject tracking processing fails to detect feature information in the current frame, a subject tracking loss flag is set to indicate that the tracked subject has been lost, and this is output to the detection information integration unit 605. Furthermore, if no feature information is set in the tracking template 609, the subject tracking processing is not executed, and the subject tracking unit 607 sets a no-tracking-subject flag and outputs this to the detection information integration unit 605.

[0063] In S703, the subject detection unit 602 performs subject detection processing on the captured image acquired in S701. Subject detection processing is the process of detecting a specific subject as a feature region within the image. Here, the target of detection is a person and various parts of the person, such as the eyes, face, head, upper body, and torso. Therefore, the position and size of the person in the image can be detected by subject detection processing. Any method may be used for subject detection processing, and a known method is used here. After the subject detection process, the subject detection unit 602 outputs the position, size, and detection confidence level of the detected feature region to the detection information integration unit 605.

[0064] Figure 9(a) shows how the head of subject 901 is detected as feature region 904 and the head of subject 902 is detected as feature region 905.

[0065] In S704, the unique object detection unit 606 performs unique object detection processing on the captured image acquired in S701. Unique object detection processing is the process of detecting objects unique to the scene as feature regions from within the image. Here, as an example of a scene-specific object, the ball in a ball game scene is used as the target for detection. Therefore, the unique object detection processing detects the position and size of the ball in the image. Any method may be used for unique object detection processing, and a known method is used here. After the unique object detection process, the unique object detection unit 606 outputs the position and size of the detected feature region to the action determination unit 604.

[0066] Figure 8(a) shows an example of the results of performing each detection process from S702 to S704 in a soccer shooting scene.

[0067] In the subject detection unit 602, the detection of the heads of subjects 801 and 802 results in subject feature regions 804 and 805. In addition, the detection of the ball 803 in the unique object detection unit 606 results in unique object region 807. Then, in the subject tracking unit 607, the detection using the template image data 808 shown in Figure 8(b), which corresponds to the tracking template 609, results in tracking feature region 806. Based on the physical features and clothing shapes and patterns included in the template image data 808, subject 801, which has a higher match in feature quantities, is detected.

[0068] In the example shown in Figure 8(a), the characteristic region of the subject is the head, but it is not limited to the head; other parts that can be identified as the subject may also be the face, eyes, whole body, upper body, lower body, torso, etc.

[0069] In S705, the joint point detection unit 603 performs joint point detection processing on the captured image acquired in S701. Joint point detection processing involves detecting the position of each joint point (predetermined part) of all subjects in the image and connecting the joint points to each other. Here, the joint points to be detected are set to a total of 10 points: the top of the person's head, neck, both elbows, both wrists, both knees, and both ankles. After detecting each joint point, the joint point detection unit 603 connects joint points that are presumed to belong to the same person. This set of 1 to 10 interconnected joint points constitutes the joint point information for one subject. Any method may be used for detecting and connecting joint points; known methods will be used here.

[0070] After the joint point detection process, the joint point detection unit 603 outputs position information and connection information for each joint point to the action determination unit 604.

[0071] In S706, the action determination unit 604 performs an action determination process using the joint point position information and connection information detected by the joint point detection unit 603, and the position and size of the unique object detected by the unique object detection unit 606. The action determination process determines whether the detected subject is performing an action that is important for filming in the sports competition being filmed. Actions that are important for filming include, for example, shooting, passing, and dribbling in soccer and basketball, and receiving, setting, and spiking in volleyball. Ku etc.These are specific actions that are often prioritized for filming in various sports competitions. Furthermore, important actions for filming also include defensive actions such as goalkeeper saves and sliding tackles in soccer. Hereafter, the state in which the subject is performing an important action for filming (a predetermined action state) will be referred to as the "action state." These important actions for filming are pre-associated with various sports competitions, and a machine learning model that stores the position information of the joint points and the position and size of the specific object corresponding to the action state is stored in non-volatile memory 56 or memory 32, etc. Then, at any point before filming begins, the sports competition can be selected using the operation unit 70, allowing the model necessary for action determination processing to be used. Alternatively, the position information of the joint points and the position and size of the specific object corresponding to the action state may be stored in non-volatile memory 56 or memory 32, etc., and this information may be used in the action determination processing.

[0072] In the action detection process, the action likelihood is calculated as an indicator to determine whether a subject is in an action state. The action likelihood is a measure of the probability that a subject is in an action state, and is calculated as a value between 0.000 and 1.000 based on the position of each joint point of each subject and the position and size of the intrinsic object. Subjects whose action likelihood is above a predetermined threshold are determined to be in an action state. The range of the action likelihood value can be set arbitrarily.

[0073] After the action determination process, the action determination unit 604 outputs the position information of each joint, the action likelihood, the action determination result, and the characteristic region information (position, size) of the unique object to the detection information integration unit 605.

[0074] Figure 9 shows an example of action detection processing for a subject 901 preparing to shoot and a subject 902 positioned as a defender in a soccer shooting scene.

[0075] Subject 901 is in an action state, as it is in a position to kick ball 903. On the other hand, subject 902 is standing at a distance from ball 903, and is therefore not in an action state.

[0076] Figure 9(b) shows the detection of 10 joint points 906 on the top of the head, neck, elbows, wrists, knees, and ankles of subject 901, and similarly, the detection of joint points 907 on subject 902. Region 908 is a characteristic region when the ball 903 is detected by the unique object detection unit 606.

[0077] In the action determination, subject 901 has a high action likelihood because the position of the corresponding joint point 906 indicates a posture of kicking a ball, and the ball's feature region 908 is detected near the ankle of joint point 906. On the other hand, subject 902 has a low action likelihood because the joint point 907 indicates a posture close to upright, and the distance between joint point 907 and the ball's feature region 908 is large. As a result, subject 901, corresponding to joint point 906, is determined to be in an action state, while subject 902, corresponding to joint point 907, is determined not to be in an action state.

[0078] Any method may be used to calculate the action likelihood. For example, a neural network, a machine learning technique, can be used to perform supervised learning, using the joint points and ball positions of the correct action state as training data. A specific example of this is the method described in Japanese Patent Application Publication No. 2021-071794. Alternatively, other machine learning techniques such as support vector machines or decision trees may be used, or a function that outputs confidence or probability values ​​based on a model may be constructed, not limited to machine learning. Furthermore, in this embodiment, the case where likelihood is used as the degree of probability that the subject is performing an action important for photography is described, but values ​​other than likelihood may be used. For example, the reciprocal of the distance between the center of gravity of the subject and the center of gravity of the eigenobject can be used as confidence.

[0079] Furthermore, if a particular subject is continuously or intermittently determined to be in an action state across multiple frames, corrections may be applied to increase the likelihood of that subject being in an action state. The reason for applying such corrections is that if a particular subject is continuously determined to be in an action state, it can be inferred that the user is framing that subject and that the subject is more likely than other subjects to continue to be in an action state.

[0080] In this embodiment, the main subject was determined using information on specific objects, but it is also possible to determine the main subject using only the joint point information of the subject. In addition, data that has undergone predetermined transformations, such as linear transformations, on the joint positions and the positions and sizes of specific objects may be used as input data.

[0081] In S707, the detection information integration unit 605 integrates the information acquired from the subject detection unit 602, the action determination unit 604, and the subject tracking unit 607 to generate integrated detection information 620. Figure 6(b) shows an example of integrated detection information 620. The integrated detection information 620 includes feature region information 621 for the same number of subjects detected by the subject detection unit 602, a subject feature region number 629 representing that number, and feature regions 622 of unique objects detected by the unique object detection unit 606.

[0082] The detection information integration unit 605 associates feature regions of a subject detected by the subject detection unit 602 with action determination results determined by the action determination unit 604, associating feature regions with action determination results that are estimated to correspond to the same subject. This association is performed, for example, by comparing each pair of joint points of the top of the head and neck that are connected with the head region or upper body region of each detected subject, and associating pairs of joint points that are in a positional relationship of a predetermined or greater distance with the feature regions of the subject. In Figure 9, feature region 904 and joint point 906, and feature region 905 and joint point 907 are associated as data related to the same subject. The associated data is then stored in feature region information 621. Feature region information 621 includes a subject detection flag 623 indicating that it is a feature region detected by the subject detection unit 602, an action likelihood 625 for each subject, and an action determination flag 626 indicating that it is in an action state.

[0083] Furthermore, the feature region information 621 includes a tracking subject flag 624 that indicates the target subject is a tracking subject. The detection information integration unit 605 compares the position and size of the tracking feature region obtained from the subject tracking unit 607 with the feature region information 621 of each subject, and if it is determined that there is an overlap of a predetermined percentage or more, it sets the tracking subject flag 624 included in the data of the feature region information 621 to TRUE. If none of the feature region information 621 satisfies the overlap condition with the tracking feature region, it adds one new independent feature region information 621, sets only the tracking subject flag 624 to TRUE, and adds 1 to the number of subject feature regions 629.

[0084] Furthermore, the integrated detection information 620 includes a tracked subject lost flag 627 set by the subject tracking unit 607, a no-tracked subject flag 628, and subject IDs 630 for identifying their respective feature regions. The subject IDs 630 are set so that for each subject detected from images of different frames, subjects that are estimated to be the same subject are assigned the same ID.

[0085] Furthermore, the integrated detection information 620 includes a subject change restriction timer 631. The subject change restriction timer 631 is set to a predetermined value each time the main subject area is determined in S708 (described later), provided that the action determination flag 626 of the feature area information 621 corresponding to the main subject area is TRUE. The subject change restriction timer 631 then counts down as time progresses if the value is greater than 0, and stops counting down when it reaches 0. The subject change restriction timer 631 is used in S708 to restrict changes to the main subject. The subject change limit timer 631 may be a numerical value representing the number of frames, or it may be set as real time.

[0086] Once the integration of all detection information is complete, the detection information integration unit 605 outputs the integrated detection information 620 to the system control unit 50.

[0087] In S708, the system control unit 50 determines the main subject area from the feature area information 621 based on the acquired integrated detection information 620. Details of the method for determining the main subject area will be described later.

[0088] In S709, the tracking template generation unit 608 acquires information on the main subject area determined by the system control unit 50, and updates the information in the tracking template 609 based on the acquired information on the main subject area. The updated tracking template 609 is then used by the subject tracking unit 607 to detect the feature area of ​​the main subject from the next frame. The above describes the procedure for setting the main subject area in this embodiment.

[0089] Next, the detailed procedure for the system control unit 50 to determine the main subject area, which takes place in S708, will be explained using the flowchart in Figure 11.

[0090] First, in S1101, the system control unit 50 acquires integrated detection information 620 from the detection information integration unit 605.

[0091] In S1102, the system control unit 50 classifies each subject feature region included in the acquired integrated detection information 620 into four categories 1001, as shown in the table in Figure 10, based on the corresponding tracking subject flag 624 and action determination flag 626. • Feature region that is in an action state and is a tracked subject → Action Tracked Subject • Feature region that is in action state but not a tracked subject → Action non-tracked subject • Feature region that is a tracked subject and not in an action state → Non-action tracked subject • Feature region that is not in an action state and is not a tracked subject → Non-action, non-tracked subject It is classified as such.

[0092] Then, action-tracking subjects are assigned a priority of +3 as the main subject, action-non-tracking subjects are assigned +2, non-action-tracking subjects are assigned +1, and non-action-non-tracking subjects are assigned 0. A higher number indicates a higher priority.

[0093] In S1103, the system control unit 50 checks the "no tracking subject" flag 628. If it is TRUE, it proceeds to S1109; otherwise, it proceeds to S1104.

[0094] In S1104, the system control unit 50 determines whether the tracked subject loss flag 627 is FALSE, that is, whether the main tracked subject was detected in the previous image. If the tracked subject loss flag 627 is FALSE (the tracked subject is detected), the process proceeds to S1105; if the tracked subject loss flag 627 is TRUE (the tracked subject is not detected), the process proceeds to S1106. Alternatively, the program may proceed to S1106 if the tracking subject loss flag 627 remains TRUE for a predetermined number of frames.

[0095] In S1105, the system control unit 50 counts down the subject change limit timer 631 according to the elapsed time since the previous frame. The subject change limit timer 631 was set to a predetermined value in S1111, which will be described later, in a previous frame.

[0096] In S1106, the system control unit 50 determines that the tracked subject has gone out of frame and clears the subject change limit timer 631 to zero.

[0097] In S1107, the system control unit 50 proceeds to S1108 if the subject change restriction timer 631 is greater than 0, assuming the timer has not expired; otherwise, it proceeds to S1109, assuming the timer has expired.

[0098] In S1108, the system control unit 50 determines the feature area in the feature area information 621 where the tracking subject flag 624 is TRUE as the main subject area. Here, since the subject change restriction timer 631 has not expired in S1107, it is decided not to change the main subject and to maintain the tracking subject from the previous frame as the main subject.

[0099] In S1109, the system control unit 50 determines the feature region with the highest priority set in S1102 from among the feature region information 621 as the main subject region. If multiple feature regions have the same priority, and they are all subjects in an action state, the feature region with the highest action likelihood 625 is selected as the main subject region. If feature regions with the same priority are all subjects in a non-action state, the feature region that is closer to the center of the image and has a larger area is selected as the main subject region based on the position and size of the feature regions. Note that even when selecting the main subject region from subjects in an action state, the selection may be based on the position and size of the feature regions. However, in any case, if the number of subject feature regions 629 is 0, the main subject region is not determined, and the process ends, returning to S709 in Figure 7.

[0100] In addition, since all feature area information 621 is considered when selecting the main subject area, if a tracked subject is selected from among them, the system will decide to maintain the tracked subject from past frames as the main subject. Therefore, the main subject is not necessarily changed during the processing in S1109.

[0101] In S1110, if the action determination flag 626 of the feature region information 621 determined as the main subject region in S1108 or S1109 is TRUE, the main subject is considered to be in an action state and the process proceeds to S1111. If it is FALSE, the process returns to S709 in Figure 7.

[0102] Next, in S1111, the system control unit 50 sets the subject change restriction timer 631 to a predetermined value. That is, if a subject in an action state is set as the main subject area, or if a subject being tracked from a past frame is in an action state in the current frame, the subject change restriction timer 631 is set. As a result, changing the main subject to another subject is restricted until the subject change restriction timer reaches 0. After completing the process in S1111, the process returns to S709 in Figure 7. Alternatively, the time set for the subject change limit timer 631 may be set to increase in the order of low, standard, and high, based on the sensitivity setting in Figure 4.

[0103] Figure 12 shows a specific example of the process for determining the main subject area shown in Figure 11, in a soccer shooting scene.

[0104] In scene 1210, the subject feature region 1214 of subject 1212 has been detected, while subject 1211, which is dribbling into the frame, and the ball 1213, which is an intrinsic object, have not been detected. Although subject 1212 was selected as the main subject region in the frame immediately preceding scene 1210, it is not in an action state as it is simply standing (priority = 1). In this scene 1210, the subject feature region 1214, which is the only subject feature region, is set as the main subject region (S1109), but since it is not in an action state, the subject change limit timer 631 is not set (FALSE in S1110).

[0105] Next, as time passes and the scene changes to 1220, the dribbling subject 1211 and the ball 1213 enter the shooting frame, and the subject feature region 1221 of subject 1211 and the intrinsic object region 1223 of ball 1213 are detected. Here, since subject 1211 is in a dribbling position and holding the ball 1213, the subject feature region 1221 is determined to be in an action state (priority = 2). On the other hand, the subject feature region 1214 for subject 1212 is still detected (FALSE in S1103), but it is not in an action state (priority = 1). Therefore, in this scene 1220, based on priority, the main subject region is changed from subject feature region 1214 to subject feature region 1221 (S1109). At the same time, in S1111, the subject change limit timer 631 is set to a predetermined value.

[0106] Next, as more time passes and the scene changes to 1230, subject 1212 assumes a sliding stance to steal the ball 1213, and subject feature region 1214 is determined to be in an action state (priority = 2). On the other hand, subject 1211 has stopped dribbling in order to keep the ball 1213, so subject feature region 1221 is less likely to be determined to be in an action state, but it is still detected as a tracked subject (priority = 1). However, at the time of scene 1230, the subject change limit timer 631 set in scene 1220 has not expired (YES in S1107), so subject feature region 1221 continues to be selected as the main subject region (S1108), and the main subject is not changed to subject feature region 1214.

[0107] The above is a detailed explanation of the procedure for determining the main subject area. With this configuration, once a subject in an action state is determined to be the main subject, even if other subjects in an action state are detected in subsequent frames, the subject will not be changed as long as the subject change restriction timer is active. Therefore, in shooting scenes where multiple subjects are simultaneously in an action state, it is possible to prevent frequent changes in the main subject area among those multiple subjects.

[0108] On the other hand, by disabling the subject change limit timer when changing subjects from a non-action subject to an action subject, highly responsive subject changes using action detection become possible in the fast-paced filming of sports competitions. Furthermore, by clearing the subject change limit timer to zero when the tracked subject is lost, it becomes possible to quickly change subjects when the tracked subject is moved out of frame and another action subject is framed.

[0109] Note that these methods for setting priorities and determining the main subject area are merely examples; any method that achieves the following two actions is acceptable. (1) The change of the main subject from a subject that is not in action to a subject that is in action should be done immediately. (2) Changing the main subject from one subject in an action state to another subject in an action state shall only be done if the specified conditions are met.

[0110] Furthermore, even if the main subject, which is in an action state, ceases to be in an action state, the rule that the main subject will only be changed if certain conditions are met may continue to be applied until a predetermined amount of time has elapsed from that point.

[0111] <Second Embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, other specific examples of means for realizing the two operations described above will be described. In this embodiment, a method for restricting the change of the main subject from one subject in an action state to another subject in an action state will be described, using means different from those of the first embodiment.

[0112] The difference between this embodiment and the first embodiment is that the determination of subject change restriction is performed by a method other than the method using the subject change restriction timer 631; otherwise, the configuration is the same. Therefore, the method of restricting the change of the main subject in this embodiment will be explained with reference to Figure 13.

[0113] Figure 13 is a flowchart showing the detailed procedure for the system control unit 50 to determine the main subject area in this embodiment. Steps that perform the same processing as described in Figure 11 in the first embodiment are given the same step numbers and their explanations are omitted.

[0114] In S1301, the system control unit 50 refers to the action determination flag 626 of the feature region information 621 corresponding to the tracked subject. If it is TRUE, it proceeds to S1302; if it is FALSE, it proceeds to S1109. The determination made here is made to restrict the change of the main subject if the tracked subject is in an action state.

[0115] In S1302, the system control unit 50 makes a decision on whether or not to suppress the change of the main subject, based on the information from the integrated detection information 620 and predetermined conditions for restricting the change of the main subject. This decision result is used in the branching step in S1303. The specific method of the decision will be described later.

[0116] In S1303, the system control unit, based on the decision result in S1302, proceeds to S1108 if it decides to suppress the change of the main subject, and to S1109 if it decides not to suppress it.

[0117] Below, we will specifically explain, using several methods, how to determine whether or not to suppress the change of the main subject in S1302. The methods described below may be implemented in addition to the subject change restriction determination by the subject change restriction timer 631 in S1104 to S1111 in Figure 11 of the first embodiment, or they may be partially or completely replaced. In the following explanation, a specific object may be represented as a ball and the subject area as a person. The flowchart in Figure 13 shows a case in which the subject change restriction determination by the subject change restriction timer 631 is completely replaced.

[0118] The following restriction conditions 1 to 13 indicate specific conditions for restricting subject changes in order to determine whether or not to suppress the change of subject in S1302. Restriction conditions 1 to 13 include conditions under which it is determined to suppress the change of subject if the condition is met, and conditions under which it is determined not to suppress the change of subject if the condition is met.

[0119] Constraint 1: If multiple subject feature regions in an action state are detected near the unique object feature region (within a predetermined distance range), and the tracked subject region is one of those subject feature regions, then it is decided to suppress subject changes. This is because, in sports competition scenes, if the subject is changed each time multiple people gather near the ball, subject changes could occur frequently.

[0120] Constraint 2: In a given sequence of frames, if the unique object feature region moves away from the tracked subject region and approaches another subject region, and that other subject is in an action state, the system will decide not to suppress the subject change; otherwise, it will decide to suppress it. This is to ensure a smooth change of the main subject in accordance with the movement of the ball when it is passed from one person to another.

[0121] Restriction condition 3: If a subject area that is in an action state is detected separately from the tracked subject, and the distance between that subject area and the intrinsic object area is closer than a predetermined value than the distance between any other subject area and the intrinsic object area, it is decided not to suppress subject changes; otherwise, it is decided to suppress them. This allows the system to stably continue to select the subject holding the ball as the primary subject in accordance with the ball being passed between subjects.

[0122] Restriction Condition 4: If a subject area that is in an action state is detected separately from the tracked subject, and the distance between that subject area and the intrinsic object area is greater than a predetermined distance, it is decided to suppress the subject change. This makes it possible to suppress the subject change from a subject located near the ball to a subject located far from the ball, and in accordance with the ball's movement, the subject holding the ball can be stably selected as the main subject.

[0123] Restriction Condition 5: If the user frames the other subject so that it is in the center of a predetermined area at a time that is within a predetermined timeframe of detecting a subject that is in an action state separate from the tracked subject, the system will decide not to suppress the subject change; otherwise, it will decide to suppress it. Here, the predetermined area is, for example, the shooting screen or the focus detection area. This allows for a smooth subject change when the subject to be photographed enters an action state and the user instantaneously frames that subject.

[0124] Restriction Condition 6: Using the user's gaze position information detected by the gaze detection unit 71, for example, if the system detects that the user's gaze is directed towards a subject in an action state at the same time that the subject in action state is detected, it is determined not to suppress the subject change; otherwise, it is determined to suppress it. This allows for instantaneous shooting of fast-moving and complex sports competition scenes. important When a scene occurs where a specific action is to be performed, the user can use the gaze detection function to quickly determine the main subject.

[0125] Restriction Condition 7: If the user directly sets the main subject using the touch display on the display unit 28, the system will decide to suppress subject changes for other subjects that are in an action state, even if other subjects are detected. This is because the subject directly specified by the user is the most important subject for the user to photograph, and restricting subject changes is more likely to align with the user's photographic intent.

[0126] Restriction Condition 8: If a subject area in an action state is detected separately from the tracked subject, but the subject detection unit 602 does not detect the face of that subject, it is decided to suppress subject change; if the face is detected, it is decided not to suppress it. This is because, from the user's perspective, a subject without a face is likely to be a subject with low suitability as the main subject. By suppressing subject change to a subject without a face, it becomes possible to select a subject that aligns with the user's shooting intentions.

[0127] Constraint 9: The system will suppress subject changes as long as the action likelihood of the tracked subject (625) does not fall below 90% of the threshold for determining that the subject is in an action state. Once it falls below 90%, the system will not suppress the change. This makes it possible to suppress frequent subject changes when multiple subjects are in an action state simultaneously. Note that the threshold may be multiplied by any other arbitrary percentage or by adding or subtracting a predetermined value, instead of 90%.

[0128] Restriction 10: In S707, the number of articulation points associated with the tracked subject is compared with the number of articulation points associated with a subject area other than the tracked subject. For subject areas with fewer articulation points than the tracked subject, it is determined to suppress subject changes. This is because, for the user, a subject hidden by other subjects is likely to be less suitable as the main subject. A low number of articulation points means that various parts of the subject's body are not visible, and suppressing subject changes to such subjects allows for subject selection that aligns with the user's shooting intentions. Furthermore, instead of using joint points to determine whether or not part of the subject is visible, various other methods may be used, such as methods for detecting parts of the subject.

[0129] Restriction Condition 11: If the tracked subject is located within a predetermined area set in the center of the screen, and another subject is located within a predetermined area set around the periphery of the screen, the system will decide to suppress the subject change to that subject. Conversely, if the tracked subject is located within a predetermined area set around the periphery of the screen, and another subject is located within a predetermined area set in the center of the screen, the system will decide not to suppress the subject change to that subject. This is because, for the user, a subject that is partially visible in the corner of the shooting screen is likely to be a subject with low suitability as the main subject. Therefore, suppressing the subject change to such subjects allows for subject selection that aligns with the user's shooting intentions.

[0130] Restriction Condition 12: If the tracked subject remains in an action state for a predetermined period of time or longer, the system will decide to suppress subject changes. This is because, for actions that continue for a relatively long period, such as dribbling in soccer, if subject changes are not suppressed, there is a possibility that the subject will frequently change to an opposing player vying for the ball. Therefore, by suppressing this, it becomes possible to stably continue to select a dribbling soccer player or similar as the main subject.

[0131] Restriction 13: If the subject selected as the main subject at the moment the user turns SW1 ON is also in an action state at the same time, it will be decided to suppress the subject change in subsequent frames. This is because it is presumed that the user started shooting in response to the subject's action, and therefore, not easily changing the main subject to another subject allows for subject selection that aligns with the user's shooting intentions.

[0132] Based on the above, S1302 determines whether or not to suppress subject changes using one or a combination of the methods described in restriction conditions 1 to 13. If a combination of multiple restriction conditions results in a conflict between the decision to suppress and the decision not to suppress, the final decision may be determined by setting a priority for each restriction method. By using the above restriction conditions to determine whether or not to suppress subject changes, it becomes possible to change subjects in accordance with the user's shooting intentions.

[0133] <Variation> Next, using the flowchart in Figure 14, we will explain an example of how to determine whether or not to suppress subject change by combining the subject change restriction timer 631 with some of the restriction conditions 1 to 13. Steps that perform the same processing as in Figure 11 described in the first embodiment and Figure 13 described in the second embodiment will be given the same step numbers and their explanations will be omitted.

[0134] In Figure 14, even if the subject change restriction timer 631 has not expired in S1107, the determination of the subject change restriction conditions in S1302 is made based on one of the restriction conditions 1 to 13 to decide whether to suppress the subject change. If it is not determined that the subject change should be suppressed, the process proceeds from S1303 to S1109, and the subject with the highest priority is selected as the main subject. On the other hand, if it is determined in S1303 that the subject change should be suppressed, the subject change is suppressed, and the process proceeds to S1108.

[0135] This allows for responsive subject changes according to the user's shooting intent, even if the subject change restriction timer has not expired, if the user decides that the main subject should be changed.

[0136] <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device.

[0137] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0138] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0139] 100: Main unit, 22: Image sensor, 50: System control unit, 24: Image processing unit, 28: Display unit, 32: Memory, 53: System timer, 56: Non-volatile memory, 60: Mode switching switch, 70: Operation unit, 71: Eye-line detection unit, 601: Image acquisition unit, 602: Subject detection unit, 603: Joint point detection unit, 604: Action determination unit, 606: Unique object detection unit, 607: Subject tracking unit, 620: Integrated detection information

Claims

1. A first detection means for detecting one or more subjects from an image, A first determination means that determines the main subject from one or more subjects detected by the first detection means based on posture information, A second determination means, different from the first determination means, determines the main subject from one or more subjects detected by the first detection means, When the main subject is determined by the first determination means, the control means restricts the change of the main subject to other subjects more than when the main subject is determined by the second determination means. An electronic device characterized by having the following features.

2. The electronic device according to claim 1, characterized in that the first determination means calculates the action likelihood of each subject based on the posture information and determines the main subject based on the likelihood.

3. The electronic device according to claim 2, characterized in that the first determination means determines that a subject whose likelihood is equal to or greater than a first threshold is in a predetermined operating state, sets a high priority for the subject, and determines the main subject based on the priority.

4. The electronic device according to claim 3, characterized in that the control means suppresses the change of the main subject when the main subject is in the predetermined operating state, as long as the likelihood is greater than or equal to a second threshold lower than the first threshold.

5. The electronic device according to claim 1, characterized in that the first determination means determines whether the subject is in a predetermined operating state based on posture information, and determines the subject determined to be in the operating state as the main subject.

6. It further has a second detection means for detecting an object, The electronic device according to claim 1, wherein the control means controls whether or not to suppress the change of the main subject based on the position of the object.

7. The electronic device according to claim 6, characterized in that when multiple subjects in a predetermined operating state are detected within a predetermined distance from the object, the control means suppresses the change of the main subject.

8. The electronic device according to claim 6, characterized in that when the object moves to approach a subject that is in a predetermined operating state different from the main subject, the control means does not suppress the change of the main subject to that subject.

9. The electronic device according to claim 1, characterized in that, when the main subject is determined by the first determination means, the control means controls the device so as not to change the main subject for a predetermined period of time.

10. The electronic device according to claim 9, characterized in that the aforementioned period is defined as the number of frames or real time.

11. The electronic device according to claim 9 or 10, characterized in that if the main subject is not detected during the period, the control means releases the restriction on changing the main subject.

12. The electronic device according to any one of claims 9 to 11, characterized in that when the main subject goes out of frame, the control means releases the restriction on changing the main subject.

13. The electronic device according to any one of claims 9 to 11, further comprising setting means for setting the sensitivity of the selection by the first determination means, and characterized in that the period is changed according to the sensitivity.

14. The electronic device according to claim 1, characterized in that the posture information includes positional information of multiple parts of the subject.

15. The electronic device according to claim 1, characterized in that the posture information includes joint point position information of the subject.

16. The electronic device according to claim 1, characterized in that the second determination means determines the main subject based on the tracking result based on the main subject in the immediately preceding frame.

17. The electronic device according to claim 1, characterized in that the second determination means determines the main subject based on the position and / or size of the subject within the image.

18. It further has a means for specifying the main subject, The electronic device according to claim 1, characterized in that the second determination means determines the subject designated by the designation means as the main subject.

19. The electronic device according to claim 1, characterized in that when a subject in a predetermined operating state different from the main subject is framed so as to be at the center of a predetermined area, the control means does not suppress the change of the main subject to that subject.

20. It further includes eye-tracking detection means, The electronic device according to claim 1, characterized in that when the gaze is directed towards a subject that is in a predetermined operating state different from the main subject, the control means does not suppress the change of the main subject to that subject.

21. The electronic device according to claim 1, characterized in that the control means suppresses changing the main subject to another subject if the face of another subject in a predetermined operating state is not detected.

22. The electronic device according to claim 1, wherein the control means suppresses the change of the main subject to a subject different from the main subject when the number of joint points associated with a subject different from the main subject is less than or equal to a predetermined number of joint points associated with the main subject.

23. The electronic device according to claim 1, wherein the control means suppresses the change of the main subject to a subject other than the main subject when the position of a subject other than the main subject is within a predetermined area in the peripheral part of the image.

24. The electronic device according to claim 1, characterized in that if the main subject remains in a predetermined operating state for a predetermined period of time, the control means suppresses the change of the main subject.

25. The electronic device according to claim 1, characterized in that, when the user has given instructions to prepare for shooting and the subject is in a predetermined operating state, the control means suppresses subsequent changes to the main subject.

26. A first detection step involves detecting one or more subjects from an image, A first determination step in which, based on posture information, a main subject is determined from one or more subjects detected in the first detection step, A second determination step, different from the first determination step, determines the main subject from one or more subjects detected in the first detection step, When the main subject is determined by the first determination step, a control step is performed to restrict the change of the main subject to other subjects more than when the main subject is determined by the second determination step. An image processing method characterized by comprising:

27. A program for causing a computer to function as one of the means of an electronic device according to any one of claims 1 to 25.

28. A computer-readable storage medium storing the program described in claim 27.

Images