Electronic apparatus, image processing method, program, and storage medium

Abstract

To prevent a main subject from frequently changing while suppressing the change of the main subject from decreasing in response when a plurality of subjects take positions where they highly seem to be the main subject.SOLUTION: An electronic apparatus has: first detection means which detects one or more subjects from images obtained through repetitive photography; determination means which determines whether each of the subject detected by the first detection means is in an operation state; selection means which selects one of the subjects detected by the first detection means as a main subject; and control means which limits changing of the main subject in subsequent images when the subject determined by the determination means to be in the operation state is selected by the selection means as the main subject.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an electronic device, an image processing method, a program, and a storage medium, and more particularly to a subject detection technique in an electronic device. [Background technology]

[0002] Conventionally, in imaging devices such as digital cameras, techniques have been proposed for detecting a subject from a captured image and performing imaging control such as focus adjustment and exposure control for the detected subject. Some imaging devices equipped with such techniques have a function to assist a user in imaging operations by automatically selecting a main subject from among the detected subjects when multiple subjects are detected from the captured image.

[0003] For example, Patent Document 1 discloses a technique for selecting a main subject by acquiring posture information of a plurality of subjects detected from a captured image and calculating the likelihood of the subjects being the main subject based on the posture information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-071794 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the conventional technology disclosed in Patent Document 1, when multiple subjects are in a posture that is highly likely to be the main subject, the likelihood of the main subject changing depending on the posture, which can result in the main subject being frequently selected among the multiple subjects changing. As a result, the main subject in the shooting scene may not be determined, making it difficult to take stable photos. On the other hand, simply restricting changes to the main subject to avoid this would result in a decrease in responsiveness to changing the main subject to a certain subject when that subject is in a state that is highly likely to be the main subject.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to prevent frequent changes of the main subject while suppressing a decrease in responsiveness to changes of the main subject when multiple subjects are assuming postures that are highly likely to be the main subject. [Means for solving the problem]

[0007] In order to achieve the above object, the electronic device of the present invention comprises a first detection means for detecting one or more subjects from images obtained by repeated photographing, a determination means for determining whether each subject detected by the first detection means is in a predetermined operating state, a selection means for selecting one of the subjects detected by the first detection means as a main subject, and a control means for restricting changes to the main subject in subsequent images when a subject determined to be in the operating state by the determination means is selected as the main subject by the selection means. [Effects of the Invention]

[0008] According to the present invention, when multiple subjects are assuming postures that are highly likely to be the main subject, it is possible to prevent frequent changes of the main subject while suppressing a decrease in responsiveness to changes of the main subject. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of the functional arrangement of a digital camera system according to a first embodiment of the present invention. [Figure 2] FIG. 3 is a diagram showing an example of the correspondence relationship between an exit pupil and a photoelectric conversion unit according to the first embodiment. [Figure 3] FIG. 2 is a schematic diagram showing an example of the configuration of a gaze detection unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a sensitivity setting screen for changing a subject by action determination according to the first embodiment. [Figure 5] 4 is a flowchart showing the procedure of a shooting preparation operation and a continuous shooting operation according to the embodiment. [Figure 6] FIG. 4A is a block diagram showing the functional configuration of an image processing unit in the process of setting a main subject region according to the first embodiment, and FIG. 4B is a diagram showing an example of integrated detection information. [Figure 7] 6 is a flowchart showing the procedure of a main subject region setting process according to the first embodiment. [Figure 8] 5A to 5C are views showing examples of results of subject tracking processing, unique object detection processing, and subject detection processing according to the first embodiment. [Figure 9] FIG. 4 is a diagram showing an example of an action determination process according to the first embodiment. [Figure 10] 5A and 5B are views for explaining the priority of subject feature regions according to the first embodiment. [Figure 11] 6 is a flowchart showing the procedure of a main subject region determination process according to the first embodiment. [Figure 12] 5A to 5C are views showing a specific example of a process for determining a main subject region according to the first embodiment. [Figure 13] 10 is a flowchart showing the procedure of a main subject region determination process according to the second embodiment. [Figure 14] 10 is a flowchart showing the procedure of a main subject region determination process according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following embodiments, the present invention will be described with reference to an interchangeable-lens digital camera. However, the present invention can be applied to any electronic device capable of incorporating both a subject detection function and an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can be applied to other electronic devices. The present invention can also be applied to a configuration in which the subject detection function and the imaging function are provided in separate devices capable of communicating with each other (for example, a main unit and a remote controller).

[0012] First Embodiment 1, 2, and 3 are block diagrams showing an example of the functional configuration of a digital camera system as an example of an electronic device according to an embodiment of the present invention. The digital camera system has a body 100 of an interchangeable lens digital camera and a lens unit 150 that is detachable from the body 100. The body 100 and the lens unit 150 may be configured as an integrated unit.

[0013] The lens unit 150 has a communication terminal 6 that comes into contact with a communication terminal 10 provided on the main body 100 when attached to the main body 100. Power is supplied from the main body 100 to the lens unit 150 through the communication terminal 10 and the communication terminal 6. Furthermore, the lens system control circuit 4 and the system control unit 50 of the main body 100 can communicate bidirectionally through the communication terminal 10 and the communication terminal 6.

[0014] In the lens unit 150, the lens group 103 is an imaging optical system composed of multiple lenses including a movable lens. The movable lenses include at least a focus lens. Depending on the lens unit 150, one or more of a variable magnification lens and an image stabilization lens may also be included. The AF drive circuit 3 includes a motor, actuator, etc. that drive the focus lens. The focus lens is driven by the lens system control circuit 4 controlling the AF drive circuit 3. The iris drive circuit 2 includes a motor actuator, etc. that drive the iris 102. The opening amount of the iris 102 is adjusted by the lens system control circuit 4 controlling the iris drive circuit 2.

[0015] The mechanical shutter 101 is driven by the system control unit 50 and adjusts the exposure time of the image sensor 22. The mechanical shutter 101 is kept fully open during video shooting.

[0016] The image sensor 22 is, for example, a CCD image sensor or a CMOS image sensor. The image sensor 22 has a plurality of pixels arranged two-dimensionally, and each pixel has one microlens, one color filter, and one or more photoelectric conversion units. In this embodiment, each pixel has multiple photoelectric conversion units, and is configured to be able to read out signals for each photoelectric conversion unit. By configuring the pixels in this way, it is possible to generate a captured image and a pair of parallax images from signals read out from the image sensor 22.

[0017] FIG. 2(a) is a diagram schematically showing the correspondence between the exit pupil of the lens unit 150 and each photoelectric conversion unit when a pixel constituting the image sensor 22 has two photoelectric conversion units.

[0018] Two photoelectric conversion units 201a and 201b provided in a pixel share one color filter 252 and one microlens 251. Light that has passed through a partial region 253a of the exit pupil is incident on the photoelectric conversion unit 201a, and light that has passed through a partial region 253b of the exit pupil is incident on the photoelectric conversion unit 201b.

[0019] Therefore, for a pixel included in a certain pixel region, an image formed by the signal read out from the photoelectric conversion unit 201a and an image formed by the signal read out from the photoelectric conversion unit 201b form a parallax image pair. The parallax image pair can be used as image signals (image A signal and image B signal) for phase-difference AF, and a normal image signal (captured image) can be obtained by adding the signals read out from the photoelectric conversion units 201a and 201b for each pixel.

[0020] In this embodiment, each pixel of the image sensor 22 functions both as a pixel for generating a signal for phase-difference AF (focus detection pixel) and as a pixel for generating a normal image signal (image for capturing). However, some pixels of the image sensor 22 may be dedicated for focus detection, while other pixels are used as imaging pixels. FIG. 2(b) shows an example of the configuration of focus detection pixels and an example of an exit pupil region 253 through which incident light passes. The focus detection pixel configured as shown in FIG. 2(b) functions in the same manner as the photoelectric conversion unit 201b in FIG. 2(a). In practice, by distributing the focus detection pixel configured as shown in FIG. 2(b) and another type of focus detection pixel that functions in the same manner as the photoelectric conversion unit 201a in FIG. 2(a) throughout the image sensor 22, it becomes possible to set a focus detection region of virtually any location and size.

[0021] The pixel having the configuration shown in Figure 2 is configured to use the image sensor for obtaining an image for recording as a sensor for phase-difference AF, but the present invention does not depend on the AF method as long as a focus detection area of variable size and position can be used. For example, the present invention can be implemented even with a configuration that uses contrast AF. When using only contrast AF, each pixel has one photoelectric conversion unit.

[0022] 1, the A / D converter 23 is used to convert the analog image signal output from the imaging element 22 into a digital image signal (image data). Note that the imaging element 22 may include the A / D converter 23.

[0023] The image data (RAW image data) output by the A / D converter 23 is processed by the image processing unit 24 as necessary, and then stored in the memory 32 via the memory control unit 15. The memory 32 is used as a buffer memory for temporarily storing image data and audio data, and as a video memory for the display unit 28.

[0024] The image processing unit 24 applies predetermined image processing to the image data to generate signals and image data, and acquire and / or generate various types of information. The image processing unit 24 may be a dedicated hardware circuit such as an ASIC designed to achieve a specific function, or may be configured to achieve a specific function by a processor such as a DSP executing software.

[0025] The image processing applied by the image processing unit 24 includes preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, etc. Preprocessing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process of interpolating values of color components not included in image data, and is also called demosaicing. Correction processes include white balance adjustment, image brightness correction, correction of optical aberrations of the lens unit 150, color correction, etc.

[0026] Detection processing is a process of detecting feature regions in an image. Feature regions are subject regions that can be detected using known techniques, such as the pupils, face, head, torso, upper body, lower body, and whole body regions of a person or animal, vehicles, scene-specific objects, arbitrary object regions, and regions with high consistency in pattern matching. Alternatively, only feature region candidates may be detected. Detection processing also includes action determination processing, which estimates the posture of a detected person and determines whether the person is performing a specific action in a sporting event, such as shooting a soccer ball or spiking a volleyball.

[0027] The data processing includes scaling, encoding and decoding, header information generation, etc. The evaluation value calculation process includes calculation of a pair of image signals for phase difference AF, an evaluation value for contrast AF, an evaluation value used for automatic exposure control, etc. Note that these are examples of image processing that can be performed by the image processing unit 24 and do not limit the image processing performed by the image processing unit 24. The evaluation value calculation process may also be performed by the system control unit 50.

[0028] The D / A converter 19 generates an analog signal suitable for display on the display unit 28 from the image data for display stored in the memory 32, and supplies the signal to the display unit 28. The display unit 28 has, for example, a liquid crystal display device, and performs display based on the analog signal from the D / A converter 19.

[0029] By repeatedly capturing images and sequentially displaying each image obtained, display unit 28 can function as an electronic viewfinder (EVF). The moving image displayed to enable display unit 28 to function as an EVF is called a live view image. Display unit 28 may be provided inside main body 100 so that it can be viewed through an eyepiece, or it may be provided on the surface of the housing of main body 100 so that it can be viewed without using an eyepiece. Alternatively, display unit 28 may be provided both inside main body 100 and on the surface of the housing.

[0030] The system control unit 50 is, for example, a CPU (also called an MPU or microprocessor). The system control unit 50 loads a program stored in a non-volatile memory 56 into the system memory 52 and executes it to control the operation of the main body 100 and the lens unit 150 and realize the functions of the camera system. The system control unit 50 controls the operation of the lens unit 150 by sending various commands to the lens system control circuit 4 via communication terminals 10 and 6.

[0031] The nonvolatile memory 56 may be rewritable. The nonvolatile memory 56 stores programs executed by the system control unit 50, various setting values of the camera system, image data of a GUI (Graphical User Interface), etc. The system memory 52 is a main memory used by the system control unit 50 when executing programs.

[0032] As part of its operation, the system control unit 50 performs automatic exposure control (AE) processing based on evaluation values generated by the image processing unit 24 or by itself, and determines the shooting conditions. For example, the shooting conditions for still image shooting are shutter speed, aperture value, and ISO sensitivity. The system control unit 50 determines one or more of the shutter speed, aperture value, and ISO sensitivity depending on the AE mode that is set. The system control unit 50 controls the aperture value (opening size) of the aperture mechanism of the lens unit 150. The system control unit 50 also controls the operation of the mechanical shutter 101.

[0033] In addition, the system control unit 50 drives the focus lens of the lens unit 150 based on the evaluation value or defocus amount generated by the image processing unit 24 or itself, and performs automatic focus adjustment (AF) processing to focus the lens group 103 on the subject within the focus detection area. The system timer 53 is an internal clock that is used by the system control unit 50 .

[0034] The operation unit 70 has multiple input devices (buttons, switches, dials, etc.) that can be operated by the user. Some of the input devices of the operation unit 70 have names corresponding to the functions assigned to them. For convenience, the shutter button 61, mode selector switch 60, and power switch 72 are illustrated separately from the operation unit 70, but are included in the operation unit 70. If the display unit 28 is a touch display, the touch panel is also included in the operation unit 70. The system control unit 50 monitors the operation of the input devices included in the operation unit 70. When the system control unit 50 detects an operation of an input device, it executes processing corresponding to the detected operation.

[0035] The shutter button 61 has a first shutter switch 62 (SW1) that turns on when pressed halfway, and a second shutter switch 64 (SW2) that turns on when pressed all the way. When the system controller 50 detects that SW1 is turned on, it executes preparatory operations for still image shooting. These preparatory operations include AE processing and AF processing. When the system controller 50 detects that SW2 is turned on, it executes still image shooting and recording operations according to the shooting conditions determined by the AE processing.

[0036] The mode selector switch 60 is an operation unit for switching settings for subject selection. Fig. 4 shows an example of a setting screen for switching the sensitivity setting for main subject selection using the action determination process described below with the mode selector switch 60. The sensitivity setting can be selected from three types: high, standard, and low.

[0037] Furthermore, the operation unit 70 of this embodiment has a gaze detection unit 71 that detects the direction of the user's gaze. The gaze detection unit 71 is not a component that is directly operated by the user, but is included in the operation unit 70 because the gaze direction detected by the gaze detection unit 71 is treated as an input.

[0038] 3(a) is a side view showing a schematic configuration example of the gaze detection unit 71 provided in the viewfinder. The gaze detection unit 71 detects the rotation angle of the optical axis of the user's eyeball 301a as they look at the display unit 28 provided inside the main body 100 through the eyepiece of the viewfinder as the gaze direction. Based on the detected gaze direction, it is possible to identify the position at which the user is gazing on the display unit 28 (the gaze point in the displayed image).

[0039] For example, a live view image is displayed on display unit 28, and the user can view the display content of display unit 28 through eyepiece lens 71d and dichroic mirror 71c by looking into the eyepiece window. Light source 71e can emit infrared light toward the eyepiece window (toward the outside of main body 100). When the user is looking through the viewfinder, the infrared light emitted by light source 71e is reflected by eyeball 301a and returns to the viewfinder. The infrared light that enters the viewfinder is reflected by dichroic mirror 71c toward light-receiving lens 71b.

[0040] The light receiving lens 71b forms an infrared light image of the eyeball on the imaging surface of the imaging element 71a. The imaging element 71a is a two-dimensional imaging element having a filter for infrared light imaging. The number of pixels of the imaging element 71a for gaze detection may be smaller than the number of pixels of the imaging element 22 for imaging. The eyeball image captured by the imaging element 71a is transmitted to the system control unit 50. The system control unit 50 detects the position of the corneal reflection of infrared light and the position of the pupil from the eyeball image, and detects the gaze direction from the positional relationship between the two. The system control unit 50 also detects the position of the display unit 28 at which the user is gazing (the gaze point in the displayed image) based on the detected gaze direction. Note that the position of the corneal reflection and the position of the pupil from the eyeball image may be detected by the image processing unit 24, and the system control unit 50 may acquire these positions from the image processing unit 24.

[0041] The present invention does not depend on the gaze detection method or the configuration of the gaze detection unit. Therefore, the configuration of the gaze detection unit 71 is not limited to that shown in FIG. 3(a). For example, as shown in FIG. 3(b), the gaze may be detected based on an image captured by a camera 71f located near the display unit 28 on the back of the main body 100. The angle of view of the camera 71f, indicated by the dotted line, is set so that the face 300 of the user capturing the image while looking at the display unit 28 is captured. The gaze direction can be detected based on images of the eye regions 301a and 301b detected from the image captured by the camera 71f. When an infrared light image is used, a light source 71e is placed near the camera 71f, and infrared light is projected onto the subject within the angle of view to capture the image. The method for detecting the gaze direction from the obtained image can be the same as the configuration shown in FIG. 3(a). Furthermore, when a visible light image is used, light projection is not necessary. When a visible light image is used, the gaze direction can be detected based on the positional relationship between the inner corner of the eye and the iris in the eye region, for example.

[0042] 1, the power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the type of battery, and the remaining battery charge. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 200. The power supply unit 30 is composed of a battery, an AC adapter, etc.

[0043] The I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. Data files such as captured images and audio are recorded on the recording medium 200. The data files recorded on the recording medium 200 can be read out via the I / F 18 and played back via the image processing unit 24 and the system control unit 50.

[0044] The communication unit 54 realizes communication with external devices by at least one of wireless communication and wired communication. Images (including live view images) captured by the image sensor 22 and images recorded on the recording medium 200 can be transmitted to external devices via the communication unit 54. In addition, image data and various other information can be received from external devices via the communication unit 54.

[0045] The orientation detection unit 55 detects the orientation of the main body 100 with respect to the direction of gravity. The orientation detection unit 55 may be an acceleration sensor or an angular velocity sensor. The system control unit 50 can record orientation information corresponding to the orientation detected by the orientation detection unit 55 during shooting in a data file that stores image data obtained during the shooting. The orientation information can be used, for example, to display a recorded image in the same orientation as when it was shot.

[0046] In a digital camera system having the above configuration, the image processing unit 24 detects, as feature regions, regions of an image captured by the image sensor 22 that are determined to match predetermined characteristics using a known method, and outputs detection information such as the position, size, and reliability of each feature region to the system control unit 50.

[0047] The system control unit 50, which has acquired the detection information, selects, from among the characteristic regions, the characteristic region that is most suitable as the main subject in that shooting scene as the main subject region. For example, if multiple characteristic regions are detected in shooting a specific sporting event, the system control unit 50 selects, as the main subject region, the region of the subject that is determined by the action determination process to be performing an action that is important in shooting the event, from among the subjects corresponding to the characteristic regions.

[0048] The system control unit 50 then performs various types of shooting control to ensure that the main subject region is captured in an appropriate image. These controls include, but are not limited to, automatic focus adjustment (AF) to focus on the characteristic region, automatic exposure control (AE) to ensure that the characteristic region is properly exposed, automatic white balance control to ensure that the characteristic region is properly white balanced, and automatic flash light intensity adjustment to ensure that the characteristic region is properly brightened. Furthermore, the system control unit 50 displays a rectangular frame corresponding to the main subject region on the shooting screen displayed on the display unit 28.

[0049] The present invention does not depend on the type or detection method of a characteristic region, and a known method can be used to detect a characteristic region, so a description of the detection method of the characteristic region will be omitted. Furthermore, the characteristic region can also be used to detect subject information. When the characteristic region is a face region, subject information may include, but is not limited to, whether or not red-eye occurs, whether or not the eyes are closed, and facial expression (e.g., smiling).

[0050] Next, with reference to the flowchart in FIG. 5, a processing procedure will be described for performing a photographing preparation operation and a continuous photographing operation while repeatedly performing focus adjustment and exposure control in the digital camera system having the above configuration.

[0051] 5 is a flowchart showing the steps of the shooting preparation operation and the continuous shooting operation after SW1 is turned ON. The processing from S501 to S506 is a series of processing corresponding to one frame in continuous shooting, and continuous shooting is performed by repeating this series of processing.

[0052] In S501, the system control unit 50 generates a live view image signal by accumulating charge in the image sensor 22, and outputs the generated image signal to the image processing unit 24 via the A / D converter 23. The image processing unit 24 generates a captured image, an image signal for phase difference AF, and an image signal for AE based on the input image signal, and stores these in the memory 32 via the memory control unit 15. These pieces of data stored in the memory 32 are read out and used as appropriate in each of the subsequent steps.

[0053] In S502, the image processing unit 24 performs a feature region detection process on the captured image generated in S501, and simultaneously executes an action determination process on the detected feature regions. Next, the image processing unit 24 outputs the number of detected feature regions, information on each feature region, action determination results, etc. to the system control unit 50. Next, the system control unit 50 sets a main subject region in the captured image based on the acquired information on the feature regions and action determination results. Details of the feature region detection process, action determination process, and main subject region setting process performed in S502 will be described later.

[0054] In S503, the system control unit 50 performs focus detection based on the image signal for phase difference AF corresponding to the main subject area set in S502, and performs AF processing to focus on the subject within the main subject area by driving the lens group 103 based on the detected defocus amount. The system control unit 50 also performs automatic exposure calculation using a known method based on the image signal for AE obtained from the pixels of the main subject area, and determines the aperture value (AV value), shutter speed (TV value), and ISO sensitivity (ISO value) using a program diagram stored in advance. If the main subject area is not set in S502, the focus position and exposure value of the immediately preceding frame are retained.

[0055] In S504, the system control unit 50 detects the state of SW2, and if SW2 is ON, the process proceeds to S505, and if SW2 is OFF, the process proceeds to S506.

[0056] In S505, the system control unit 50 adjusts the opening size of the diaphragm 102 based on the aperture value determined in S503, and then drives the shutter 101 based on the shutter speed also determined in S503 to expose the image sensor 22 and generate a still image signal. The generated still image signal is subjected to a gain according to the ISO sensitivity and then transmitted to the image processing unit 24, which generates a still image based on the received still image signal and outputs it to the system control unit 50. The system control unit 50 then stores the received still image in the recording medium 200 and displays the captured image on the display unit 28.

[0057] In S506, the system control unit 50 detects the states of SW1 and SW2, and if either SW1 or SW2 is ON, proceeds to S501, and if both are OFF, stops the shooting preparation operation and the continuous shooting operation.

[0058] The above is the procedure for the shooting preparation operation and continuous shooting operation in this embodiment. Note that during continuous shooting, the live view image capture of S501 may not be performed, and the processes of S502 and S503 may be performed using the still image captured in the previous frame. Alternatively, the processes of S501 to S503 may be repeated multiple times between still image captures during continuous shooting.

[0059] Next, the main subject region setting process performed in S502 will be described in detail. Fig. 6(a) is a functional block diagram of image processing unit 24, and Fig. 7 is a flowchart showing the procedure for the feature region detection and main subject selection process. Note that in the following description, a ball game played by multiple people will be described as an example of a target scene, but scenes to which this embodiment can be applied are not limited to this.

[0060] First, in S701, the system control unit 50 reads out the captured image generated in S501 or S505 of the previous frame, and outputs an image signal of the read captured image to the image acquisition unit 601. The image acquisition unit 601 outputs the acquired image signal to the subject detection unit 602, joint point detection unit 603, unique object detection unit 606, and subject tracking unit 607.

[0061] In S702, the subject tracking unit 607 reads the tracking template 609 and performs subject tracking processing on the captured image acquired in S701. The subject tracking processing is processing for detecting an area in the image that has features that highly match the tracking template as a feature area. Since the tracking template 609 contains feature information of the feature area corresponding to the main subject in the immediately preceding frame, the subject tracking processing can detect the position and size in the current frame of the main subject in the immediately preceding frame. Note that any method for subject tracking processing may be used, and a known method is used here.

[0062] After the subject tracking process, the subject tracking unit 607 outputs the position, size, and tracking reliability of the detected characteristic area to the detection information integration unit 605. If the subject tracking process fails to detect characteristic information in the current frame, a tracking subject lost flag indicating that the tracked subject has been lost is set, and this is output to the detection information integration unit 605. If characteristic information is not set in the tracking template 609, the subject tracking process is not executed, and the subject tracking unit 607 sets a tracking subject not present flag and outputs this to the detection information integration unit 605.

[0063] In S703, the subject detection unit 602 executes subject detection processing on the captured image acquired in S701. The subject detection processing is processing for detecting a specific subject from within the image as a characteristic region. Here, the objects to be detected are people and their respective body parts, such as the eyes, face, head, upper body, and torso. Therefore, the subject detection processing can detect the position and size of people within the image. Any method for the subject detection processing may be used, and a known method is used here. After the subject detection process, the subject detection unit 602 outputs the position, size, and detection reliability of the detected characteristic region to the detection information integration unit 605 .

[0064] FIG. 9( a ) is a diagram showing how the head of a subject 901 is detected as a characteristic region 904 and the head of a subject 902 is detected as a characteristic region 905 .

[0065] In S704, the unique object detection unit 606 performs unique object detection processing on the captured image acquired in S701. The unique object detection processing is processing for detecting an object unique to a scene in an image as a feature region, and here, a ball in a captured scene of a ball game is used as an example of an object unique to a scene. Therefore, the unique object detection processing detects the position and size of the ball in the image. Any method for the unique object detection processing may be used, and here, a known method is used. After the unique object detection process, the unique object detection unit 606 outputs the position and size of the detected feature region to the action determination unit 604 .

[0066] FIG. 8(a) is a diagram showing an example of the results of executing the detection processes of S702 to S704 in a soccer shooting scene.

[0067] The subject detection unit 602 detects the heads of subjects 801 and 802, and the results are subject feature regions 804 and 805. The unique object detection unit 606 detects ball 803, and the result is unique object region 807. The subject tracking unit 607 detects using template image data 808 shown in FIG. 8(b), which corresponds to tracking template 609, and the result is tracking feature region 806. Based on the physical features and the shape and pattern of clothing contained in template image data 808, subject 801 with a higher matching of feature amounts is detected.

[0068] In the example shown in Figure 8(a), the feature region of the subject is the head, but it is not limited to this and may be any other part that can be determined as the subject, such as the face, eyes, whole body, upper body, lower body, torso, etc.

[0069] In S705, the joint point detection unit 603 executes joint point detection processing on the captured image acquired in S701. The joint point detection processing is processing for detecting the positions of each joint point (predetermined parts) of all subjects in the image and connecting the joint points together. In this example, the joint points to be detected are a total of 10 points: the top of the person's head, neck, both elbows, both wrists, both knees, and both ankles. After detecting each joint point, the joint point detection unit 603 connects the joint points that are estimated to belong to the same person. A collection of 1 to 10 joint points that are connected to each other becomes the joint point information for one subject. Any method may be used for detecting articulation points and processing connections, and a known method is used here.

[0070] After the joint point detection process, the joint point detection unit 603 outputs the position information and connection information of each joint point to the action determination unit 604 .

[0071] In S706, the action determination unit 604 performs an action determination process using the position information and connection information of the joint points detected by the joint point detection unit 603 and the position and size of the unique object detected by the unique object detection unit 606. The action determination process is a process for determining whether the detected subject is performing an action important for filming in the sporting event to be filmed. An important action for filming is a specific action that is often given priority for filming in various sporting events, such as shooting, passing, and dribbling in soccer or basketball, or receiving, tossing, and spiking in volleyball. Important actions for filming also include defensive actions such as a goalkeeper's save or sliding in soccer. Hereinafter, a state in which the subject is performing an important action for filming (a predetermined motion state) is referred to as an "action state." These important actions for filming are previously associated with various sporting events, and a machine-learned model of the position information of the joint points and the position and size of the unique object corresponding to the action state is previously stored in the non-volatile memory 56, memory 32, or the like. Then, at any point before the start of shooting, the model required for the action determination process can be used by selecting a sporting event using the operation unit 70. Alternatively, position information of joint points and position and size information of specific objects corresponding to action states may be stored in the non-volatile memory 56, memory 32, or the like, and the position information of joint points and position and size information of specific objects required for the action determination process may be used.

[0072] In the action determination process, an action likelihood is calculated as an index for determining whether a subject is in an action state. The action likelihood indicates the likelihood that the subject is in an action state, and is calculated as a numerical value between 0.000 and 1.000 based on the positions of each joint point of each subject and the positions and sizes of the inherent objects. Then, a subject whose action likelihood is equal to or greater than a predetermined threshold is determined to be in an action state. Note that the numerical range of the action likelihood may be set arbitrarily.

[0073] After the action determination process, the action determination unit 604 outputs the position information of each joint, the action likelihood, the action determination result, and the feature region information (position, size) of the unique object to the detection information integration unit 605.

[0074] FIG. 9 is a diagram showing an example of action determination processing for a subject 901 about to shoot and a subject 902 ready to defend in a soccer shooting scene.

[0075] Subject 901 is in an action state, as he is preparing to kick ball 903. On the other hand, subject 902 is away from ball 903 and is therefore not in an action state.

[0076] 9(b) is a diagram showing how a total of ten positions of the top of the head, neck, elbows, wrists, knees, and ankles of subject 901 are detected as joint points 906, and how a joint point 907 of subject 902 is similarly detected. Also, region 908 is a feature region when ball 903 is detected by the unique object detection unit 606.

[0077] In the action determination, the action likelihood of object 901 is calculated to be high because the position of the corresponding joint point 906 indicates a posture of kicking a ball, and a feature region 908 of the ball is detected near the ankle of joint point 906. On the other hand, the action likelihood of object 902 is calculated to be low because joint point 907 indicates a posture close to standing upright, and the distance between joint point 907 and the feature region 908 of the ball is large. As a result, object 901 corresponding to joint point 906 is determined to be in an action state, and object 902 corresponding to joint point 907 is determined to be not in an action state.

[0078] Any method may be used to calculate the action likelihood, for example, a method using a neural network, which is a type of machine learning technique, to perform supervised learning using the joint points and ball positions in the correct action state as training data. A specific example is the method described in JP 2021-071794 A. Other machine learning techniques, such as support vector machines and decision trees, may also be used, or a function that outputs a reliability or probability value based on a certain model may be constructed, not limited to machine learning. Furthermore, in this embodiment, a case is described in which the likelihood is used as the degree of possibility that the subject is performing an action that is important for filming, but a value other than likelihood may also be used. For example, the reliability may be the inverse of the distance between the center of gravity of the subject and the center of gravity of the eigenobject.

[0079] Furthermore, if a given subject continues to be determined to be in an action state continuously or intermittently over multiple frames, corrections may be made to increase the action likelihood of that subject. The reason for making such corrections is that if a specific subject continues to be determined to be in an action state, it can be inferred that the user is framing that subject, and that that subject is more likely than other subjects to continue to be in an action state in the future.

[0080] In this embodiment, the main subject is determined using information about the unique objects, but it is also possible to determine the main subject using only the joint point information of the subject.In addition, data that has undergone a predetermined transformation, such as a linear transformation, on the joint positions and the positions and sizes of the unique objects may be used as input data.

[0081] In S707, the detection information integration unit 605 integrates the information acquired from the subject detection unit 602, the action determination unit 604, and the subject tracking unit 607 to generate integrated detection information 620. Fig. 6(b) is an example of the integrated detection information 620. The integrated detection information 620 includes feature region information 621 for the same number of subjects detected by the subject detection unit 602, a subject feature region count 629 indicating the number of subjects, and feature regions 622 of unique objects detected by the unique object detection unit 606.

[0082] The detection information integration unit 605 associates feature regions of subjects detected by the subject detection unit 602 with action determination results determined by the action determination unit 604, which are estimated to correspond to the same subject. The association is performed by, for example, comparing each pair of joint points of the top of the head and the neck that are connected with the head region or upper body region of each detected subject, and associating pairs of joint points that are closer in position to the subject's feature region than a predetermined distance. In FIG. 9 , feature region 904 and joint point 906, and feature region 905 and joint point 907 are associated as data associated with the same subject. The associated data is then stored in feature region information 621. The feature region information 621 includes a subject detection flag 623 indicating that the feature region is detected by the subject detection unit 602, an action likelihood 625 for each subject, and an action determination flag 626 indicating that the feature region is in an action state.

[0083] Furthermore, the feature region information 621 includes a tracking subject flag 624 indicating that the target subject is a tracking subject. The detection information integrating unit 605 compares the position and size of the tracking feature region acquired from the subject tracking unit 607 with the feature region information 621 of each subject, and if it is determined that there is an overlap of a predetermined percentage or more, sets the tracking subject flag 624 included in the data of the feature region information 621 to TRUE. Note that if none of the feature region information 621 satisfies the overlap condition with the tracking feature region, one new independent piece of feature region information 621 is added, only the tracking subject flag 624 is set to TRUE, and 1 is added to the number of subject feature regions 629.

[0084] Furthermore, the integrated detection information 620 includes a tracked subject lost flag 627 and a tracked subject no flag 628 set by the subject tracking unit 607, and a subject ID 630 for identifying each characteristic area. The subject ID 630 is set so that the same ID is assigned to subjects that are estimated to be the same subject when detected from images of different frames.

[0085] The integrated detection information 620 also includes a subject change restriction timer 631. Each time a main subject region is determined in S708, which will be described later, a predetermined value is set in the subject change restriction timer 631 if the action determination flag 626 in the feature region information 621 corresponding to the main subject region is TRUE. If the value of the subject change restriction timer 631 is greater than 0, the timer counts down as time passes, and stops counting down when the value reaches 0. The subject change restriction timer 631 is used to restrict changes to the main subject in S708. The subject change limit timer 631 may be a numerical value representing the number of frames, or may be set as real time.

[0086] When the integration of all the detection information is completed, the detection information integration unit 605 outputs the integrated detection information 620 to the system control unit 50.

[0087] In S708, the system control unit 50 determines a main subject region from the feature region information 621 based on the acquired integrated detection information 620. The method for determining the main subject region will be described in detail later.

[0088] In S709, the tracking template generation unit 608 acquires information about the main subject region determined by the system control unit 50, and updates the information about the tracking template 609 based on the acquired information about the main subject region. The updated tracking template 609 is used by the subject tracking unit 607 to detect the characteristic region of the main subject from the next frame. The above is the procedure for the main subject region setting process in this embodiment.

[0089] Next, the detailed procedure performed in S708 by the system control unit 50 to determine the main subject region will be described with reference to the flowchart of FIG.

[0090] First, in S1101, the system control unit 50 acquires the integrated detection information 620 from the detection information integration unit 605.

[0091] In S1102, the system control unit 50 classifies each subject feature region included in the acquired integrated detection information 620 into four categories 1001 as shown in the table of FIG. 10 based on the corresponding tracking subject flag 624 and action determination flag 626. Feature area that is in an action state and is a tracking subject → Action tracking subject Feature area in an action state that is not a tracking subject → Action non-tracking subject Feature area where the subject is not in action but is being tracked → Non-action tracking subject Feature areas that are not in an action state and are not a tracking subject → Non-action, non-tracking subject Classify as:

[0092] The priority of the action tracking subject as the main subject is set to +3, the priority of the action non-tracking subject is set to +2, the non-action tracking subject is set to +1, and the non-action non-tracking subject is set to 0. The higher the priority number, the higher the priority.

[0093] In S1103, the system control unit 50 checks the no-object-to-be-tracked flag 628, and if it is TRUE, the process proceeds to S1109, and if it is FALSE, the process proceeds to S1104.

[0094] In S1104, the system control unit 50 determines whether the tracking subject lost flag 627 is FALSE, that is, whether the tracking main subject has been detected in the previous image. If the tracking subject lost flag 627 is FALSE (the tracking subject has been detected), the process proceeds to S1105, and if the tracking subject lost flag 627 is TRUE (the tracking subject has not been detected), the process proceeds to S1106. Note that the process may proceed to S1106 if the tracking subject lost flag 627 remains TRUE for a predetermined number of frames.

[0095] In S1105, the system control unit 50 counts down the subject change restriction timer 631 in accordance with the time elapsed since the immediately preceding frame. Note that the subject change restriction timer 631 was set to a predetermined value in S1111 (to be described later) for the previous frame.

[0096] In S1106, the system control unit 50 determines that the tracking subject has gone out of frame, and clears the subject change limit timer 631 to zero.

[0097] In S1107, if the subject change restriction timer 631 is greater than 0, the system control unit 50 determines that the timer has not expired and proceeds to S1108, and if the value is 0, the system control unit 50 determines that the timer has expired and proceeds to S1109.

[0098] In S1108, system control unit 50 determines, as the main subject region, a feature region for which tracking subject flag 624 is TRUE from feature region information 621. Here, since subject change restriction timer 631 has not expired in S1107, it is determined not to change the main subject, but to maintain the tracking subject from the previous frame as the main subject.

[0099] In S1109, the system control unit 50 determines the feature region with the highest priority set in S1102 from among the feature region information 621 as the main subject region. When multiple feature regions have the same priority and are related to subjects in an action state, the feature region with the highest action likelihood 625 is determined as the main subject region. Furthermore, when feature regions with the same priority are related to subjects in a non-action state, the feature region that is closer to the center of the image and has the largest area is determined as the main subject region based on the position and size of the feature region. Note that when selecting a main subject region from subjects in an action state, the selection may also be based on the position and size of the feature region. However, in either case, if the number of subject feature regions 629 is 0, the main subject region is not determined, the process ends, and the process returns to S709 in FIG. 7.

[0100] Note that, since all feature area information 621 is considered when selecting the main subject area, if a tracking subject is selected from that, it is determined that the tracking subject from the previous frame will be maintained as the main subject. Therefore, the main subject is not necessarily changed in the processing of S1109.

[0101] In S1110, if the action determination flag 626 of the feature region information 621 determined as the main subject region in S1108 or S1109 is TRUE, it is determined that the main subject is in an action state and the process proceeds to S1111. If it is FALSE, the process returns to S709 in FIG.

[0102] Next, in S1111, system control unit 50 sets subject change restriction timer 631 to a predetermined value. That is, if a subject in an action state is set as the main subject area, or if a subject being tracked from a past frame is in an action state in the current frame, subject change restriction timer 631 is set. This restricts changing the main subject to another subject until the subject change restriction timer reaches 0. After completing the processing of S1111, the process returns to S709 in FIG. 7. The time set in the subject change limit timer 631 may be set to be longer in the order of low, standard, and high based on the sensitivity setting in FIG.

[0103] FIG. 12 is a diagram showing a specific example of the process of determining the main subject region shown in FIG. 11 in a soccer shooting scene.

[0104] In scene 1210, subject feature region 1214 of subject 1212 has been detected, but subject 1211, who is dribbling into the field of view, and ball 1213, which is a specific object, have not been detected. Also, subject 1212 was selected as the main subject region in the frame immediately preceding scene 1210, but is simply standing and is not in an action state (priority=1). In this scene 1210, subject feature region 1214, which is the only subject feature region, is set as the main subject region (S1109), but because the subject is not in an action state, subject change restriction timer 631 is not set (FALSE in S1110).

[0105] Next, as time passes and the scene changes to 1220, a dribbling subject 1211 and a ball 1213 enter the shooting screen, and a subject feature region 1221 of the subject 1211 and a unique object region 1223 of the ball 1213 are detected. Here, the subject 1211 is in a dribbling pose and is holding the ball 1213, so the subject feature region 1221 is determined to be in an action state (priority=2). On the other hand, the subject feature region 1214 for the subject 1212 continues to be detected (FALSE in S1103), but is not in an action state (priority=1). Therefore, in this scene 1220, the main subject region is changed from the subject feature region 1214 to the subject feature region 1221 based on the priority (S1109). At the same time, the subject change limit timer 631 is set to a predetermined value in S1111.

[0106] Subsequently, as time passes and the scene changes to 1230, subject 1212 assumes a sliding stance to steal ball 1213, and subject characteristic region 1214 is determined to be in an action state (priority = 2). On the other hand, subject 1211 has stopped dribbling to keep ball 1213, and subject characteristic region 1221 is less likely to be determined to be in an action state, but is still detected as a tracking subject (priority = 1). However, at the time of scene 1230, subject change restriction timer 631 set in scene 1220 has not yet expired (YES in S1107), so subject characteristic region 1221 continues to be selected as the main subject region (S1108), and the main subject is not changed to subject characteristic region 1214.

[0107] The above is a detailed explanation of the procedure for determining the main subject area. With this configuration, once a subject in an action state is determined as the main subject, even if another subject in an action state is detected in a subsequent frame, the subject will not be changed while the subject change limit timer is running. Therefore, in a shooting scene in which multiple subjects are simultaneously in action, it is possible to prevent the main subject area from frequently changing between these multiple subjects.

[0108] On the other hand, by not enabling the subject change limit timer when changing from a non-action subject to an action subject, highly responsive subject changes using action judgment are possible when shooting sports competitions where scenes change rapidly.Furthermore, by clearing the subject change limit timer to zero when the tracking subject is lost, it is possible to quickly change subjects when the tracking subject is taken out of frame and another action subject is framed.

[0109] Note that these methods of setting the priority and determining the main subject area are merely examples, and any method may be used as long as it is a means for realizing the following two operations. (1) The change of the main subject from a subject not in action to a subject in action is performed immediately. (2) The change of the main subject from a subject in an action state to a subject in an action state is performed only when a predetermined condition is met.

[0110] Furthermore, even if a main subject in an action state is no longer in the action state, the rule that the main subject is changed only when a predetermined condition is met may continue to be applied until a predetermined time has elapsed.

[0111] <Second embodiment> Next, a second embodiment of the present invention will be described, which will explain another specific example of the means for realizing the above-mentioned two operations. In this embodiment, a method for restricting a change of the main subject from a subject in an action state to another subject in an action state will be described, using a means different from that in the first embodiment.

[0112] The difference between this embodiment and the first embodiment is that the subject change restriction determination is performed using a method other than the method using the subject change restriction timer 631, but the other configurations are the same. Therefore, the method of restricting the change of the main subject in this embodiment will be described with reference to Fig. 13 .

[0113] 13 is a flowchart showing a detailed procedure for determining the main subject region by the system control unit 50 in this embodiment. Note that the same step numbers are used for steps that perform the same processing as in FIG. 11 described in the first embodiment, and descriptions thereof will be omitted.

[0114] In S1301, the system control unit 50 refers to the action determination flag 626 of the feature region information 621 corresponding to the tracking subject, and if TRUE, proceeds to S1302, and if FALSE, proceeds to S1109. The determination made here is made to impose restrictions on changing the main subject when the tracking subject is in an action state.

[0115] In S1302, system control unit 50 determines whether to suppress a change of the main subject based on the information in integrated detection information 620 and predetermined conditions for restricting a change of the main subject. The result of this determination is used in branching at S1303. The specific method of determination will be described later.

[0116] In S1303, if the system control unit determines based on the determination result in S1302 that the change of the main subject should be suppressed, the process proceeds to S1108, and if it determines that the change should not be suppressed, the process proceeds to S1109.

[0117] Hereinafter, a method for determining whether or not to suppress the change of the main subject in S1302 will be specifically described using a number of methods. The methods described below may be implemented in addition to, or may partially or completely replace, the subject change restriction determination by the subject change restriction timer 631 in S1104 to S1111 of Fig. 11 of the first embodiment. In the following description, a unique object may be referred to as a ball, and a subject area may be referred to as a person. The flowchart in Fig. 13 illustrates a case in which the subject change restriction determination by the subject change restriction timer 631 is completely replaced.

[0118] The following restriction conditions 1 to 13 indicate specific conditions for restricting a change of a subject for determining whether or not to restrict a change of a subject in S1302. Note that restriction conditions 1 to 13 include a condition for determining that a change of a subject should be restricted if the condition is met, and a condition for determining that a change of a subject should not be restricted if the condition is met.

[0119] Restriction condition 1: If multiple subject feature regions in an action state are detected near the unique object feature region (within a predetermined distance), and the tracking subject region is one of those subject feature regions, it is determined that subject changes should be suppressed. This is because, in a sports competition scene, when multiple people gather near the ball, changing the subject each time could result in frequent subject changes.

[0120] Restriction condition 2: In a predetermined series of frames, if the unique object feature region moves away from the tracked subject region and approaches another subject region, and the other subject is in an action state, it is determined that the subject change should not be suppressed, but it is determined that it should be suppressed in all other cases. This is because, when the ball is passed from one person to another, the main subject can be changed smoothly in accordance with the movement of the ball due to the pass.

[0121] Restriction condition 3: If a subject area in an action state is detected separately from the tracked subject, and the distance between that subject area and the specific object area is closer than the distance between any other subject area and the specific object area by a predetermined amount, it is determined not to suppress the subject change, and in all other cases it is determined to suppress it. This makes it possible to stably continue to select the subject holding the ball as the main subject in accordance with the passing of the ball between subjects.

[0122] Restriction condition 4: If a subject area in an action state is detected separately from the tracked subject, and the distance between that subject area and the specific object area is greater than the distance between the tracked subject and the specific object area by a predetermined amount, it is determined that the subject change should be suppressed. This makes it possible to suppress the change of subject from a subject positioned near the ball to a subject positioned farther from the ball, and to stably continue to select the subject holding the ball as the main subject in accordance with the movement of the ball.

[0123] Restriction condition 5: If the user frames a subject in an action state other than the tracking subject so that the other subject is located at the center of a predetermined area at a timing closer than a predetermined time to the timing at which the tracking subject is detected, the system determines not to suppress the change of the subject; otherwise, the system determines to suppress the change of the subject. Here, the predetermined area is, for example, the shooting screen or the focus detection area. This allows for a smooth change of the subject when the subject to be photographed is in an action state and the user instantaneously frames the subject.

[0124] Restriction condition 6: Using information about the user's gaze position detected by the gaze detection unit 71, if, for example, it is detected that the user is moving their gaze toward a subject in an action state at the same time as that subject is detected, it is determined not to suppress the subject change, and in all other cases it is determined to suppress the change. This allows the user to use the gaze detection function to quickly determine the main subject when a scene occurs in which sudden, vertical and horizontal action occurs during filming of a fast-moving and complex sports competition scene.

[0125] Restriction condition 7: When the user directly sets the main subject using the touch display of display unit 28, even if another subject in an action state is detected, it is determined that a subject change for that subject will be restricted. This is because the subject directly specified by the user is the most important subject for the user to photograph, and restricting subject changes is likely to be in line with the user's intentions for photographing.

[0126] Restriction condition 8: If a subject area in an action state is detected separately from the tracking subject, but the subject detection unit 602 does not detect the face of that subject, it is determined that the subject change should be suppressed, but if a face is detected, it is determined that the subject change should not be suppressed. This is because, for the user, a subject without a face is likely to be a subject with low suitability as a main subject. By suppressing the subject change to a subject without a face, it is possible to select a subject that is in line with the user's photographic intent.

[0127] Restriction condition 9: It is determined that a change of the subject should be suppressed unless the action likelihood 625 of the tracked subject falls below 90%, which is the threshold for determining that the subject is in an action state. Once it falls below 90%, it is determined that no suppression should be performed. This makes it possible to prevent frequent changes of the subject when multiple subjects are in an action state at the same time. Note that instead of the threshold of 90%, a number multiplied by any other percentage or a number obtained by adding or subtracting a predetermined number may be used.

[0128] Restriction condition 10: In S707, the number of joint points associated with the tracking subject is compared with the number of joint points associated with a subject area other than the tracking subject, and it is determined that a change to the subject should be suppressed for a subject area with fewer joint points than the tracking subject. This is because, for the user, a subject hidden by other subjects is likely to be a subject with low suitability as a main subject. A small number of joint points means that each part of the subject's body is not captured, so suppressing the change to such a subject makes it possible to select a subject that is in line with the user's intentions for shooting. It should be noted that, instead of the method using articulation points, various other methods may be used to determine whether or not a part of the subject is visible, such as a method of detecting the part of the subject.

[0129] Restriction condition 11: If the position of the tracking subject is within a predetermined area set in the center of the screen and another subject is within a predetermined area set on the periphery of the screen, it is determined that the subject change to that subject should be suppressed. Conversely, if the position of the tracking subject is within a predetermined area set on the periphery of the screen and another subject is within a predetermined area set in the center of the screen, it is determined that the subject change to that subject should not be suppressed. This is because, for the user, a subject that is cut off in the corner of the shooting screen is likely to be a subject that is not suitable as a main subject. Therefore, by suppressing the subject change to such a subject, it is possible to select a subject that is in line with the user's shooting intentions.

[0130] Restriction condition 12: If the tracking subject continues to take action for a predetermined period of time or longer, it is determined that subject changes should be suppressed. This is because, for actions that continue for a relatively long period of time, such as soccer dribbling, if subject changes are not suppressed, there is a possibility that the subject will frequently change to opposing players competing for the ball. Therefore, by suppressing this, it is possible to continue to stably select a dribbling soccer player as the main subject.

[0131] Restriction condition 13: If the subject selected as the main subject at the moment the user turns on SW1 is also in an action state at the same timing, it is determined that the subject change will be suppressed in subsequent frames. This is because it is assumed that the user has started shooting in response to the subject's action, and therefore it is better not to simply change the main subject to another subject, as this will enable the user to select a subject that is in line with their photographic intent.

[0132] As described above, in S1302, it is determined whether or not to suppress the change of the subject using any one of the methods of restriction conditions 1 to 13, or a combination of multiple methods. When a combination of multiple restriction conditions results in a conflict between a decision to suppress and a decision not to suppress, the final decision may be made by setting a priority for each restriction means. By determining whether or not to suppress the change of the subject using the above restriction conditions, it is possible to change the subject in accordance with the user's intentions for shooting.

[0133] <Modification> Next, an example of determining whether or not to suppress a change of subject by combining the subject change restriction timer 631 and some of the restriction conditions 1 to 13 will be described with reference to the flowchart of Fig. 14. Note that steps that perform the same processes as those in Fig. 11 described in the first embodiment and Fig. 13 described in the second embodiment will be assigned the same step numbers, and descriptions thereof will be omitted.

[0134] 14, even if the subject change restriction timer 631 has not expired in S1107, the determination of the subject change restriction conditions in S1302 determines whether to suppress the subject change based on any of restriction conditions 1 to 13. If it is not determined that the subject change should be suppressed, the process proceeds from S1303 to S1109, where the subject with the highest priority is selected as the main subject. On the other hand, if it is determined in S1303 that the subject change should be suppressed, the subject change is suppressed, and the process proceeds to S1108.

[0135] As a result, even if the subject change limit timer has not expired, if it is determined that the main subject should be changed, it is possible to change the subject with high responsiveness in accordance with the user's intention to take a photograph.

[0136] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0137] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0138] <Summary> The disclosure of this embodiment includes the following configuration.

[0139] (Item 1) a first detection means for detecting one or more subjects from images obtained by repeatedly photographing; a determining means for determining whether each subject detected by the first detecting means is in a predetermined motion state; a selection means for selecting one of the subjects detected by the first detection means as a main subject; a control means for restricting a change of the main subject in subsequent images when the subject determined to be in the moving state by the determination means is selected as the main subject by the selection means; An electronic device comprising: (Item 2) The electronic device described in item 1 is characterized in that the control means controls so as not to change the main subject for a predetermined period of time after it is determined that the subject selected as the main subject is in the moving state. (Item 3) The apparatus further includes a setting unit for setting the sensitivity of the selection by the selection unit, Item 3. The electronic device according to item 2, characterized in that when the sensitivity is a first sensitivity, the predetermined period is made longer than when the sensitivity is a second sensitivity that is higher than the first sensitivity. (Item 4) 4. The electronic device according to any one of items 1 to 3, characterized in that, when the subject selected as the main subject in one image is not in the moving state, and another subject in the moving state is detected in a subsequent image, the selection means reselects the subject in the moving state as the main subject. (Item 5) 5. The electronic device described in any one of items 1 to 4, characterized in that, if the subject in the moving state selected as the main subject is not detected in the subsequent images for a predetermined period of time, the control means controls to cancel the restriction on changing the main subject. (Item 6) 6. The electronic device described in any one of items 1 to 5, characterized in that, in the subsequent image, when the subject in the moving state selected as the main subject moves out of the frame, the control means controls to release the restriction on changing the main subject. (Item 7) further comprising second detection means for detecting a predetermined object associated with the operating state; 7. The electronic device according to any one of items 1 to 6, wherein the selection unit selects the main subject based on the determination result by the determination unit and the position of the object. (Item 8) Item 7. The electronic device described in item 7, characterized in that, in the subsequent images, when multiple subjects including the subject in the moving state selected as the main subject are detected within a predetermined distance range from the object, the control means controls to suppress a change of the main subject. (Item 9) Item 8. The electronic device described in item 7, characterized in that, in the subsequent images, when the object moves from the subject in the moving state selected as the main subject to approach another subject in the moving state, the control means controls so as not to suppress the change of the main subject to the other subject. (Item 10) Item 7. The electronic device described in item 7, characterized in that, in the subsequent images, if the distance between the object and another subject in the moving state is closer than the distance between the object and the subject in the moving state selected as the main subject, the control means controls so as not to suppress the change of the main subject to the other subject. (Item 11) Item 7. The electronic device described in item 7, characterized in that, in the subsequent images, if the distance between the object and another subject in the moving state is greater than the distance between the object and the subject in the moving state selected as the main subject, the control means controls to suppress changing the main subject to the other subject. (Item 12) 12. The electronic device described in any one of items 1 to 11, characterized in that, in the subsequent image, when the first detection means detects another subject in the moving state that is different from the subject in the moving state selected as the main subject, and when the other subject is framed to be at the center of a predetermined area, the control means controls so as not to suppress the change of the main subject to the other subject. (Item 13) Further comprising a line of sight detection means, 13. The electronic device described in any one of items 1 to 12, characterized in that, in the subsequent image, the first detection means detects another subject in the moving state that is different from the subject in the moving state selected as the main subject, and when the gaze is focused on the other subject, the control means controls so as not to suppress changing the main subject to the other subject. (Item 14) the first detection means has means for determining whether or not a face of each of the detected subjects is included in the image; The electronic device described in any one of items 1 to 13 is characterized in that, when the subject selected as the main subject is in the moving state and the face of another subject in the moving state is not captured, the control means controls to suppress changing the main subject to the other subject. (Item 15) the first detection means has a means for determining whether or not a body part of each of the detected subjects is photographed; The electronic device described in any one of items 1 to 14 is characterized in that, when the subject selected as the main subject is in the moving state and the number of parts of another subject in the moving state that are captured is less than the number of parts of the main subject by a predetermined amount, the control means controls to suppress changing the main subject to the other subject. (Item 16) The electronic device described in any one of items 1 to 15 is characterized in that, when the subject selected as the main subject is in the moving state and the position of another subject in the moving state is within a predetermined area on the periphery of the image, the control means controls to suppress changing the main subject to the other subject. (Item 17) The electronic device described in any one of items 1 to 16 is characterized in that the control means controls to suppress changing of the main subject to another subject in the moving state when the subject selected as the main subject remains in the moving state for a predetermined period of time. (Item 18) the selection means includes a determination means for determining the main subject in response to a predetermined operation by a user; The electronic device described in any one of items 1 to 17 is characterized in that, when the determination unit determines the subject in the moving state as the main subject in one image, the control unit controls to suppress changing the main subject to another subject in the moving state in subsequent images. (Item 19) further comprising a designation means for designating a main subject, 19. The electronic device according to any one of items 1 to 18, wherein the selection means selects the subject selected by the designation means as a main subject. (Item 20) Item 19 is a diagram showing an electronic device according to the present invention, wherein the control means controls the electronic device so that, when the subject selected by the designation means is set as the main subject in one image, the main subject is not changed to another subject in the moving state in subsequent images. (Item 21) The electronic device described in any one of items 1 to 20, characterized in that the determination means calculates the likelihood of the motion state for each of the objects, and determines that an object whose likelihood is equal to or greater than a predetermined first threshold is in the motion state. (Item 22) 22. The electronic device according to item 21, wherein the selection means determines a priority based on the likelihood and selects the subject with the highest priority as the main subject. (Item 23) The camera further includes a tracking means for tracking the subject selected as the main subject, 22. The electronic device according to item 21, wherein the selection means determines priorities between the tracking result by the tracking means and the likelihood, and selects the subject with the highest priority as the main subject. (Item 24) 24. The electronic device according to any one of items 21 to 23, wherein the determining means calculates the likelihood based on position information of a plurality of predetermined parts of each of the subjects. (Item 25) The electronic device described in any one of items 21 to 24, characterized in that when the subject selected as the main subject is in the moving state, the control means controls to suppress a change of the main subject when the likelihood of the main subject is equal to or greater than a second threshold value lower than the first threshold value. (Item 26) 26. The electronic device according to any one of items 1 to 25, further comprising an imaging means for capturing the images. (Item 27) a detection step of detecting one or more subjects from images obtained by repeatedly photographing; a determination step of determining whether each subject detected in the detection step is in a predetermined motion state; a selection step of selecting one of the subjects detected in the detection step as a main subject; a limiting step of limiting changes of the main subject in subsequent images when the subject determined to be in the moving state in the determining step is selected as the main subject in the selecting step; An image processing method comprising: (Item 28) A program for causing a computer to function as each means of the electronic device described in any one of items 1 to 26. (Item 29) Item 29. A computer-readable storage medium storing the program described in Item 28.

[0140] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0141] 100: main body, 22: image sensor, 50: system control unit, 24: image processing unit, 28: display unit, 32: memory, 53: system timer, 56: non-volatile memory, 60: mode switch, 70: operation unit, 71: gaze detection unit, 601: image acquisition unit, 602: subject detection unit, 603: joint point detection unit, 604: action determination unit, 606: unique object detection unit, 607: subject tracking unit, 620: integrated detection information

Claims

1. a first detection means for detecting one or more subjects from images obtained by repeatedly photographing; a determining means for determining whether each subject detected by the first detecting means is in a predetermined motion state; a selection means for selecting one of the subjects detected by the first detection means as a main subject; a control means for restricting a change of the main subject in subsequent images when the subject determined to be in the moving state by the determination means is selected as the main subject by the selection means; An electronic device comprising:

2. 2. The electronic device according to claim 1, wherein the control means controls the main subject not to be changed for a predetermined period of time after the subject selected as the main subject is determined to be in the moving state.

3. The apparatus further includes a setting unit for setting the sensitivity of the selection by the selection unit, 3. The electronic device according to claim 2, wherein the predetermined period is set longer when the sensitivity is a first sensitivity than when the sensitivity is a second sensitivity higher than the first sensitivity.

4. 2. The electronic device according to claim 1, wherein, when a subject selected as the main subject in one image is not in the moving state, and another subject in the moving state is detected in a subsequent image, the selection means reselects the subject in the moving state as the main subject.

5. 2. The electronic device according to claim 1, wherein the control means controls to cancel the restriction on changing the main subject when the subject in the moving state selected as the main subject is not detected in the subsequent images for a predetermined period of time.

6. 2. The electronic device according to claim 1, wherein, when the subject in the moving state selected as the main subject moves out of the frame in the subsequent image, the control means controls to cancel the restriction on changing the main subject.

7. further comprising second detection means for detecting a predetermined object associated with the motion state; 2. The electronic device according to claim 1, wherein the selection means selects the main subject based on the determination result by the determination means and the position of the object.

8. 8. The electronic device according to claim 7, wherein, when a plurality of subjects including the subject in the moving state selected as the main subject are detected within a predetermined distance range from the object in the subsequent images, the control means controls to suppress a change of the main subject.

9. 8. The electronic device according to claim 7, wherein, when the object moves from the subject in the moving state selected as the main subject to approach another subject in the moving state in the subsequent image, the control means controls so as not to suppress the change of the main subject to the other subject.

10. 8. The electronic device according to claim 7, wherein, in the subsequent image, when the distance between the object and another subject in the moving state is closer than the distance between the object and the subject in the moving state selected as the main subject, the control means controls so as not to suppress the change of the main subject to the other subject.

11. 8. The electronic device according to claim 7, wherein, in the subsequent image, if the distance between the object and another subject in the moving state is greater than the distance between the object and the subject in the moving state selected as the main subject, the control means controls to suppress changing the main subject to the other subject.

12. 2. The electronic device according to claim 1, wherein, in the subsequent image, when the first detection means detects another subject in the moving state that is different from the subject in the moving state selected as the main subject, and when the other subject is framed so as to be at the center of a predetermined area, the control means controls so as not to suppress the change of the main subject to the other subject.

13. Further comprising a line of sight detection means, 2. The electronic device according to claim 1, wherein, in the subsequent image, the first detection means detects another subject in the moving state that is different from the subject in the moving state selected as the main subject, and when the gaze is focused on the other subject, the control means controls so as not to suppress the change of the main subject to the other subject.

14. the first detection means has means for determining whether or not a face of each of the detected subjects is photographed; 2. The electronic device according to claim 1, wherein the control means controls to suppress changing of the main subject to another subject in the moving state when the subject selected as the main subject is in the moving state and the face of the other subject in the moving state is not captured in the image.

15. the first detection means has means for determining whether or not a body part of each of the detected subjects is photographed; The electronic device according to claim 1, characterized in that the control means controls to suppress changing the main subject to another subject when the subject selected as the main subject is in the moving state and the number of parts of another subject in the moving state is less than the number of parts of the main subject by a predetermined amount.

16. 2. The electronic device according to claim 1, wherein the control means controls to suppress changing of the main subject to another subject when the subject selected as the main subject is in the moving state and when the position of another subject in the moving state is within a predetermined area on the periphery of the image.

17. 2. The electronic device according to claim 1, wherein the control means controls to suppress changing of the main subject to another subject in the moving state when the subject selected as the main subject remains in the moving state for a predetermined period of time.

18. the selection means includes a determination means for determining the main subject in response to a predetermined operation by a user; 2. The electronic device according to claim 1, wherein when the determining means determines a subject in the moving state as the main subject in one image, the control means controls to suppress changing the main subject to another subject in the moving state in subsequent images.

19. further comprising a designation means for designating a main subject, 2. The electronic device according to claim 1, wherein the selection means selects the subject selected by the designation means as a main subject.

20. 20. The electronic device according to claim 19, wherein the control means controls so that, when the subject selected by the designation means is set as the main subject in one image, the main subject is prevented from being changed to another subject in the moving state in subsequent images.

21. 2. The electronic device according to claim 1, wherein the determining means calculates a likelihood of the moving state for each of the objects, and determines that an object whose likelihood is equal to or greater than a predetermined first threshold is in the moving state.

22. 22. The electronic device according to claim 21, wherein the selection means determines a priority based on the likelihood, and selects the subject with the highest priority as the main subject.

23. The camera further includes a tracking means for tracking the subject selected as the main subject, 22. The electronic device according to claim 21, wherein the selection means determines priorities between the tracking result by the tracking means and the likelihood, and selects the subject with the highest priority as the main subject.

24. 22. The electronic device according to claim 21, wherein the determining means calculates the likelihood based on position information of a plurality of predetermined parts of each of the subjects.

25. 22. The electronic device according to claim 21, wherein the control means controls to suppress a change of the main subject when the subject selected as the main subject is in the moving state and the likelihood of the main subject is equal to or greater than a second threshold value that is lower than the first threshold value.

26. 2. The electronic device according to claim 1, further comprising an image capturing unit for capturing each of the images.

27. a detection step of detecting one or more subjects from images obtained by repeatedly photographing; a determination step of determining whether each subject detected in the detection step is in a predetermined motion state; a selection step of selecting one of the subjects detected in the detection step as a main subject; a limiting step of limiting changes of the main subject in subsequent images when the subject determined to be in the moving state in the determining step is selected as the main subject in the selecting step; An image processing method comprising:

28. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 26.

29. A computer-readable storage medium storing the program according to claim 28.

Images