Imaging apparatus, control method for the same, and program

The imaging device addresses the challenge of capturing high-quality images of faces and body parts by using detection and evaluation mechanisms to prioritize subjects, ensuring balanced and unbiased image quality in group settings.

JP2024168624APending Publication Date: 2024-12-05CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023085462
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing imaging devices struggle to capture high-quality images that include both faces and body parts of multiple subjects without bias, particularly in scenarios like school events where movements and positions of limbs and whole bodies are important, and existing technologies do not effectively evaluate the quality of such images.

Method used

An imaging device with a configuration that includes a detection mechanism to identify registered subjects, evaluate image quality based on face and body part criteria, and display evaluation values, allowing for focused photography on subjects with the highest quality scores.

Benefits of technology

The device ensures high-quality capture of faces and body parts for multiple subjects, reducing image biases and improving the overall quality of photographs by prioritizing subjects based on predefined evaluation criteria.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168624000001_ABST
    Figure 2024168624000001_ABST
Patent Text Reader

Abstract

To acquire, for each of a plurality of persons, a high-quality photographed image including a face and other body parts than the face in a balanced manner.SOLUTION: An imaging apparatus 100 comprises: a useable footage table that registers a plurality of subjects to be photographed; a subject detection unit 140 that detects the subjects registered in the useable footage table from a photographed image acquired by an image pick-up device 107; an evaluation item definition table and point definition tables 1 to 3 in which evaluation criteria are set for evaluating states of the subjects detected by the subject detection unit 140; and a control unit 121 that, for each of the subjects registered in the useable footage table, compares a face and other body parts than the face of the subject detected by the subject detection unit 140 with the evaluation criteria to determine an evaluation value (useable footage) representing the quality of the photographed image. The control unit 121 displays the obtained evaluation value on a display 131 for each of the subjects registered in the useable footage table.SELECTED DRAWING: Figure 14
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method and program thereof. [Background technology]

[0002] There is known an imaging device that detects the state of a subject, takes a photograph when the subject is in a predetermined state, and counts the number of times the photographs have been taken. For example, an imaging device disclosed in Patent Document 1 automatically takes a photograph when it detects that the subject is smiling. When the number of captured smiling images of a smiling subject reaches a reference number, this imaging device automatically takes a photograph when it detects a facial expression (crying face, troubled face, etc.) or a movement (gesture) other than a smile of the subject.

[0003] In addition, when the imaging device disclosed in Patent Document 2 acquires a captured image, it performs subject recognition processing in the captured image, judges whether the state of the subject is a specific state set in advance, and increments and stores the number of times the subject has been captured when it is judged to be in the specific state. This allows the photographer to easily grasp the number of times each subject has been captured in a specific state while capturing multiple subjects. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-273281 A [Patent Document 2] JP 2009-290819 A Summary of the Invention [Problem to be solved by the invention]

[0005] Photographers often require images that include not only the face of a person, but also other body parts such as the hands and feet, half the body, and the whole body. For example, when photographing children's performances or competitions at school events such as athletic meets, it is often desirable to have images that show not only the children's faces, but also the movements and gestures of the hands and feet, and the movements of the whole body.

[0006] In this case, it is important for evaluating the quality of the captured image that information other than the person's face is captured at the same time as the person's face. In other words, the quality evaluation of the captured image needs to include information other than the person's face in addition to the person's face. In addition, it is desirable to obtain captured images of the subjects in a desired state evenly for each of the multiple people. Therefore, if the quality of the captured images obtained for each person can be easily grasped during shooting, the photographer can prioritize photographing the people for whom high-quality captured images cannot be obtained.

[0007] In response to such demands, the imaging device described in Patent Document 1 limits gestures that can be detected to facial movements such as yawning, sneezing, winking, etc. Furthermore, Patent Document 2 adds up the number of times images are taken according to whether or not a predetermined condition is met, but does not mention quality evaluation of the captured images.

[0008] An object of the present invention is to provide a technique that makes it possible to acquire high-quality photographed images including faces and other body parts of a plurality of people without bias. [Means for solving the problem]

[0009] The imaging device of the present invention is characterized in that it comprises an imaging means, a registration means for registering multiple subjects to be photographed, a detection means for detecting subjects registered in the registration means from a photographed image acquired by the imaging means, a setting means for setting evaluation criteria for evaluating the state of the subjects detected by the detection means, an evaluation means for determining an evaluation value representing the quality of the photographed image for each of the multiple subjects by comparing the face and body parts other than the face of the detected subject with the evaluation criteria, and a control means for displaying the sum of the evaluation values ​​for each of the multiple subjects on a display device. Effect of the Invention

[0010] According to the present invention, it is possible to obtain high-quality photographed images including faces and other body parts of a plurality of people without bias. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a schematic configuration of an imaging apparatus according to an embodiment. [Diagram 2] FIG. 2 is a front view illustrating an image arrangement of an imaging element included in the imaging device. [Diagram 3] 1 is a front view of one imaging pixel and a cross-sectional view taken along the arrow aa in the front view. [Figure 4] 4 is a cross-sectional view showing the relationship between the exit pupil of the imaging optical system and the photoelectric conversion unit of the imaging pixel. FIG. [Diagram 5] FIG. 2 is a diagram illustrating pupil division by an image sensor. [Figure 6] 5A and 5B are schematic diagrams illustrating the relationship between the defocus amount and the phase difference of a focus detection signal. [Figure 7] 2 is a schematic diagram illustrating a focus detection area from which an image sensor acquires a focus detection signal; FIG. [Figure 8] 4 is a flowchart of a photographing operation in the imaging device. [Figure 9] 13 is a flowchart showing details of the imaging process in S806. [Figure 10] FIG. 13 is a diagram showing an example of a shot result table indicating the total shot result for each subject. [Figure 11] 13A and 13B are diagrams illustrating an example of an evaluation item definition table and an evaluation point definition table. [Figure 12] FIG. 2 is a diagram for explaining body parts of a subject set in an evaluation point definition table 1. [Figure 13] FIG. 2 is a diagram showing an example of a photographed scene. [Figure 14] FIG. 13 is a diagram showing a display example of a shot content table. [Figure 15] 13 is a flowchart showing details of the subject tracking AF process in S804. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] 1 is a diagram showing a schematic configuration of an image capturing device 100 according to an embodiment. The image capturing device 100 is specifically a digital camera, but is not limited to this, and may be a digital video camera or a mobile terminal (electronic device) such as a smartphone having an image capturing function and a function for analyzing captured images.

[0014] The imaging device 100 includes a first lens group 101, an aperture 102, a second lens group 103, a third lens group 105, an optical low-pass filter 106, a zoom actuator 111, an aperture actuator 112, and a focus actuator 114. The imaging device 100 also includes a focus drive circuit 126, an aperture drive circuit 128, a strobe device 115, an AF assist light emitting unit 116, a strobe control circuit 122, an assist light drive circuit 123, a control unit 121, an image sensor 107, and an image sensor drive circuit 124. The imaging device 100 further includes a shutter 108, an image processing unit 125, a display unit 131, a group of operation switches 132, a flash memory 133, a subject detection unit 140, a storage unit 141, an attitude acquisition unit 142, and an external device interface unit 143.

[0015] The first lens group 101, the aperture 102, the second lens group 103, the third lens group 105, and the optical low-pass filter 106 constitute an imaging optical system. The first lens group 101, which is disposed closest to the subject (front side) in the imaging optical system, is held so as to be movable in the optical axis direction. The amount of light incident on the imaging element 107 can be adjusted by adjusting the aperture diameter of the aperture 102. The second lens group 103 moves in the optical axis direction together with the aperture 102, and performs a magnification change operation (zoom operation) together with the first lens group 101 moving in the optical axis direction. The third lens group 105 is a focus lens that moves in the optical axis direction to perform focus adjustment. The optical low-pass filter 106 is an optical element for reducing false colors and moire in a captured image.

[0016] The zoom actuator 111 rotates a cam barrel (not shown) around the optical axis (centered on the optical axis) to move the first lens group 101 and the second lens group 103 in the optical axis direction by a cam provided on the cam barrel, thereby performing a magnification change operation. The aperture actuator 112 adjusts the amount of light incident on the image sensor 107 by driving a plurality of light-shielding blades (not shown) of the aperture 102 in the opening and closing directions. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform a focusing operation (focusing) on ​​the subject.

[0017] The focus driving circuit 126 drives the focus actuator 114 in accordance with a control signal from the control unit 121, and moves the third lens group 105 in the optical axis direction. This allows a focusing operation on a predetermined subject to be performed. The aperture driving circuit 128 drives the aperture actuator 112 in accordance with a control signal from the control unit 121, and this adjusts the aperture diameter of the aperture 102. The zoom driving circuit 129 drives the zoom actuator 111 in accordance with a zoom operation by the photographer (user). In this embodiment, the imaging optical system, each actuator, and each driving circuit are integrally provided with the main body of the imaging device 100 equipped with the image sensor 107, but are not limited to this and may be configured as an interchangeable lens that is detachable from the main body of the imaging device 100.

[0018] The strobe device 115 has a light-emitting element such as a xenon tube or an LED, and irradiates light onto the subject. The AF auxiliary light emission unit 116 has a light-emitting element such as an LED, and improves focus detection performance for dark subjects or low-contrast subjects by projecting an image of a mask having a predetermined opening pattern onto the subject via a projection lens. The strobe control circuit 122 controls the strobe device 115 to emit light in synchronization with exposure timing for still image shooting. The auxiliary light drive circuit 123 controls the AF auxiliary light emission unit 116 to emit light in synchronization with focus detection operation.

[0019] The control unit 121 has a CPU, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, etc., and controls the overall operation of each unit constituting the imaging device 100 by expanding various programs stored in the ROM into the RAM. The imaging element 107 is composed of a two-dimensional CMOS photosensor including a plurality of pixels that photoelectrically converts a subject image formed by the imaging optical system and its peripheral circuits, and is disposed on the imaging plane of the imaging optical system. The imaging element drive circuit 124 receives a command from the control unit 121 and controls the operation of the imaging element 107. As a result, the imaging element 107 A / D converts an analog signal generated by photoelectric conversion, and transmits the generated digital signal (image data) to the control unit 121.

[0020] The shutter 108 includes a focal plane shutter and a shutter drive circuit, and the shutter drive circuit drives the focal plane shutter in accordance with a control signal from the control unit 121. The focal plane shutter blocks light from the image sensor 107 while a signal is being read out from the image sensor 107. During exposure of the image sensor 107, the focal plane shutter is opened to guide incident light to the image sensor 107.

[0021] The image processing unit 125 performs a predetermined image processing on the image data stored in the RAM in the control unit 121. Examples of image processing performed by the image processing unit 125 include so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing, but are not limited to these. The image processing unit 125 further detects the positions of body parts (including joints) of each subject, the positions and size information of specific objects, the center of gravity of the subject, and position information of the face and eyes from the image, and stores the various detected information together with the image data in the RAM in the control unit 121. Note that the various information detected by the image processing unit 125 may be used for image processing (for example, white balance adjustment processing, etc.).

[0022] The display unit 131 has a display device such as a liquid crystal panel, and displays information about the shooting mode of the imaging device 100, a live view video, a captured image for confirmation, an index of the focus detection area, an in-focus image, etc. The operation switch group 132 includes a power switch, a release switch, a zoom operation switch, a shooting mode selection switch, etc., and is operated by the photographer. Note that a touch panel is superimposed on the display device of the display unit 131, and the photographer can perform various operations by touching the touch panel. The flash memory 133 is a recording medium that is detachable from the imaging device 100, and records captured images.

[0023] The subject detection unit 140 detects a subject from a live view image (hereinafter referred to as "LV image") based on dictionary data generated by machine learning (see the explanation of S804 (subject tracking AF processing) described later). The subject detection unit 140 also detects a subject from a captured image (an image stored in the flash memory 133 in S905 described later). The subject detection unit 140 is configured, for example, with different CNNs (convolutional neural networks) that have been machine-learned, and estimates the position of a subject included in a captured image. The subject detection unit 140 may be realized by a GPU (graphics processing unit) or a circuit specialized for estimation processing by CNN.

[0024] The machine learning of CNN is performed by any method. For example, a configuration may be adopted in which a predetermined computer such as a server performs machine learning of CNN, and the imaging device 100 acquires the trained CNN from the predetermined computer. Alternatively, the predetermined computer may input training image data and perform supervised learning using the position of a subject corresponding to the training image data as teacher data, thereby training CNN and generating a trained CNN. The training of CNN may be performed by the imaging device 100.

[0025] In this embodiment, the subject detection unit 140 detects a person as a subject and detects body parts of the person, such as the face, eyes, hands, feet, and torso. Examples of body parts to be detected will be described later. The subject detection unit 140 estimates the subject position (person position) in the captured image based on image data of the captured image and dictionary data. The dictionary data for detecting the entire person and the dictionary data for detecting body parts such as the person's face may be stored separately in the storage unit 141.

[0026] Furthermore, the subject detection unit 140 performs subject detection while sequentially switching dictionary data for each subject. Dictionary data is data in which features of a subject are registered. The dictionary data for each subject is stored in the storage unit 141. The control unit 121 determines which dictionary data to use from among a plurality of dictionary data for subject detection based on the subject priority set in advance and the settings of the imaging device 100.

[0027] The subject detection unit 140 may estimate not only the subject position but also the size, reliability, etc., and output the estimated information, or may output other information. The subject detection unit 140 may detect the subject by identifying an individual whose face has been registered in advance (personal authentication). In this case, the imaging device 100 is provided with an authentication image registration mode (face registration mode), and in the authentication image registration mode, feature information indicating at least the facial feature amount of the person to be registered is registered in the dictionary data. In the personal authentication, first, the face of the person present in the captured image and its organ detection (for example, organ detection such as eyes and mouth) are performed to extract the facial feature amount of the person, and the similarity with the facial feature amount registered in the dictionary data in advance is calculated. Then, if the obtained similarity is equal to or greater than a predetermined threshold, it is determined that the face of the person present in the captured image is the face of the person registered in the dictionary data.

[0028] Also, the subject detection unit 140 may detect a plurality of types of subjects. In this case, the subject detection unit 104 uses dictionary data for each type. For example, each dictionary data may include dictionary data for detecting a person as a subject, dictionary data for detecting an animal, dictionary data for detecting a vehicle, dictionary data for detecting a tool such as a ball, a glove, or a bat, etc.

[0029] As described above, the memory unit 141 stores dictionary data for each object and object recognition images for object recognition. The memory unit 141 also stores an evaluation item definition table and a score definition table that are referenced when evaluating the quality of a captured image, and a shot quality table that indicates the quality evaluation results of the captured image for each registered object. Details of these various tables will be described later.

[0030] The orientation acquisition unit 142 estimates the orientation of each of the multiple subjects detected by the subject detection unit 140 and acquires orientation information. As described above, the subject detection unit 140 detects subjects from the LV image and the captured image, so the orientation acquisition unit 142 also acquires orientation information of the subjects from the LV image and the captured image.

[0031] The content of the posture information to be acquired by posture acquisition unit 142 is set according to the type of subject, but in this embodiment, a person is assumed to be the subject, and posture acquisition unit 142 acquires the positions of a plurality of body parts of the subject. The body parts of the subject acquired by posture acquisition unit 142 will be described later with reference to Fig. 12 etc. Note that the posture estimation method is not particularly limited, and for example, the method described in the following document by Cao, Zhe et al. can be used.

[0032] Cao, Zhe, et al., "Realtime multi-person 2d pose estimation using part affinity fields.", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

[0033] The external device interface unit 143 connects an external device to the imaging device 100 to exchange information. Examples of the external device include, but are not limited to, a smartphone and an external monitor. The communication connection between the imaging device 100 and the external device may be a wired connection such as a USB, or a wireless connection such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The imaging device 100 may be configured to acquire, as necessary, a subject recognition image stored in the external device connected to the imaging device 100, an evaluation item definition table and a score definition table (to be described later), from the external device.

[0034] Next, the configuration of the image sensor 107 and focus detection by the image sensor 107 will be described with reference to FIGS. 2 to 7. FIG. 2 is a front view for explaining an image array of the image sensor 107, and shows a pixel array of 4 pixel columns by 4 pixel rows of the image sensor 107 as viewed from the subject side in the optical axis direction (z direction). One pixel unit 200 has four imaging pixels arranged in 2 rows by 2 columns. One pixel unit 200 has an imaging pixel (R pixel 200R) having a spectral sensitivity of R (red) at the upper left, an imaging pixel (G pixel 200G) having a spectral sensitivity of G (green) at the upper right and lower left, and an imaging pixel (B pixel 200B) having a spectral sensitivity of B (blue) at the lower right. Each imaging pixel has a first focus detection pixel 201 and a second focus detection pixel 202 divided in the horizontal direction (x direction).

[0035] In this embodiment, the pixel pitch P of the imaging pixels in the imaging element 107 is 4 μm, and the number of imaging pixels N is horizontal (x) × vertical (y) = 5575 columns × 3725 rows = approximately 20.75 million pixels. The pixel pitch PAF of the focus detection pixels is 2 μm, and the number of focus detection pixels NAF is horizontal × vertical = 11150 columns × 3725 rows = approximately 41.5 million pixels. However, such number of imaging pixels and arrangement of the imaging pixels of the imaging element 107 and each imaging pixel are only examples. For example, in the configuration of FIG. 2, each imaging pixel is divided into two in the horizontal direction, but it may also be divided into two in the vertical direction. In addition, each imaging pixel of the imaging element 107 has a first focus detection pixel 201 and a second focus detection pixel 202, but the imaging pixel and the first and second focus detection pixels may be provided as separate pixels. For example, the first and second focus detection pixels may be discretely arranged among a plurality of imaging pixels.

[0036] Fig. 3(a) is a diagram showing one imaging pixel (which may be any of the R pixel 200R, the G pixel 200G, and the B pixel 200B) when the imaging element 107 is viewed from the subject side in the optical axis direction. Fig. 3(b) is a cross-sectional view taken along the arrow aa shown in Fig. 3(a).

[0037] Each imaging pixel has one microlens 305 for collecting incident light. Each imaging pixel has photoelectric conversion units 301, 302 that are divided into N parts (two parts in this embodiment) in the x direction. The photoelectric conversion units 301, 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively. The centers of gravity of the photoelectric conversion units 301, 302 are decentered to the -x side and +x side, respectively, with respect to the optical axis of the microlens 305.

[0038] In the imaging pixel, a color filter 306 of any one of RGB colors is provided between the microlens 305 and the photoelectric conversion units 301, 302. The spectral transmittance of the color filter may be changed for each photoelectric conversion unit, or the color filter may be omitted. Light incident on the imaging pixel from the imaging optical system is collected by the microlens 305, dispersed by the color filter 306, received by the photoelectric conversion units 301, 302, and photoelectrically converted in the photoelectric conversion units 301, 302 to generate an analog signal.

[0039] Fig. 4 is a schematic diagram showing the relationship between the exit pupil of the imaging optical system and the photoelectric conversion units 301 and 302 of the imaging pixels using a cross-sectional view taken along the arrow bb in Fig. 3(a). Note that the x and y directions are reversed between the cross-sectional view taken along the arrow bb in Fig. 4 and the cross-sectional view taken along the arrow aa in Fig. 3(b).

[0040] A first pupil region 501, the center of gravity of which is decentered on the +X side of the exit pupil, is an area that is substantially conjugate with the light receiving surface of the photoelectric conversion unit 301 on the -x side of the imaging pixel by the microlens 305. The light flux that passes through the first pupil region 501 is received by the photoelectric conversion unit 301 (=first focus detection pixel 201). A second pupil region 502, the center of gravity of which is decentered on the -X side of the exit pupil, is an area that is substantially conjugate with the light receiving surface of the photoelectric conversion unit 302 on the +x side of the imaging pixel by the microlens 305. The light flux that passes through the second pupil region 502 is received by the photoelectric conversion unit 302 (=second focus detection pixel 202). The pupil region that can receive light in the entire imaging pixel is indicated by a pupil region 500 that combines the photoelectric conversion units 301, 302 (first focus detection pixel 201 and second focus detection pixel 202).

[0041] FIG. 5 is a diagram for explaining pupil division by the image sensor 107. The light beams passing through the first pupil region 501 and the second pupil region 502 are incident on each imaging pixel of the image sensor 107 at different angles, and are received by the first focus detection pixel 201 and the second focus detection pixel 202, which are divided into two. In this embodiment, output signals from the multiple first focus detection pixels 201 of the image sensor 107 are collected to generate a first focus detection signal, and output signals from the multiple second focus detection pixels 202 are collected to generate a second focus detection signal. Then, the output signals from the multiple first focus detection pixels 201 and the output signals from the multiple second focus detection pixels 202 are added together to generate an imaging pixel signal. In addition, the imaging pixel signals from the multiple imaging pixels are combined to generate an imaging signal for generating an image with a resolution corresponding to the number of effective pixels N.

[0042] FIG. 6 is a schematic diagram for explaining the relationship between the defocus amount of the imaging optical system and the phase difference (image shift amount) between the first focus detection signal and the second focus detection signal generated by the image sensor 107. As described with reference to FIG. 4 and FIG. 5, the exit pupil of the imaging optical system is divided into two parts, a first pupil region 501 and a second pupil region 502. The defocus amount is expressed by the absolute value |d| of the magnitude of the distance from the imaging position C of the incident light beam from the object 610, 620 to the imaging surface 600 of the image sensor 107. A front focus state in which the imaging position C is on the object side of the imaging surface 600 is represented by a negative sign (-), and a back focus state in which the imaging position C is on the opposite side of the object from the imaging surface 600 is represented by a positive sign (+). In the in-focus state in which the imaging position C is on the imaging surface 600, d=0. In the example of FIG. 6, the imaging optical system is in-focus (d=0) with respect to the object 610, and in front focus (d<0) with respect to the object 620. The combination of the front focus state (d<0) and the back focus state (d>0) is called the defocus state (|d|>0).

[0043] In the front focus state (d<0), the light beam from the subject 620 that passes through the first pupil region 501 (second pupil region 502) is once collected and then spreads to a width Γ1 (Γ2) centered on the center of gravity position G1 (G2) of the light beam, forming a blurred image (fuzzy image) on the imaging surface 600. This blurred image is received by the first focus detection pixel 201 (second focus detection pixel 202) of the image sensor 107, and a first focus detection signal (second focus detection signal) is generated. In other words, the first focus detection signal (second focus detection signal) is a signal that represents an image of the subject 620 blurred by the blur width Γ1 (Γ2) at the center of gravity position G1 (G2) of the light beam on the imaging surface 600.

[0044] At this time, the blur width Γ1 (Γ2) of the subject image increases approximately in proportion to the increase in the magnitude of the defocus amount |d|. Similarly, the magnitude (|p|) of the image shift amount p (= the difference G1-G2 between the center of gravity positions of the light beams) between the first focus detection signal and the second focus detection signal also increases approximately in proportion to the increase in the magnitude of the defocus amount |d|. In the back-focus state (d>0), the direction of the image shift between the first focus detection signal and the second focus detection signal is opposite to that in the front-focus state, but the same can be said about the front-focus state (d<0).

[0045] That is, as the defocus amount |d| increases, the image shift amount between the first and second focus detection signals increases. In this embodiment, focus detection is performed using an image plane phase difference detection method in which the defocus amount is calculated from the image shift amount between the first and second focus detection signals obtained using the image sensor 107.

[0046] 7 is a schematic diagram for explaining focus detection areas in the image sensor 107 from which the first and second focus detection signals are obtained. In FIG. 7, A(n,m) indicates the n-th focus detection area in the x direction and the m-th focus detection area in the y direction among a plurality of focus detection areas (n,m=1,2,3, totaling nine, three in each of the x direction and y direction) set in the effective pixel area 700 of the image sensor 107. The first and second focus detection signals are generated from output signals from a plurality of first and second focus detection pixels 201, 202 (not shown in FIG. 7) included in the focus detection area A(n,m). In FIG. 7, I(n,m) indicates an index for indicating the position of the focus detection area A(n,m) on the display unit 131.

[0047] Note that the nine focus detection areas A(n,m) (n,m=1,2,3) shown in FIG. 7 are merely an example, and the number, positions, and sizes of the focus detection areas A(n,m) are not limited. For example, at least one focus detection area may be set in a predetermined range centered on a position designated by the photographer or a subject position detected by the subject detection unit 140. In this embodiment, the focus detection areas are arranged so that focus detection results can be obtained with high resolution when acquiring a defocus map, which will be described later. For example, the image sensor 107 is arranged with a total of 9600 focus detection areas divided into 120 horizontally and 80 vertically.

[0048] The above configuration is common to the following first and second embodiments.

[0049] First Embodiment Next, the photographing operation by the imaging device 100, that is, a series of operations from displaying an LV image on the display unit 131 of the imaging device 100, through exposure for acquiring a still image, to storing the photographed image, will be described.

[0050] 8 is a flowchart of the photographing operation of the imaging device 100. Each process (step) indicated by an S number in this flowchart is realized by the CPU in the control unit 121 expanding a predetermined program stored in the ROM into the RAM and comprehensively controlling the operation of each unit (each block) constituting the imaging device 100. When the control unit 121 detects that the power switch of the imaging device 100 is turned on, it initializes the imaging device 100, and starts the process of S801 when the initialization is completed.

[0051] In S801, the control unit 121 sends a drive signal for the image sensor 107 to the image sensor drive circuit 124. This causes the image sensor 107 to start driving, and the control unit 121 acquires an image signal from the image sensor 107. Specifically, the control unit 121 acquires first and second focus detection signals from a plurality of first and second focus detection pixels included in each focus detection area shown in FIG. 7. The control unit 121 also adds up the first and second focus detection signals of all effective pixels of the image sensor 107 to generate an image signal, causes the image processing unit 125 to perform image processing on the image signal, and acquires image data from the image processing unit 125. Note that when the image sensor and the first and second focus detection pixels are provided separately, the control unit 121 acquires image data by performing a complementation process on the focus detection pixels.

[0052] In S802, the control unit 121 causes the image processing unit 125 to generate an LV image from the image data obtained in S801, and displays the generated LV image on the display unit 131. The LV image is a reduced image matched to the resolution of the display unit 131, and the photographer can adjust the shooting composition, exposure conditions, and the like while viewing the LV image. When the LV image is displayed on the display unit 131, the control unit 121 performs exposure adjustment based on the photometric value obtained from the image data. The exposure adjustment is realized by appropriately adjusting the exposure time, the opening amount of the diaphragm 102, and the gain of the output from the image sensor 107.

[0053] In S803, the control unit 121 determines whether or not a switch SW1, which instructs the start of preparation for imaging by half-pressing a release switch included in the operation switch group 132, has been turned on. If the control unit 121 determines that the switch SW1 is not turned on (No in S803), it repeats the process of S803 to monitor the timing when the switch SW1 turns on. On the other hand, if the control unit 121 determines that the switch SW1 is turned on (Yes in S803), it executes the process of S804.

[0054] In S804, the control unit 121 performs subject tracking AF processing. In the subject tracking AF processing, detection of a subject area using the obtained imaging signal and focus detection signal, setting of a focus detection area, and predictive AF processing for suppressing the influence of a time lag from focus detection processing to actual exposure are performed. Details of the processing in S804 will be described later.

[0055] In S805, the control unit 121 determines whether or not the switch SW2, which instructs the start of imaging processing, has been turned on by fully pressing the release switch. If the control unit 121 determines that SW2 is not on (No in S805), it executes the process of S803, and if it determines that SW2 is on (Yes in S805), it executes the process of S806.

[0056] In S806, the control unit 121 executes imaging processing (a series of processes from exposure for acquiring a still image by the imaging element 107 to storage of image data). Details of the processing in S806 will be described later.

[0057] In S807, the control unit 121 determines whether or not the power switch included in the operation switch group 132 has been turned off. If the control unit 121 determines that the power switch remains on (No in S807), it executes the process of S803, and if it determines that the power switch has been turned off (Yes in S807), it ends this process. Note that, in the first embodiment, the subject tracking AF process is performed after it is detected in S803 that SW1 is turned on, but the subject tracking AF process may be performed before SW1 is turned on.

[0058] Next, the imaging process in S806 will be described in detail with reference to a flowchart shown in FIG.

[0059] In S901, the control unit 121 determines conditions such as shutter speed, aperture value, ISO sensitivity, etc., and performs exposure control. The exposure control can be performed using brightness information acquired from the LV image data. The control unit 121 transmits the determined aperture value to the aperture drive circuit 128 to drive the aperture 102. The control unit 121 also transmits the determined shutter speed to the shutter 108 to open the focal plane shutter. Furthermore, the control unit 121 causes the image sensor 107 to accumulate charge during the exposure period via the image sensor drive circuit 124.

[0060] In S902, the control unit 121 causes the image sensor drive circuit 124 to read out the image sensing signals (read out the still image) from all pixels of the image sensor 107. The control unit 121 also causes the image sensor drive circuit 124 to read out one of the first and second focus detection signals from the focus detection region in the image sensor 107. The first focus detection signal or the second focus detection signal read out here is used to detect the focus state when the captured image is later played back. Note that by subtracting one of the first and second focus detection signals from the image sensing signal, the other focus detection signal can be obtained.

[0061] In S903, the control unit 121 causes the image processing unit 125 to perform defective pixel correction processing on the imaging data that was read out from the imaging element 107 and A / D converted in S902.

[0062] In S904, the control unit 121 causes the image processing unit 125 to perform image processing and encoding such as demosaic (color interpolation), white balance processing, gamma correction (tone correction), color conversion, and edge enhancement on the imaging data after the defective pixel correction processing in S903.

[0063] In S905, the control unit 121 stores the image data of the captured image obtained by the image processing and encoding processing in S904 as an image data file in the flash memory 133. The image data file of the captured image includes the image data itself and the first focus detection signal or the second focus detection signal acquired in S902.

[0064] In S906, the control unit 121 stores the characteristic information of the imaging device 100 in the flash memory 133 in association with the captured image stored in S905. The characteristic information of the imaging device 100 includes, for example, the following first to sixth pieces of information. The first piece of information is imaging conditions (aperture value, shutter speed, ISO sensitivity, etc.). The second piece of information is information on the image processing performed by the image processing unit 125. The third piece of information is information on the light receiving sensitivity distribution of the imaging pixels and focus detection pixels of the imaging element 107. This third piece of information is information on the sensitivity of the imaging element 107 according to the distance (position) on the optical axis from the imaging element 107. Since the third piece of information depends on the microlens 305 and the photoelectric conversion units 301 and 302, it may be information on these or information on the change in sensitivity with respect to the angle of incidence of light. The fourth piece of information is information on vignetting of the imaging light flux in the imaging device 100. The fifth piece of information is information on the distance from the mounting surface of the imaging optical system in the imaging device 100 to the imaging element 107. The sixth information is information relating to a manufacturing error of the imaging device 100.

[0065] In S907, the control unit 121 stores photographing lens characteristic information as characteristic information of the imaging optical system in association with the image data stored in S905 in the flash memory 133. The photographing lens characteristic information includes, for example, information on the exit pupil, information on a frame such as a lens barrel that blocks light beams, information on the focal length and F-number at the time of imaging, information on the aberration of the imaging optical system, information on manufacturing errors of the imaging optical system, and information on the position of the third lens group 105 at the time of imaging (subject distance).

[0066] In S908, the control unit 121 stores information (image related information) related to the image data stored in S905 in the flash memory 133. The image related information includes, for example, information related to focus detection operation before image capture, information related to subject movement, and information related to focus detection accuracy (defocus map). In addition, in S908, the control unit 121 performs quality evaluation of the captured image (calculation of the shot result, which will be described later), updates the shot result table, which will be described later, with the obtained result, and stores it.

[0067] In S909, the control unit 121 displays a preview of the captured image on the display unit 131. This allows the photographer to easily check the captured image. After completing the process of S909, the control unit 121 ends this subroutine and executes the process of S807.

[0068] Next, a description will be given of control in the imaging device 100 for automatically evaluating the quality of the captured image stored in S905 (hereinafter referred to as "capture quality") based on a preset photographing policy and displaying the evaluation result on the display unit 131. The evaluation result of the capture quality is expressed as an evaluation value (hereinafter referred to as "shot result") that quantifies the capture quality, and the contents of the shot result table are updated with the obtained shot result.

[0069] Before photographing a subject (person) to be photographed, the photographer registers an authentication image in the imaging device 100 to enable image authentication of the subject. For example, it is assumed that an image of the face (and the upper body or entire body as necessary) of the subject to be photographed is stored in the flash memory 133. In this case, the photographer operates the operation switch group 132 to set the imaging device 100 to an authentication image registration mode, specifies an image of the subject to be photographed, and registers it as the authentication image. Note that in the authentication image registration mode, the subject to be registered may be photographed, and the obtained captured image may be registered as the authentication image.

[0070] 10 is a diagram showing an example of a shot amount table showing the total shot amount for each subject and the score for each evaluation item (details will be described later). The shot amount table is stored in flash memory 133, and holds the authentication image (face image data) of the subject as well as the total shot amount. As will be described later in detail, the shot amount is calculated for each captured image, and the obtained shot amount is added for each corresponding subject to determine the total shot amount for each subject.

[0071] The shot quality table in Fig. 10 shows an example in which three people are registered as subjects to be photographed using the above-mentioned authentication image registration method. The shot quality table includes a value that uniquely identifies the registered subject (A, B, C in Fig. 10) and the authentication image used to register the subject. "Focus," "Cutout," and "Size" in the shot quality table correspond to "Item 2," "Item 1," and "Item 3," respectively, which will be described later with reference to Fig. 11.

[0072] When the footage table has already been created, a subject can be additionally registered, and the numerical data of the footage table can also be reset or the footage table can be deleted. The footage table may be configured so that when a subject that has not been registered in advance is found in a photographed image, the footage table is automatically registered using the photographed image of the subject. The footage table may be stored in an external device such as a smartphone connected via the external device interface unit 143, or may be stored in the flash memory 133.

[0073] Next, the photographer sets a photography policy before photographing a subject (person) to be photographed. In the first embodiment, the photographed image (the image stored in S905) is evaluated and the result is obtained as the amount of photographed images, so that setting the photography policy is, in other words, setting a standard for evaluating the photographed image.

[0074] Fig. 11(a) is a diagram showing an example of an evaluation item definition table, and Figs. 11(b) to (d) are diagrams showing examples of score definition tables 1 to 3 for defining scores for each evaluation item in the evaluation item definition table.

[0075] The shooting policy is set using evaluation items for automatically evaluating the shooting quality of a captured image, weighting of each evaluation item, and points for each evaluation item. In the first embodiment, the photographer sets the shooting policy using the evaluation item definition table and point definition tables 1 to 3.

[0076] The photographer operates the operation switch group 132 to set the imaging device 100 to an imaging policy setting mode. Score definition tables 1, 2, and 3 in Fig. 11(b) to (d) play an auxiliary role for making detailed settings for items 1 to 3 in the evaluation item definition table in Fig. 11(a). The evaluation item definition table and score definition tables 1 to 3 are stored in the storage unit 141 of the imaging device 100, but are not limited to this and may be stored in the flash memory 133 or an external device connected via the external device interface unit 143.

[0077] The evaluation item definition table sets three evaluation items 1 to 3 that are considered important in the quality of a captured image, and weights that indicate which of evaluation items 1 to 3 the photographer considers important are set as numerical values. The evaluation item definition table lists the focus state (focus / blur) as "item 1," the subject's cut-off and overlay state (obstruction state) as "item 2," and the subject's shooting size as "item 3," and the weights of items 1 to 3 are set to 6, 3, and 1, respectively. This indicates that the photographer considers the focus state to be the most important, followed by the absence of cut-off and overlay of the subject.

[0078] The photographer can set the desired photography policy by setting the weight value of the evaluation item definition table to a predetermined value. When two or more items are defined, weighting is performed, and when three or more items are defined, the same weighting may be set for at least two of them.

[0079] 11(b), body parts that are considered important to be included in a photographed image among a plurality of body parts of a subject are set in score definition table 1. First, the body parts will be described.

[0080] Fig. 12 is a diagram for explaining the body parts (including body joints) of a subject that are set in score definition table 1. In Fig. 12, "NH" represents the top of the head (face), "NA" represents the arm (elbow), "NB" represents the neck / shoulder, "ND" represents the hand, "NC" represents the waist (groin), "NL" represents the leg (knee), and "NF" represents the foot. In score definition table 1, the order in which these body parts are preferably shown in the captured image is set according to three priorities. The photographer sets for each body part in score definition table 1 whether it is necessary (priority: high), optional (priority: medium), or not necessary (priority: low) to be photographed, with "○ (priority: high)", "△ (priority: medium)", or "× (priority: low)".

[0081] The priority that can be set varies depending on the body part, and information on the priority that can be set is also stored in score definition table 1. Here, it is not possible to select "X" for the top of the head, neck / shoulders, arms, hands, waist, legs, and upper body. Such settings are not essential. However, it is important that at least the top of the head (face) appears in the captured image, and therefore the priority for the top of the head (face) may always be set to "high."

[0082] In score definition table 1 in Fig. 11(b), the whole body, waist, legs, and feet are set to "△" and the rest are set to "◯". This indicates that while it is optional for the photographer to capture the whole body of the subject, it is important that at least the upper body (hands, arms, and top of the head) is captured. Note that score definition table 1 in Fig. 11(b) is configured to set priorities for all of the body parts shown in Fig. 12, but it may also be configured to allow priorities to be set for only some of the body parts.

[0083] Fig. 13(a) is a diagram showing an example of a captured image, in which the legs (NL), waist (NC), and feet (NF) of a subject are hidden by an obstruction 1310. The posture acquisition unit 142 determines that the conditions set in the score definition table 1 in Fig. 11(b) are satisfied even if the whole body, waist (NC), legs (NL), and feet (NF) are not captured. Note that the posture acquisition unit 142 can recognize and acquire information on the positions of the body parts of a person shown in Fig. 12 and the connection state between the body parts from the captured image using a known posture estimation method described in the above-mentioned literature by Cao, Zhe, et al.

[0084] Fig. 13(b) is a diagram showing another example of a captured image, in which the subject's waist (NC), legs (NL), waist (NC) and feet (NF), as well as the arms (NA) and hands (ND) are hidden by an obstruction 1320 (or another subject). In this case, the posture acquisition unit 142 cannot acquire the neck / shoulders (NB), arms (NA) and hands (ND) that are required to appear in the captured image, and therefore it is determined that the conditions set in the score definition table 1 in Fig. 11(b) are not satisfied. Note that the obstructions 1310 and 1320 may be other people, etc.

[0085] The score for item 2 is "1" if the condition set in score definition table 1 is met, and "0" if the condition is not met. Therefore, the score for item 2 is "1" for the captured image in Fig. 13(a) and "zero (0)" for the captured image in Fig. 13(b).

[0086] In addition, when the "whole body" of the body part in the score definition table 1 is set to "○ (priority: high)", other body parts may also be automatically set to "○". In the score definition table 1 and FIG. 12, an example is shown in which the positions of the top of the head, neck / shoulders, elbows, hands, waist, knees, and feet are acquired as body parts, but the positions of the body parts may be some of these, or other positions may be acquired. In addition to the positions of the body parts, information such as axes connecting the body parts may be used, and any information that can acquire which part of the subject is being photographed can be used as posture information.

[0087] In the score definition table 2 in Fig. 11(c), scores for the degree of focus (degree of focus), that is, the score of item 1, are defined. The degree of focus is classified into three ranks, "small", "medium", and "large", and the amount of blur and a score are defined for each rank. The amount of blur can be determined using the first focus detection signal or the second focus detection signal read out in S902 and stored in the flash memory 133 in S905. Alternatively, a defocus amount normalized by the permissible circle of confusion diameter, F-number, etc. included in the characteristic information of the imaging device 100 stored at the time of shooting can be used.

[0088] 11(c), the degree of blur "small" is set to a score of "1" when the amount of blur is 4 or less, the degree of blur "medium" is set to a score of "0.5" when the amount of blur is greater than 4 and less than 10, and the degree of blur "large" is set to a score of "zero (0)" when the amount of blur is greater than 10. In other words, the smaller the amount of blur, the larger the score is set.

[0089] 11(d), the size of the subject in the captured image is ranked according to the angle of view ratio, and a score is set for each rank. In other words, in the score definition table 3, the score of the item 3 is defined for each rank.

[0090] Here, the angle of view ratio is the ratio (percentage) of the subject's shooting size (image area) to the vertical angle of view. The subject's shooting size can be calculated by detecting the body parts of the subject in the captured image. When capturing an image with the imaging device 100 in a vertical position instead of the normal horizontal position, the imaging device 100 detects vertical shooting and sets a different angle of view from the horizontal shooting as the reference angle of view. In the score definition table 3 in FIG. 11(d), the score for the shooting size "large" is set to "1" when the subject's angle of view ratio is 50% or more, and the score for the shooting size "medium" is set to "0.5" when the subject's angle of view ratio is 30% or more and less than 50%. Also, the score for the shooting size "small" is set to "zero (0)" when the subject's angle of view ratio is less than 30%.

[0091] Using the weights defined for items 1 to 3 in the evaluation item definition table and the scores in the score definition tables 1 to 3, the quality of the shots Tp for each subject and each captured image registered in the quality of the shots table is calculated, for example, as the sum of values ​​obtained by multiplying each score by each weight, using the following formula 1. In the following formula 1, f1, f2, and f3 are the scores for items 1, 2, and 3, respectively, and w1, w2, and w3 are the weights for items 1, 2, and 3, respectively.

[0092]

number

[0093] 11 are merely examples, and other ranks and values ​​may be used. The settings in the item definition table and score definition tables 1 to 3 may be the same for subjects A to C registered in the footage table (collectively settable), or may be different (settable for each subject).

[0094] During shooting by the imaging device 100, a shot quality table (see FIG. 10) is displayed on the display unit 131 together with the LV image (or a preview image of the shot image).

[0095] Fig. 14 is a diagram showing an example of a state in which a shot result table 1400 is superimposed on an LV image on the display unit 131. Note that only the shot result table 1400 in the LV image in Fig. 14 will be described here, and subjects B and C in the LV image will be described later.

[0096] The shot quality table 1400 displays subjects in ascending order of total shot quality. The total shot quality of each subject is expressed as the sum of the shot quality calculated for each photographed image of each subject. The values ​​of items 1 to 3 in the shot quality table represent the total score for all images.

[0097] For example, in the shot quality table 1400, it is assumed that the number of shot images is three for each of subjects A to C. For subject B, it is shown that the scores for items 1 to 3 were "1" for all shot images. For subject A, for example, it is shown that all shot images were scored 1 in item 3, but there were two shot images scored 1 in items 2 and 1 (or one shot image scored 1 and two shot images scored 0.5). For subject C, it is shown that for example, item 2 had one shot image scored 1 and two shot images scored 0, and items 1 and 3 had one shot image scored 1 and two shot images scored 0.5 (or two shot images scored 1 and one shot image scored 0).

[0098] Based on the shot result table 1400, the photographer next shoots the subject C with the smallest total shot result as the main subject, and by appropriately shifting the shooting target to the subject with the smallest total shot result, a shooting result can be obtained in which the difference in the total shot results between the subjects A to C is kept small. The method by which the photographer identifies the subject with the smallest total shot result in the LV image will be described later.

[0099] In addition, when there is a difference in the total amount of footage of multiple subjects, it is not necessary to always select the subject with the smallest total amount of footage as the next main subject. For example, if the difference in the total amount of footage between the first and second smallest subjects is equal to or less than a preset threshold, the subject with the second smallest total amount of footage may be selected as the subject to be photographed.

[0100] Next, the subject tracking AF process executed in S804 by the control unit 121 of the imaging device 100 when the photographer performs manual shooting will be described. Fig. 15 is a flowchart of the subject tracking AF process (a subroutine of S804) executed in the imaging device 100.

[0101] In S1501, the control unit 121 calculates the amount of image shift between the first and second focus detection signals obtained in each of the multiple focus detection areas described with reference to Fig. 7, and calculates a defocus amount for each focus detection area from the obtained image shift amount. As described above, the focus detection areas are arranged on the image sensor 107 at a total of 9,600 points, divided into 120 horizontally and 80 vertically, and a group of focus detection results obtained from the total of 9,600 focus detection areas is called a defocus map.

[0102] In S1502, subject detection unit 140 performs subject detection processing on the LV image (frame) upon receiving a command from control unit 121. Here, depending on the state of the obtained LV image, there are cases where the subject cannot be detected, in which case subject detection unit 140 performs processing using a method such as template matching to estimate the position of the subject.

[0103] As shown in the LV image of Fig. 14, when a person's face is detected by the subject detection process, a frame is drawn around the detected person's face. In the case of manual shooting by the photographer, the photographer generally changes the shooting direction so that at least one person's face is detected. Therefore, in the following description, it is assumed that at least one person is detected in S1502.

[0104] In S1503, the subject detection unit 140, which has received a command from the control unit 121, performs personal authentication processing as subject authentication processing. The personal authentication processing is performed by determining whether or not the face of a person present in the LV image is the face of a person registered in the shot quality table.

[0105] As described above, in S1502, a frame is drawn around the detected face. If the face detected in the LV image in S1503 is not registered in the footage table, the control unit 121 displays the frame in white, and if the face is registered in the footage table, the control unit 121 displays the frame in orange. In this way, by changing the color of the frame drawn around the detected face, the photographer can distinguish whether the detected face is a subject registered in the footage table or not.

[0106] In S1504, the posture acquisition unit 142, which has received a command from the control unit 121, acquires posture information of one or more subjects detected by the subject detection unit 140. Specifically, the posture acquisition unit 142 acquires two-dimensional coordinates (x, y) of each of NH (top of the head), NA (arms), NB (neck / shoulders), ND (hands), NC (waist), NL (legs), and NF (feet) in the LV image. Here, the unit of the two-dimensional coordinates (x, y) is pixels.

[0107] The process of S1504 is performed in consideration of the process of determining the main subject in the next S1505, but is not necessarily required. This is because the score definition table 1 described above is set as an evaluation standard for the captured image, and is not set for evaluating the state of the subject in the LV image. However, incorporating control to detect posture information from the LV image and set as the main subject a subject that satisfies all of the body parts set to "high" priority in the score definition table 1 from among the subjects to be registered in the yield table will contribute to the acquisition of a captured image with a high yield.

[0108] In S1505, the control unit 121 performs a main subject determination process, that is, a process of selecting a subject to be tracked so that the focus state (in focus state) is maintained. In S1505, the control unit 121 performs display control such as changing the color of the frame superimposed on the face of the person selected as the main subject to, for example, red, or blinking the orange frame displayed in S1503. This allows the photographer to easily recognize which subject among the subjects registered in the yield table has been selected as the main subject.

[0109] In the first S1505, since the total footage of subjects A to C is zero (0) in the footage table, any one of subjects A to C may be selected as the main subject. For example, a person who is more likely to produce a large amount of footage among subjects A to C may be selected as the main subject. One example is a person who satisfies all of the body parts set to "high" priority in score definition table 1 when posture information is detected in S1504, or a person who occupies a large area in the LV image.

[0110] In S1505 from the second time onwards, in principle, the main subject is selected so that the total footage of the multiple subjects registered in the footage table is as equal as possible. That is, the control unit 121 selects the person with the smallest total footage as the main subject. In the case of the LV image shown in Fig. 14, the subject C with the smallest total footage in the footage table 1400 is preferentially selected as the main subject.

[0111] In S1506, the control unit 121 sets a focus detection area for the main subject determined in S1505. In the first embodiment, because the person's face is detected in S1502 and a captured image in which the face is blurred or shaken is undesirable, in this embodiment, the focus detection area is set to the person's face (head). When subject C is selected as the main subject in the LV image in Fig. 14, of the multiple focus detection areas shown in Fig. 7, the focus detection area inside the frame drawn to surround the head of subject C is selected.

[0112] In S1507, the control unit 121 acquires a focus detection signal in the focus detection area set in S1506 and calculates a defocus amount. The defocus amount is not limited to the focus detection result in the focus detection area set in S1506, and may be calculated by adding focus detection results in a plurality of surrounding focus detection areas. Note that the control unit 121 may acquire the focus detection result in S1501 in the focus detection area set in S1506, and at that time, may also acquire the focus detection results in the neighboring focus detection areas.

[0113] In S1508, the control unit 121 performs predictive AF processing using the defocus amount obtained in S1507 and multiple defocus amounts that are time-series data of the timings at which past focus detection was performed. Predictive AF control is necessary when there is a time lag between the focus detection timing and the exposure timing. In predictive AF control, the image plane position of the subject in the optical axis direction is predicted at the exposure timing when a predetermined time has elapsed since the focus detection timing. In predicting the image plane position of the subject, a formula for a predictive curve is obtained by performing multivariate analysis (e.g., the least squares method) using historical data of the image plane position of the subject in the past and the time. The predicted image plane position of the subject can be obtained by substituting the predicted time of the exposure timing of the captured image into the formula for the predicted curve obtained.

[0114] It is preferable that the subject to be subjected to predictive AF control is not only the main subject but also a plurality of detected subjects. In addition, when the position of a body part is acquired in S1504, the position of the subject may be predicted using the focus detection result for the body part and time series data of the position of the body part, thereby making it possible to predict not only the position of the face (head) of the subject but also the position of the body part of the subject. Furthermore, the three-dimensional position of the subject may be predicted. For example, when the screen is defined as the xy direction and the optical axis direction is defined as the z direction, the position of the subject at the exposure timing may be predicted from the xy position of the subject obtained by subject detection in S1502 and time series data of the z direction position based on the defocus amount obtained in S1507.

[0115] In S1509, control unit 121 performs focus adjustment processing. Specifically, control unit 121 calculates the drive amount of the focus lens using the main subject determination result in S1505, the defocus amount obtained in S1507, and the result of the predicted AF processing in S1508. Control unit 121 generates a control signal for driving focus actuator 114 and transmits it to focus actuator 114, and focus actuator 114 moves third lens group 105, which is the focus lens, in the optical axis direction. This makes it possible to focus on the predicted position of the subject determined in S1508.

[0116] When the process of S1509 ends, the control unit 121 ends the AF control subroutine and executes the process of S805 (see FIG. 8). When the control unit 121 detects that the switch SW2 is turned on in S805, it sends an imaging command signal to the image sensor drive circuit 124, which causes imaging processing to be executed in S806. The control unit 121 stores the captured image (image data) in the flash memory 133 or the like in S905 in S806, and evaluates the quality of the captured image based on the evaluation item definition table and score definition tables 1 to 3 in S908 in S806. The control unit 121 updates the contents of the shot result table stored in the flash memory 133 with the obtained evaluation result. In addition, the control unit 121 selects the next shooting candidate along with the update of the shot result table, the details of which will be described later.

[0117] In the above description, it is assumed that a plurality of subjects are photographed, but the use of the footage table is also useful when the main subject is one person. In other words, by obtaining the total footage according to the evaluation item definition table and score definition tables 1 to 3 and making it possible to refer to the footage table, the photographer can easily determine to what extent he or she has been able to photograph the subject in the desired state. In addition, in the above description, the facial expression of the subject is not scored and added to the footage, but the facial expression may be added as an evaluation item to the evaluation item definition table, and a score definition table related to the facial expression may be provided.

[0118] As described above, according to the first embodiment, taking into consideration the quality of the captured image including the face and other body parts such as the hands and feet, half body, and whole body, the image is captured so that the total amount of images captured for the multiple subjects is not significantly biased. This makes it possible to suppress the occurrence of cut-offs and overlaps of the subjects in the captured image, and obtain high-quality images capturing the movements of the parts other than the face without bias for the multiple subjects. According to the first embodiment, it is possible to provide a useful image capture assistant means in situations where it is required to capture the movements of the subjects other than their faces at school events such as athletic meets, etc., with equal image capture with suppressed variation in the number of captured images for each subject.

[0119] Incidentally, with the recent widespread use of digital cameras, digital video cameras, and smartphones, there are more and more opportunities to retouch images after shooting using video editing software, or to create images using computational photography that utilizes AI technology. In such cases, the captured image can be processed to adjust the blur and brightness of the subject to make the subject look more attractive or to make the scene or landscape more dazzling, thereby enhancing the appeal, artistic value, and added value of the photograph.

[0120] Computational photography is mainly used to adjust the blur, brightness, and color of an image. Therefore, it is desirable that the captured image is captured so that it can be processed without changing the shape of the subject. In other words, examples of images that are not processed by computational photography include so-called cut-out images, in which a part of the subject is cut off at the edge of the image, and so-called overlapping images, in which subjects are photographed overlapping each other. In other words, processing that changes the shape of the subject itself, such as processing a cut-out image into an image in which the entire body of the subject is captured, or processing an overlapping image into an image without overlapping, is rarely performed in computational photography.

[0121] In this way, it is desirable for the target image of computational photography to be an image in which the whole body is photographed, an image without any cut-offs, and an image without any overlapping of the subject. Therefore, in the above first embodiment, by incorporating at least one of the following two conditions, it is possible to efficiently obtain a photographed image suitable for drawing by computational photography after shooting.

[0122] That is, the first condition is to increase the weight of "cut-off" (item 2 in the item definition table) in the shot quality table and to take into consideration the total shot quality. The second condition is to make the process of S1504 mandatory and to select a subject that satisfies the conditions defined in score definition table 1 as the main subject in S1505, although score definition tables 1 to 3 are originally used to evaluate shot images. In this case, the posture information of the subject in S1504 is further used in the focus adjustment process in S1509. By shooting using at least one of these conditions, it is possible to obtain many shot images with little cut-off.

[0123] <Second embodiment> In the second embodiment, the control when the imaging device 100 performs automatic shooting according to the image recognition result of the subject and the total shot footage will be described. Note that, since the control (program) executed by the imaging device 100 is different between the first embodiment and the second embodiment, the following description will focus on the differences from the first embodiment.

[0124] When performing automatic shooting, the imaging device 100 is generally attached to an automatic camera platform or the like that can automatically change the shooting direction so that a subject search can be performed if a subject registered in the shooting quality table cannot be detected within the shooting angle of view.

[0125] When performing automatic shooting, the photographer operates the operation switch group 132 of the imaging device 100 to set the imaging device 100 to an automatic shooting mode, or an external device issues an instruction to execute automatic shooting to the control unit 121 via the external device interface unit 143. As in the first embodiment, the photographer sets a shooting policy in advance. In the automatic shooting mode, shooting is automatically performed according to the process of the flowchart in FIG. 8, and in automatic shooting, the photographer often does not look at the display unit 131. Therefore, it is not necessarily necessary to superimpose the shot result table 1400 on the LV image as shown in FIG. 14, and the shot result table 1400 may be held (stored) in the flash memory 133 or the like.

[0126] In the first embodiment, the item definition table and the score definition tables 1 to 3 are used to evaluate the state of the subject in the captured image in principle, but in the second embodiment, the definitions in the score definition tables 1 to 3 are also used to determine the shooting conditions. Note that the shot quality table is updated by evaluating the captured image in the same way as in the first embodiment, with some exceptions.

[0127] The following describes first to eighth control operations that can be performed when a plurality of subjects are detected simultaneously in S1502 during automatic photography.

[0128] In the first control, the control unit 121 detects posture information of a person within the shooting angle of view in S1504, and counts the body parts with high priority in the score definition table 1 for all subjects registered in the shooting quality table. Then, the control unit 121 obtains the shooting part ratio for each subject by dividing the number of body parts counted for each subject by the total number of body parts counted for each subject. The control unit 121 further performs an AF operation to select a main subject preferentially from subjects with a high shooting part ratio and to focus on the main subject. Note that the first control is substantially performed when two or more people among the multiple subjects registered in the shooting quality table are detected as subjects in the LV image.

[0129] In the second control, when it is determined that a plurality of subjects are within a range of a predetermined defocus amount by referring to the defocus map acquired in S1501, an AF operation is performed so that the plurality of subjects are brought into focus. Specifically, the control unit 121 issues a command to the aperture driving circuit 128 to drive the aperture actuator 112, and drives the aperture 102 to narrow the aperture (to increase the F value) and performs shooting. This is to ensure the depth of field, assuming that an editor will perform image processing after shooting. At this time, if the difference in the defocus amount of the plurality of subjects exceeds a predetermined range, the control unit 121, in principle, refers to the shot result table and performs an AF operation to focus on the subjects in order of the smallest total shot result, and performs shooting. On the other hand, in order to prioritize the AF speed in the imaging device 100, the shooting order of the subjects may be changed so that the driving direction of the third lens group 105, which is a focus lens, is a one-way operation.

[0130] In the third control, regardless of the total footage of each subject registered in the footage table, AF operation is performed to prioritize focusing on the subject that is closest to the imaging device 100. The third control is desirably applied when, for example, a scene is being photographed in which a subject is approaching from the back to the front, and the weight of item 3 (shooting size) in the evaluation item definition table is set to a larger value than the other items 1 and 2.

[0131] In the fourth control, for a photographed image in which a preset number of subjects or more are detected and the subject size or the subject's face size is equal to or smaller than a preset size, the amount of shots is not calculated regardless of the weighting setting value of item 3 (photography size) in the evaluation item definition table. This control is preferably applied in cases where it is desired to exclude photographs that are assumed to be taken equally, such as group photographs.

[0132] In the fifth control, when an unregistered subject is detected adjacent to a registered subject, the aperture 102 is driven to focus on both subjects simultaneously, and shooting is performed. This is because a subject adjacent to a registered subject is presumed to be related to the registered subject, and it is determined that it is desirable to select the subject as a main subject together with the registered subject. In this case, it is also desirable to configure the unregistered subject detected adjacent to the registered subject to be automatically added and registered in the shot quality table.

[0133] In the sixth control, a predetermined number of images are continuously shot (continuous shooting) while switching the main subject in order from the subject with the smallest total shot amount. This makes it possible to prevent unevenness in the number of shot images for each subject.

[0134] In the seventh control, when there is a large movement in the LV image such as multiple subjects regularly lining up and moving across the entire range of the shooting angle of view, the subjects registered in the shooting height table are focused on in ascending order of total shooting height or in order of closestness. An example of a shooting scene suitable for such AF control is shooting a march at an athletic meet. On the other hand, when there is no large movement in the subject, it is possible to determine whether a subject of a predetermined size or larger crosses or is blocked by using a defocus map or information on the body parts of the subject acquired by the posture acquisition unit 142, and to determine the timing when there is no crossing or blocking, and perform shooting according to the total shooting height. An example of a shooting scene suitable for such AF control is shooting a group competition at an athletic meet.

[0135] In the eighth control, the positional relationship of the limbs is determined using information on the body parts of the subject acquired by the posture acquisition unit 142, and a subject determined to be moving its limbs is preferentially focused on and photographed. This type of control is preferably applied in cases where a subject that is moving is to be photographed preferentially over a stationary subject. Also, when it is determined from the information on the body parts of the subject acquired by the posture acquisition unit 142 that multiple subjects are in an up-down positional relationship, it is preferable to preferentially focus on the upper subject and photograph it. This is because, for example, in performances or competitions performed by several people at an athletic meet, the person at the top is often the main character.

[0136] Thus, according to the second embodiment, in automatic photography with a preset photography policy, AF control and photography quality evaluation are performed according to the state of the subject (photography scene) within the photography angle of view, which can increase the efficiency of acquiring photographed images according to the preset photography policy in automatic photography.

[0137] Incidentally, in recent years, there are more and more opportunities to perform automatic shooting using an automatic camera platform, gimbal, etc. In automatic shooting, as long as shooting can be performed without cut-offs or overlaps, the captured image can be processed or edited after shooting to a state desired by an editor by utilizing computational photography, etc. In response to such a demand, according to the second embodiment, an image without cut-offs or overlaps can be obtained by automatic shooting more efficiently than in the first embodiment, and therefore a shooting method that meets the demand to utilize computational photography, etc. after shooting can be provided.

[0138] The disclosure of the above embodiment includes the following configurations and methods. (Configuration 1) An imaging device comprising: an imaging means; a registration means for registering a plurality of subjects to be photographed; a detection means for detecting subjects registered in the registration means from a photographed image acquired by the imaging means; a setting means for setting an evaluation criterion for evaluating a state of the subject detected by the detection means; an evaluation means for determining an evaluation value representing the quality of the photographed image for each of the plurality of subjects by comparing the face and body parts other than the face of the detected subject with the evaluation criterion; and a control means for displaying the sum of the evaluation values ​​for each of the plurality of subjects on a display device. (Configuration 2) An imaging device comprising: an imaging means; a registration means for registering a plurality of subjects; a detection means for detecting subjects registered in the registration means from an image acquired by the imaging means; a setting means for setting an evaluation criterion for evaluating a state of the subject detected by the detection means; an evaluation means for determining an evaluation value representing the quality of the captured image for each of the plurality of subjects by comparing the face and body parts other than the face of the subject detected by the detection means from the captured image as the image with the evaluation criterion; and a control means for controlling the imaging means using at least one of the evaluation value and the evaluation criterion. (Configuration 3) The imaging device according to configuration 1 or 2, wherein the evaluation criteria define face and other body parts that must appear in the captured image. (Configuration 4) The imaging device according to Configuration 3, wherein the evaluation criteria further define at least one of a focus state of the subject in the captured image and a capture size of the subject in the captured image. (Configuration 5) The imaging device described in Configuration 4, wherein the evaluation criteria are defined as the focus state, the face and other body parts, and the shooting size, and weights are defined for each of the focus state, the face and other body parts, and the shooting size, the evaluation means holds a table in which ranked scores are defined for each of the focus state, the body parts, and the shooting size, and the evaluation value is calculated as the sum of values ​​obtained by multiplying the scores calculated from the table for each of the focus state, the body parts, and the shooting size by the corresponding weights. (Configuration 6) An imaging device as described in configuration 4 or 5, characterized in that when a predetermined number or more of subjects are detected from the captured image and the size of the subjects or the size of the subjects' faces is equal to or smaller than a predetermined size, the evaluation means does not calculate the evaluation value even if the captured size of the subject in the captured image is defined as the evaluation criterion. (Configuration 7) The imaging device according to configuration 2, wherein the control means controls the imaging means so as to capture an image of the subject having the smallest sum of the evaluation values ​​among the plurality of subjects. (Configuration 8) The imaging device according to configuration 2, wherein the control means performs continuous photographing by the imaging means while switching between subjects to be photographed in order from the subject having the smallest sum of the evaluation values ​​among the plurality of subjects. (Configuration 9) The imaging device described in configuration 2, characterized in that the evaluation criteria define face and non-facial body parts that are required to be captured in the image, the detection means detects a body part of a subject in a live view image as the image when the subject is any of the multiple subjects, and the control means controls the imaging means to acquire a captured image of a subject whose body part detected by the detection means among the multiple subjects satisfies the body part conditions defined in the evaluation criteria. (Configuration 10) The imaging device described in Configuration 2, characterized in that the evaluation criteria define face and non-facial body parts that are required to be included in the image, the detection means detects body parts defined in the evaluation criteria for each of the plurality of subjects from a live view image as the image, and the control means controls the imaging means to prioritize focusing on a subject having a larger value obtained by dividing the number of body parts detected for each of the two or more subjects by the total number of body parts detected by the detection means when two or more of the plurality of subjects are included in the live view image. (Configuration 11) The imaging device described in Configuration 2, wherein the control means controls the imaging means so that, when a subject not registered in the registration means is detected adjacent to any of the plurality of subjects in a live view image as the image, the adjacent subject is simultaneously brought into focus. (Configuration 12) The imaging device described in Configuration 2, characterized in that when the detection means detects multiple subjects that are regularly aligned and move across the entire range of the shooting angle of view in a live view image as the image, the control means controls the imaging means to focus on the subjects registered in the registration means in order of the smallest sum of the evaluation values ​​or in order of the subjects determined to be closest. (Configuration 13) A program for causing a computer to function as the registration means, the detection means, the setting means, the evaluation means and the control means of the imaging device described in any one of configurations 1 to 12. (Method 1) A control method for an imaging device, comprising the steps of: registering a plurality of subjects in a registration means; setting evaluation criteria for evaluating the state of the subjects in a captured image acquired by an imaging means provided in the imaging device; detecting the subjects registered in the registration means from the captured image; comparing the face and body parts other than the face of the detected subjects with the evaluation criteria to obtain an evaluation value representing the quality of the captured image for each of the registered subjects; and displaying a sum of the evaluation values ​​for each of the registered subjects on a display means. (Method 2) A method for controlling an imaging device, comprising the steps of: registering a plurality of subjects in a registration means; setting evaluation criteria for evaluating the state of the subjects in an image acquired by an imaging means provided in the imaging device; detecting the subjects registered in the registration means from the image; comparing the face and other body parts of the subject detected from the captured image as the image with the evaluation criteria to obtain an evaluation value representing the quality of the captured image for each of the registered subjects; and controlling the imaging means using at least one of the evaluation value and the evaluation criteria.

[0139] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. That is, the above embodiments have given several representative examples of implementing the present invention according to the shooting scene, shooting purpose, shooting use, subject detection results, etc., but the present invention is not limited to the above examples. In addition, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0140] For example, in the evaluation item definition table in Fig. 11(a), items 1 to 3 are defined, but at least one item may be defined. This is because, when only one item is set, for example, if the total yield does not increase even if shooting is continued, the photographer can recognize that the intended captured image is not being obtained.

[0141] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0142] 100 Imaging device 107 Image sensor 121 Control Unit 125 Image Processing Unit 131 Display section 140 Object detection section 141 Storage section 142 Attitude acquisition part 1400 Footage Table

Claims

1. An imaging means; A registration means for registering a plurality of subjects to be photographed; a detection means for detecting a subject registered in the registration means from a photographed image acquired by the imaging means; a setting means for setting an evaluation standard for evaluating a state of the subject detected by the detection means; an evaluation means for calculating an evaluation value representing a quality of the photographed image for each of the plurality of subjects by comparing the face and other body parts of the detected subjects with the evaluation criteria; and a control means for displaying, on a display device, the sum of the evaluation values ​​for each of the plurality of subjects.

2. An imaging means; A registration means for registering a plurality of subjects; a detection means for detecting a subject registered in the registration means from an image acquired by the imaging means; a setting means for setting an evaluation standard for evaluating a state of the subject detected by the detection means; an evaluation means for calculating an evaluation value representing a quality of the photographed image for each of the plurality of subjects by comparing the face and body parts other than the face of the subject detected by the detection means from the photographed image as the image with the evaluation criteria; and a control unit that controls the imaging unit using at least one of the evaluation value and the evaluation criterion.

3. 3. The imaging apparatus according to claim 1, wherein the evaluation criteria define face and other body parts than the face that are required to appear in the captured image.

4. 4. The image pickup apparatus according to claim 3, wherein the evaluation criteria further define at least one of a focus state of the subject in the captured image and a photographed size of the subject in the captured image.

5. As the evaluation criteria, the focus state, the face and other body parts, and the photographing size are defined, and weights are defined for each of the focus state, the face and other body parts, and the photographing size; the evaluation means holds a table in which a ranked score is defined for each of the focusing state, the body part, and the imaging size; 5. The imaging device according to claim 4, wherein the evaluation value is calculated as a sum of values ​​obtained by multiplying the score calculated from the table for each captured image for each of the focus state, the body part, and the shooting size by the corresponding weight.

6. The imaging device according to claim 5, characterized in that when a predetermined number or more of subjects are detected from the captured image and the size of the subjects or the size of the faces of the subjects is equal to or smaller than a predetermined size, the evaluation means does not calculate the evaluation value even if the captured size of the subject in the captured image is defined as the evaluation criterion.

7. 3. The image pickup apparatus according to claim 2, wherein the control means controls the image pickup means so as to photograph a subject that has the smallest sum of the evaluation values ​​among the plurality of subjects.

8. 3. The imaging apparatus according to claim 2, wherein the control means performs continuous photographing by the imaging means while switching between the subjects to be photographed in order from the subject having the smallest sum of the evaluation values ​​among the plurality of subjects.

9. The evaluation criteria defines face and non-facial body parts that are required to be present in the image; the detection means detects a body part of a subject in a live view image as the image when the subject is any one of the plurality of subjects; The imaging device according to claim 2, characterized in that the control means controls the imaging means to acquire an image of a subject whose body part detected by the detection means among the plurality of subjects satisfies the body part condition defined in the evaluation criterion.

10. The evaluation criteria defines face and non-facial body parts that are required to be present in the image; the detection means detects a body part defined in the evaluation criterion from a live view image serving as the image for each of the plurality of subjects; 3. The imaging device according to claim 2, wherein the control means controls the imaging means so that, when two or more of the multiple subjects are captured in the live view image, the control means controls the imaging means to preferentially focus on a subject having a larger value obtained by dividing the number of body parts detected for each of the two or more subjects by the total number of body parts detected by the detection means.

11. The imaging device according to claim 2, characterized in that, when a subject that is not registered in the registration means is detected adjacent to any of the plurality of subjects in a live view image as the image, the control means controls the imaging means so that the adjacent subjects are simultaneously focused on.

12. 3. The imaging device according to claim 2, wherein, when the detection means detects a plurality of subjects that are regularly aligned and move across the entire range of the shooting angle of view in a live view image as the image, the control means controls the imaging means to focus on the subjects registered in the registration means in order of the smallest sum of the evaluation values ​​or in order of the subjects determined to be closest.

13. A control method for an imaging device, comprising: A step of registering a plurality of subjects in a registration means; setting an evaluation criterion for evaluating a state of a subject in a photographed image acquired by an imaging unit included in the imaging device; detecting a subject registered in the registration means from the captured image; A step of calculating an evaluation value representing a quality of the captured image for each of the registered subjects by comparing the face and other body parts of the detected subjects with the evaluation criteria; and displaying the sum of the evaluation values ​​for each of the registered subjects on a display means.

14. A control method for an imaging device, comprising: A step of registering a plurality of subjects in a registration means; setting an evaluation criterion for evaluating a state of a subject in an image acquired by an imaging means included in the imaging device; detecting a subject registered in the registration means from the image; A step of calculating an evaluation value representing a quality of the photographed image for each of the registered subjects by comparing the face and body parts other than the face of the subject detected from the photographed image as the image with the evaluation criteria; and controlling the imaging means using at least one of the evaluation value and the evaluation criterion.

15. 3. A program for causing a computer to function as the registration means, the detection means, the setting means, the evaluation means and the control means of the imaging device according to claim 1.

Citation Information

Patent Citations

  • Photographing device, photography control program, and image reproducing device and image reproducing program

    JP2009290819A

  • Imaging apparatus

    JP2010273281A