Imaging support device, imaging device, imaging support method, and program

Abstract

To provide an imaging support device that can support imaging by an imaging device according to a frequency with which the features of a subject are classified into categories.SOLUTION: An imaging support device includes a processor and a memory connected to or built into the processor, and the processor acquires frequency information indicating a frequency of features classified into categories on the basis of the features of a subject contained in an image obtained by imaging using the imaging device, and performs support processing to support imaging by the imaging device on the basis of the frequency information.SELECTED DRAWING: Figure 23

Description

Technical Field

[0001] The technology of the present disclosure relates to an imaging support device, an imaging device, an imaging support method, and a program.

Background Art

[0002] Japanese Patent Application Laid-Open No. 2007-006033 discloses an object determination device that selects a face to be processed from a plurality of faces included in an image. The object determination device described in Japanese Patent Application Laid-Open No. 2007-006033 includes a face detection unit that detects a face from an image, a face information recording unit that associates and records a face detected in the past by the face detection unit with a detection history related to the detection, and a face selection unit that selects a face to be processed from the faces included in the image based on the detection history.

[0003] Japanese Patent Publication No. 2009-252069 discloses an image processing apparatus characterized by comprising a face recognition dictionary, an image acquisition means, a face region detection means, a feature extraction means, a discrimination means, a face recognition dictionary modification means, and a classification means. The face recognition dictionary registers face features for each person to determine whether or not they are the same person. The image acquisition means acquires an image containing a person. The face region detection means detects a face region from the image acquired by the image acquisition means. The feature extraction means extracts face features from the face region based on the face region detected by the face region detection means. The discrimination means determines whether or not the face features of the same person are registered in the face recognition dictionary based on the face features extracted by the feature extraction means and the face features registered in the face recognition dictionary. If the discrimination means determines that the face features of the same person are registered in the face recognition dictionary, the face recognition dictionary modification means modifies the registered face features based on the extracted face features. If the discrimination means determines that the face features of the same person are not registered in the face recognition dictionary, the extracted face features are registered as the face features of a new person. The classification means classifies the face of the person in the image acquired by the image acquisition means as a known face if the discrimination means determines that the facial features of the same person are registered in the face recognition dictionary, and classifies the face of the person in the image acquired by the image acquisition means as an unknown face if the discrimination means determines that the facial features of the same person are not registered in the face recognition dictionary.

[0004] Japanese Patent Publication No. 2012-099943 discloses an image processing apparatus comprising: a storage means for storing face image data; a face detection means for detecting faces from a video signal; a face recognition means for determining whether or not a face detected by the face detection means is included in the face image data stored in the storage means; and an image processing means. When the image processing means detects faces from the video signal that are determined to be included in the face image data stored in the storage means and faces that are determined not to be included, it performs image processing on the regions of the faces determined to be included in the face image data and the regions of the faces determined not to be included in the face image data, thereby increasing the image quality compared to other regions.

[0005] Japanese Patent Publication No. 2009-003012 discloses an imaging device characterized by comprising: an imaging lens capable of changing the lens focal position by driving at least a portion of a plurality of lenses arranged along the optical axis along the optical axis; an image acquisition means; a subject detection means; a focus evaluation value calculation means; a selection means; and a recording means. The image acquisition means continuously performs shooting while changing the lens focal position of the imaging lens and acquires a plurality of image data. The subject detection means detects the main subject area in accordance with the movement state of the subject among the plurality of image data obtained by the image acquisition means. The focus evaluation value calculation means calculates the focus evaluation value of the main subject area obtained by the subject detection means for each of the plurality of image data. The selection means selects at least one of the plurality of image data based on the focus evaluation value obtained by the focus evaluation value calculation means. The recording means causes the image data selected by the selection means to be recorded on a recording medium. [Overview of the Initiative]

[0006] One embodiment of the technology of this disclosure provides an imaging support device, an imaging device, an imaging support method, and a program that can support imaging by an imaging device according to the frequency with which the characteristics of a subject are classified into categories. [Means for solving the problem]

[0007] A first aspect of the technology of this disclosure is an imaging support device comprising a processor and memory connected to or built into the processor, wherein the processor acquires frequency information indicating the frequency of features classified into categories based on features of a subject identified from an image captured by an imaging device, and performs support processing to support imaging by the imaging device based on the frequency information.

[0008] A second aspect of the technology of this disclosure is an imaging support device according to the first aspect, wherein the categories are classified into a plurality of categories, including at least one target category, the target category is a category determined based on frequency information, and the support process includes a process that supports imaging of a target category subject having characteristics belonging to the target category.

[0009] A third aspect of the technology of this disclosure is an imaging support device according to the second aspect, wherein the support processing includes a display processing that displays a recommendation for imaging of a target category subject.

[0010] A fourth aspect of the technology of this disclosure is an imaging support device according to the third aspect, wherein the display processing is a process of displaying a display image obtained by imaging by an imaging device on a display, and displaying a target category subject image that indicates a target category subject within the display image in a manner that can be distinguished from other image areas.

[0011] A fifth aspect of the technology of this disclosure is an imaging support device relating to any one of the second to fourth aspects, wherein the processor detects a target category subject based on the imaging results of an imaging device, and, on the condition that a target category subject has been detected, acquires an image including an image corresponding to the target category subject.

[0012] A sixth aspect of the technology of this disclosure is an imaging support device relating to any one of the second to fifth aspects, wherein the processor displays an object indicating a designated imaging range determined according to an externally given instruction and an object indicating a target category subject in a different display manner.

[0013] A seventh aspect of the technology of this disclosure is an imaging support device relating to any one of the second to sixth aspects, wherein the processor performs a default processing when the degree of difference between a first imaging condition given from an external source and a second imaging condition given to a target category subject is greater than or equal to a default difference.

[0014] An eighth aspect of the technology of this disclosure is an imaging support device relating to any one of the second to seventh aspects, wherein the target category is a low-frequency category that is relatively less frequent among multiple categories.

[0015] The ninth aspect of the technology of this disclosure is an imaging support device relating to any one of the second to eighth aspects, wherein when a subject of a target category is imaged by an imaging device, the target category is a category determined according to the state of the subject of the target category, and is a category into which the characteristics of the subject of the target category are classified.

[0016] A tenth aspect of the technology of this disclosure is an imaging support device relating to any one of the second to ninth aspects, wherein, when multiple objects are imaged by an imaging device, the target category is an object target category that can identify each of the multiple objects themselves.

[0017] An eleventh aspect of the technology of this disclosure is an imaging support device relating to any one of the first to tenth aspects, wherein the category is made up of at least one unit.

[0018] A twelfth aspect of the technology of this disclosure is an imaging support device according to the eleventh aspect, wherein one of the units is a period of time.

[0019] A thirteenth aspect of the technology of this disclosure is an imaging support device according to the eleventh or twelfth aspect, wherein one of the units is position.

[0020] A fourteenth aspect of the technology of this disclosure is an imaging support device relating to any one of the first to thirteenth aspects, wherein a processor causes a classifier to classify features, and the classifier classifies features when the scene to be captured by the imaging device matches a specific scene.

[0021] A 15th aspect of the technology of this disclosure is an imaging support device according to the 14th aspect, wherein the specific scene is a scene that was captured in the past.

[0022] A sixteenth aspect of the technology of this disclosure is an imaging support device relating to any one of the first to fifteenth aspects, wherein the support process includes a process for displaying frequency information.

[0023] A seventeenth aspect of the technology of the present disclosure is an imaging support device according to the sixteenth aspect, wherein the support process includes a process of supporting imaging related to a category corresponding to the specified frequency information when the frequency information is specified by a reception device while the frequency information is displayed.

[0024] An eighteenth aspect of the technology of the present disclosure is an imaging support device including a processor and a memory connected to or incorporated in the processor, wherein the processor acquires frequency information indicating the frequency of imaging images classified into categories based on the features of subjects included in the imaging images obtained by imaging with an imaging device, and performs a support process for supporting imaging by the imaging device based on the frequency information.

[0025] A nineteenth aspect of the technology of the present disclosure is an imaging device including the imaging support device according to any one of the first to eighteenth aspects and an image sensor, wherein the processor supports imaging using the image sensor by performing the support process.

[0026] A twentieth aspect of the technology of the present disclosure is an imaging support method including acquiring frequency information indicating the frequency of features classified into categories based on the features of subjects specified from the imaging images obtained by imaging with an imaging device, and performing a support process for supporting imaging by the imaging device based on the frequency information.

[0027] A twenty - first aspect of the technology of the present disclosure is an imaging support method including acquiring frequency information indicating the frequency of imaging images classified into categories based on the features of subjects specified from the imaging images obtained by imaging with an imaging device, and performing a support process for supporting imaging by the imaging device based on the frequency information.

[0028] A 22nd aspect according to the technology of the present disclosure is a program for causing a computer to execute a process including acquiring frequency information indicating the frequency of features classified into categories based on features of a subject specified from a captured image obtained by being captured by an imaging device, and performing support processing for supporting imaging by the imaging device based on the frequency information.

[0029] A 23rd aspect according to the technology of the present disclosure is a program for causing a computer to execute a process including acquiring frequency information indicating the frequency of captured images classified into categories based on features of a subject specified from a captured image obtained by being captured by an imaging device, and performing support processing for supporting imaging by the imaging device based on the frequency information.

Brief Description of the Drawings

[0030] [Figure 1] It is a perspective view showing an example of the appearance of the imaging device. [Figure 2] It is a rear view showing an example of the appearance of the rear side of the imaging device shown in FIG. 1. [Figure 3] It is a schematic diagram showing an example of the arrangement of each pixel included in the photoelectric conversion element of the imaging device. [Figure 4] It is a conceptual diagram showing an example of the incident characteristics of subject light with respect to the first phase difference pixel and the second phase difference pixel included in the photoelectric conversion element shown in FIG. 3. [Figure 5] It is a schematic configuration diagram showing an example of the configuration of the non-phase difference pixel included in the photoelectric conversion element shown in FIG. 3. [Figure 6] It is a schematic configuration diagram showing an example of the hardware configuration of the imaging device. [Figure 7] It is a block diagram showing an example of the configuration of the controller included in the imaging device. [Figure 8] It is a block diagram showing an example of the main part functions of the CPU included in the imaging device. [Figure 9] It is a conceptual diagram showing an example of the processing content when the CPU operates as an acquisition unit and a control unit. [Figure 10]This is a conceptual diagram illustrating an example of processing when the CPU acts as the control unit. [Figure 11] This is a conceptual diagram illustrating an example of the processing content when the CPU operates as both a subject recognition unit and a control unit. [Figure 12] This is a conceptual diagram illustrating an example of the processing performed when the CPU operates as a feature extraction unit. [Figure 13] This is a conceptual diagram illustrating an example of the processing performed when the CPU operates as a feature extraction unit and a classification unit. [Figure 14] This is a conceptual diagram illustrating an example of how subject characteristics are classified into multiple categories by a classification unit. [Figure 15] This is a conceptual diagram illustrating an example of the process by which the subject recognition unit recognizes a subject based on live view image data. [Figure 16] This is a conceptual diagram illustrating an example of the processing involved when an imaging support screen is displayed within a live view image. [Figure 17] This is a screen diagram showing an example of a configuration in which a bubble chart or similar is displayed within the imaging support screen. [Figure 18] This is a screen diagram showing an example of a configuration in which a histogram and other information are displayed within the imaging support screen. [Figure 19] This is a screen diagram showing an example of how the live view image is displayed when image acquisition support processing is performed. [Figure 20A] This flowchart shows an example of the image acquisition support processing flow. [Figure 20B] This is a continuation of the flowchart shown in Figure 20A. [Figure 21] This is a screen diagram showing an example of the display content in a live view image where the object indicating the specified imaging area and the subject image of the target category are displayed in different ways. [Figure 22] This flowchart shows the first modified example of the image acquisition support processing flow. [Figure 23] This flowchart shows a second modified example of the image acquisition support processing flow. [Figure 24] This flowchart shows a third modified example of the image acquisition support processing flow. [Figure 25] This flowchart shows a fourth modified example of the image acquisition support processing flow. [Figure 26] This flowchart shows the fifth modified example of the image acquisition support processing flow. [Figure 27] This is a conceptual diagram showing an example of calculating the degree of difference between the first imaging condition and the second imaging condition. [Figure 28] This is a conceptual diagram illustrating an example of the processing steps taken to increase the depth of field. [Figure 29] This is a conceptual diagram showing examples of how to calculate the focus position for subjects within a specified imaging range and for subjects within a target category. [Figure 30] This block diagram shows an example of the processing steps involved when a focus bracketing method is used for full exposure imaging. [Figure 31] This flowchart shows the sixth modified example of the image acquisition support processing flow. [Figure 32] This is a block diagram showing an example of how subject-specific category groups are structured. [Figure 33] This is a screen diagram showing an example of a configuration in which a period category bubble chart and the like are displayed within the imaging support screen. [Figure 34] This is a screen diagram showing an example of a configuration in which a period category histogram and other elements are displayed within the imaging support screen. [Figure 35] This is a time chart showing an example of a mode in which imaging support processing is performed at predetermined time intervals, and an example of a mode in which imaging support processing is performed for each position checkpoint. [Figure 36] This is a flowchart showing the seventh modified example of the image acquisition support processing flow. [Figure 37] This is a conceptual diagram illustrating an example of how the exposed image is classified into multiple categories by the classification unit. [Figure 38] This is a conceptual diagram showing an example of a 4-quadrant face category bubble chart. [Figure 39] This block diagram shows an example of how training data, including the exposure image data obtained from the processed exposure imaging, is used in machine learning of a trained model. [Figure 40] This block diagram shows an example of a method in which an imaging device causes an external device to perform imaging support processing. [Figure 41] This block diagram shows an example of how an imaging support processing program is installed from a storage medium containing the program to a controller in an imaging device. [Modes for carrying out the invention]

[0031] Hereinafter, an example of an embodiment of the imaging support device, imaging device, imaging support method, and program relating to the technology of this disclosure will be described with reference to the attached drawings.

[0032] First, let's explain the terminology used in the following explanation.

[0033] CPU stands for "Central Processing Unit". RAM stands for "Random Access Memory". IC stands for "Integrated Circuit". ASIC stands for "Application Specific Integrated Circuit". PLD stands for "Programmable Logic Device". FPGA stands for "Field-Programmable Gate Array". SoC stands for "System-on-a-chip". SSD stands for "Solid State Drive". USB stands for "Universal Serial Bus". HDD stands for "Hard Disk Drive". EEPROM stands for "Electrically Erasable and Programmable Read Only Memory". EL stands for "Electro-Luminescence". I / F stands for "Interface". UI stands for "User Interface". TOF stands for "Time of Flight". fps stands for "frames per second". MF stands for "Manual Focus". AF stands for "Auto Focus". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device". RTC stands for "Real Time Clock". GPS stands for "Global Positioning System". LAN stands for "Local Area Network". WAN stands for "Wide Area Network". GNSS stands for "Global Navigation Satellite System".For the sake of explanation, a CPU is used as an example of a "processor" in the technology disclosed below. However, a "processor" in the technology disclosed may be a combination of multiple processing units, such as a CPU and a GPU. When a combination of a CPU and a GPU is applied as an example of a "processor" in the technology disclosed, the GPU operates under the control of the CPU and is responsible for performing image processing.

[0034] In this specification, “perpendicular” means not only perfect perpendicularity but also an error that is generally acceptable in the art to which the disclosed technology belongs, provided that the error is not contrary to the spirit of the disclosed technology. In this specification, “matching” means not only perfect matching but also an error that is generally acceptable in the art to which the disclosed technology belongs, provided that the error is not contrary to the spirit of the disclosed technology.

[0035] As an example, as shown in Figure 1, the imaging device 10 is a digital camera with interchangeable lenses and an omitted reflex mirror. The imaging device 10 comprises an imaging device body 12 and an interchangeable lens 14 that is interchangeably attached to the imaging device body 12. Here, an example of the imaging device 10 is a digital camera with interchangeable lenses and an omitted reflex mirror, but the technology of this disclosure is not limited to this, and it may also be a digital camera with a fixed lens, a digital camera that does not omit a reflex mirror, or a digital camera built into various electronic devices such as smart devices, wearable terminals, cell observation devices, ophthalmic observation devices, or surgical microscopes.

[0036] The imaging device body 12 is equipped with an image sensor 16. The image sensor 16 is a CMOS image sensor. The image sensor 16 captures an imaging area that includes the group of subjects. When the interchangeable lens 14 is attached to the imaging device body 12, the subject light representing the subjects passes through the interchangeable lens 14 and is formed on the image sensor 16, and image data representing the image of the subjects is generated by the image sensor 16.

[0037] In this embodiment, a CMOS image sensor is used as an example for the image sensor 16. However, the technology of this disclosure is not limited to this, and the technology of this disclosure can be applied even if the image sensor 16 is another type of image sensor, such as a CCD image sensor.

[0038] A release button 18 and a dial 20 are provided on the top surface of the imaging device body 12. The dial 20 is operated when setting the operating mode of the imaging system and the operating mode of the playback system, and when the dial 20 is operated, the imaging device 10 selectively sets the operating mode to either imaging mode or playback mode.

[0039] The release button 18 functions as both an imaging preparation instruction unit and an imaging instruction unit, and can detect two stages of pressing: an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from, for example, the standby position to an intermediate position (half-press position), and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to the final pressed position (full-press position). Hereinafter, the state in which the button is pressed from the standby position to the half-press position will be referred to as the "half-press state," and the state in which the button is pressed from the standby position to the full-press position will be referred to as the "full-press state." Depending on the configuration of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 18, and the imaging instruction state may be a state in which the user's finger has moved away from contact with the release button 18.

[0040] As an example, as shown in Figure 2, a touch panel display 22 and instruction keys 24 are provided on the back of the imaging device body 12.

[0041] The touch panel display 22 comprises a display 26 and a touch panel 28 (see also Figure 3). An example of the display 26 is an organic EL display. The display 26 may be other types of displays, such as a liquid crystal display or an inorganic EL display, instead of an organic EL display.

[0042] The display 26 displays images and / or text information. When the imaging device 10 is in imaging mode, the display 26 is used to display the live view image obtained by imaging for the live view image, that is, by continuous imaging. Imaging for the live view image (hereinafter also referred to as "imaging for the live view image") is performed according to a frame rate of, for example, 60fps. 60fps is merely an example; the frame rate may be less than 60fps or more than 60fps.

[0043] Here, "live view image" refers to a moving image for display based on image data obtained by capturing data with the image sensor 16. Live view images are also commonly referred to as through images. The live view image is an example of a "display image" related to the technology disclosed herein.

[0044] The display 26 is also used to display still images obtained when the imaging device 10 is instructed to take still images via the release button 18. Furthermore, the display 26 is also used to display playback images when the imaging device 10 is in playback mode, as well as to display menu screens and the like.

[0045] The touch panel 28 is a transmissive touch panel and is superimposed on the surface of the display area of ​​the display 26. The touch panel 28 receives user input by detecting contact from an object such as a finger or stylus pen. For the sake of explanation, the "fully pressed state" described above also includes the state in which the user has turned on the soft key for starting image capture via the touch panel 28.

[0046] Furthermore, in this embodiment, an out-cell type touch panel display in which the touch panel 28 is superimposed on the surface of the display area of ​​the display 26 is given as an example of the touch panel display 22, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be used as the touch panel display 22.

[0047] The instruction key 24 accepts various instructions. Here, "various instructions" refers to, for example, instructions to display a menu screen where various menus can be selected, instructions to select one or more menus, instructions to confirm the selection, instructions to clear the selection, instructions to zoom in, zoom out, and various other instructions such as frame-by-frame playback. These instructions may also be given via the touch panel 28.

[0048] As an example, as shown in Figure 3, the image sensor 16 is equipped with a photoelectric conversion element 30. The photoelectric conversion element 30 has a light-receiving surface 30A. The photoelectric conversion element 30 is positioned within the imaging device body 12 (see Figure 1) such that the center of the light-receiving surface 30A coincides with the optical axis OA (see Figure 1). The photoelectric conversion element 30 has a plurality of photosensitive pixels arranged in a matrix, and the light-receiving surface 30A is formed by the plurality of photosensitive pixels. The photosensitive pixels are pixels that have a photodiode PD, which photoelectrically convert the received light and output an electrical signal corresponding to the amount of light received. There are two types of photosensitive pixels included in the photoelectric conversion element 30: phase-difference pixels P, which are so-called image plane phase-difference pixels, and non-phase-difference pixels N, which are pixels different from the phase-difference pixels P.

[0049] A color filter is placed inside the photodiode PD. The color filter includes a G filter corresponding to the G (green) wavelength range, which contributes most to obtaining the luminance signal, an R filter corresponding to the R (red) wavelength range, and a B filter corresponding to the B (blue) wavelength range.

[0050] Generally, non-phase-difference pixels N are also called normal pixels. The photoelectric conversion element 30 has three types of photosensitive pixels as non-phase-difference pixels N: R pixels, G pixels, and B pixels. The R pixels, G pixels, B pixels, and phase-difference pixels P are arranged regularly with a predetermined periodicity in both the row direction (for example, the horizontal direction when the bottom surface of the imaging device body 12 is in contact with a horizontal surface) and the column direction (for example, the vertical direction which is perpendicular to the horizontal direction). The R pixels are pixels corresponding to photodiode PDs on which R filters are placed, the G pixels and phase-difference pixels P are pixels corresponding to photodiode PDs on which G filters are placed, and the B pixels are pixels corresponding to photodiode PDs on which B filters are placed.

[0051] Multiple phase-difference pixel lines 32A and multiple non-phase-difference pixel lines 32B are arranged on the light-receiving surface 30A. Phase-difference pixel lines 32A are horizontal lines containing phase-difference pixels P. Specifically, phase-difference pixel lines 32A are horizontal lines in which phase-difference pixels P and non-phase-difference pixels N are mixed. Non-phase-difference pixel lines 32B are horizontal lines containing only multiple non-phase-difference pixels N.

[0052] On the light-receiving surface 30A, phase-difference pixel lines 32A and a predetermined number of non-phase-difference pixel lines 32B are arranged alternately along the column direction. The "determined number of lines" here refers to, for example, 2 lines. Although 2 lines are used as an example of the predetermined number of lines here, the technology of this disclosure is not limited to this, and the predetermined number of lines may be 3 or more lines, or it may be tens of lines, tens of lines, or hundreds of lines, etc.

[0053] The phase difference pixel line 32A is arranged in a column direction with two rows skipped from the first row to the last row. Some of the pixels in the phase difference pixel line 32A are phase difference pixels P. Specifically, the phase difference pixel line 32A is a horizontal line in which phase difference pixels P and non-phase difference pixels N are arranged periodically. Phase difference pixels P are broadly divided into first phase difference pixels L and second phase difference pixels R. In the phase difference pixel line 32A, first phase difference pixels L and second phase difference pixels R are arranged alternately as G pixels at intervals of several pixels in the line direction.

[0054] The first phase difference pixels L and the second phase difference pixels R are arranged to appear alternately in the column direction. In the example shown in Figure 3, in the fourth column, the first phase difference pixels L, second phase difference pixels R, first phase difference pixels L, and second phase difference pixels R are arranged in that order from the first row along the column direction. That is, the first phase difference pixels L and second phase difference pixels R are arranged alternately from the first row along the column direction. Also in the example shown in Figure 3, in the tenth column, the second phase difference pixels R, first phase difference pixels L, second phase difference pixels R, and first phase difference pixels L are arranged in that order from the first row along the column direction. That is, the second phase difference pixels R and first phase difference pixels L are arranged alternately from the first row along the column direction.

[0055] The photoelectric conversion element 30 is divided into two regions. Specifically, the photoelectric conversion element 30 has a non-phase difference pixel region 30N and a phase difference pixel region 30P. The phase difference pixel region 30P is a group of phase difference pixels consisting of multiple phase difference pixels P, which receive subject light and generate phase difference image data as an electrical signal corresponding to the amount of light received. The phase difference image data is used, for example, for distance measurement. The non-phase difference pixel region 30N is a group of non-phase difference pixels consisting of multiple non-phase difference pixels N, which receive subject light and generate non-phase difference image data as an electrical signal corresponding to the amount of light received. The non-phase difference image data is displayed, for example, as a visible light image on the display 26 (see Figure 2).

[0056] As an example, as shown in Figure 4, the first phase difference pixel L comprises a light-shielding member 34A, a microlens 36, and a photodiode PD. In the first phase difference pixel L, the light-shielding member 34A is positioned between the microlens 36 and the light-receiving surface of the photodiode PD. The left half in the row direction of the light-receiving surface of the photodiode PD (the left side when viewing the subject from the light-receiving surface (in other words, the right side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 34A.

[0057] The second phase-difference pixel R comprises a light-shielding member 34B, a microlens 36, and a photodiode PD. In the second phase-difference pixel R, the light-shielding member 34B is positioned between the microlens 36 and the light-receiving surface of the photodiode PD. The right half of the photodiode PD's light-receiving surface in the row direction (the right side when viewing the subject from the light-receiving surface (in other words, the left side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 34B. For the sake of explanation, in the following, when it is not necessary to distinguish between the light-shielding members 34A and 34B, they will be referred to simply as "light-shielding member" without any reference numerals.

[0058] The interchangeable lens 14 is equipped with an imaging lens 40. The light beam passing through the exit pupil of the imaging lens 40 is broadly divided into left-region passing light 38L and right-region passing light 38R. Left-region passing light 38L refers to the left half of the light beam passing through the exit pupil of the imaging lens 40 when viewed from the phase difference pixel P side toward the subject side, and right-region passing light 38R refers to the right half of the light beam passing through the exit pupil of the imaging lens 40 when viewed from the phase difference pixel P side toward the subject side. The light beam passing through the exit pupil of the imaging lens 40 is divided into left and right by a microlens 36, a light-shielding member 34A, and a light-shielding member 34B, which function as pupil division parts, and the first phase difference pixel L receives the left-region passing light 38L as subject light, and the second phase difference pixel R receives the right-region passing light 38R as subject light. As a result, the photoelectric conversion element 30 generates a first phase difference image data corresponding to the subject image corresponding to the light passing through the left region 38L, and a second phase difference image data corresponding to the subject image corresponding to the light passing through the right region 38R.

[0059] In the imaging device 10, for example, the distance to the subject, i.e., the subject distance, is measured based on the amount of shift α (hereinafter also simply referred to as "shift amount α") between the first phase difference image data for one line and the second phase difference image data for one line in the same phase difference pixel line 32A. Since the method for deriving the subject distance from the shift amount α is a known technique, a detailed explanation is omitted here.

[0060] As an example, as shown in Figure 5, the non-phase difference pixel N differs from the phase difference pixel P in that it does not have a light-shielding member. The photodiode PD of the non-phase difference pixel N receives light passing through the left region 38L and light passing through the right region 38R as the subject light.

[0061] As an example, as shown in Figure 6, the imaging lens 40 comprises an objective lens 40A, a focusing lens 40B, and an aperture 40C. The objective lens 40A, focusing lens 40B, and aperture 40C are arranged in that order along the optical axis OA from the subject side (object side) to the imaging device body 12 side (image side).

[0062] The interchangeable lens 14 also includes a slide mechanism 42, a motor 44, and a motor 46. A focus lens 40B is mounted on the slide mechanism 42 so as to slide along the optical axis OA. A motor 44 is connected to the slide mechanism 42, and the slide mechanism 42 moves the focus lens 40B along the optical axis OA by receiving power from the motor 44. The aperture 40C is a variable aperture. A motor 46 is connected to the aperture 40C, and the aperture 40C adjusts the exposure by receiving power from the motor 46. The components and / or operating method of the interchangeable lens 14 can be changed as needed.

[0063] Motors 44 and 46 are connected to the imaging device body 12 via a mount (not shown), and their drive is controlled according to commands from the imaging device body 12. In this embodiment, stepping motors are used as an example of motors 44 and 46. Therefore, motors 44 and 46 operate in synchronization with pulse signals according to commands from the imaging device body 12. In the example shown in Figure 6, motors 44 and 46 are provided on the interchangeable lens 14, but this is not limited to this, and either motor 44 or 46 may be provided on the imaging device body 12, or both motors 44 and 46 may be provided on the imaging device body 12.

[0064] In the imaging device 10, when in imaging mode, MF mode and AF mode are selectively set according to instructions given to the imaging device body 12. MF mode is an operation mode in which the focus is adjusted manually. In MF mode, for example, when the user operates the focus ring of the interchangeable lens 14, the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount the focus ring is operated, thereby adjusting the focus.

[0065] In AF mode, the imaging device body 12 calculates the focus position according to the subject distance and adjusts the focus by moving the focus lens 40B toward the calculated focus position. Here, the focus position refers to the position of the focus lens 40B on the optical axis OA when the image is in focus.

[0066] For the sake of explanation, the control that adjusts the focus lens 40B to the focus position will also be referred to as "AF control" below. Also, for the sake of explanation, the calculation of the focus position will also be referred to as "AF calculation" below. In the imaging device 10, the CPU 48A, described later, performs AF calculation to detect the focus on multiple subjects. Then, the CPU 48A, described later, performs focusing on the subjects based on the results of the AF calculation, that is, the focus detection results.

[0067] The imaging device body 12 includes an image sensor 16, a controller 48, an image memory 50, a UI device 52, an external I / F 54, a photoelectric conversion element driver 56, a motor driver 58, a motor driver 60, a mechanical shutter driver 62, and a mechanical shutter actuator 64. The imaging device body 12 also includes a mechanical shutter 72. The image sensor 16 also includes a signal processing circuit 74.

[0068] The input / output interface 70 is connected to a controller 48, image memory 50, UI device 52, external I / F 54, photoelectric conversion element driver 56, motor driver 58, motor driver 60, mechanical shutter driver 62, and signal processing circuit 74.

[0069] The controller 48 comprises a CPU 48A, a storage 48B, and a memory 48C. The CPU 48A is an example of a "processor" relating to the technology of this disclosure, the memory 48C is an example of a "memory" relating to the technology of this disclosure, and the controller 48 is an example of an "imaging support device" and a "computer" relating to the technology of this disclosure.

[0070] The CPU 48A, storage 48B, and memory 48C are connected via a bus 76, which is connected to an input / output interface 70.

[0071] In the example shown in Figure 6, for illustrative purposes, a single bus is depicted as bus 76, but multiple buses are also possible. Bus 76 may be a serial bus, or a parallel bus including a data bus, address bus, and control bus, etc.

[0072] Storage 48B stores various parameters and programs. Storage 48B is a non-volatile memory device. Here, EEPROM is used as an example of storage 48B. EEPROM is merely an example; HDD and / or SSD may be used as storage 48B instead of EEPROM, or in conjunction with EEPROM. Memory 48C temporarily stores various information and is used as work memory. An example of memory 48C is RAM, but it is not limited to this, and other types of memory devices may be used.

[0073] Storage 48B stores various programs. The CPU 48A reads the necessary programs from storage 48B and executes the read programs on memory 48C. The CPU 48A controls the entire imaging device body 12 according to the programs executed on memory 48C. In the example shown in Figure 6, the image memory 50, UI system device 52, external I / F 54, photoelectric conversion element driver 56, motor driver 58, motor driver 60, and mechanical shutter driver 62 are controlled by the CPU 48A.

[0074] A photoelectric conversion element driver 56 is connected to the photoelectric conversion element 30. The photoelectric conversion element driver 56 supplies imaging timing signals, which define the timing of imaging performed by the photoelectric conversion element 30, to the photoelectric conversion element 30 according to instructions from the CPU 48A. The photoelectric conversion element 30 performs reset, exposure, and output of electrical signals according to the imaging timing signals supplied by the photoelectric conversion element driver 56. Examples of imaging timing signals include a vertical synchronization signal and a horizontal synchronization signal.

[0075] When the interchangeable lens 14 is attached to the imaging device body 12, the subject light incident on the imaging lens 40 is imaged onto the light-receiving surface 30A by the imaging lens 40. Under the control of the photoelectric conversion element driver 56, the photoelectric conversion element 30 converts the subject light received by the light-receiving surface 30A into electrical signals and outputs an electrical signal corresponding to the amount of subject light as analog image data representing the subject light to the signal processing circuit 74. Specifically, the signal processing circuit 74 reads out the analog image data from the photoelectric conversion element 30 in units of one frame and for each horizontal line using an exposure sequential readout method. The analog image data is broadly classified into analog phase-difference image data generated by phase-difference pixels P and analog non-phase-difference image data generated by non-phase-difference pixels N.

[0076] The signal processing circuit 74 generates digital image data by digitizing the analog image data input from the photoelectric conversion element 30. The signal processing circuit 74 includes a non-phase difference image data processing circuit 74A and a phase difference image data processing circuit 74B. The non-phase difference image data processing circuit 74A generates digital non-phase difference image data by digitizing analog non-phase difference image data. The phase difference image data processing circuit 74B generates digital phase difference image data by digitizing analog phase difference image data.

[0077] For the sake of clarity, in the following, when it is not necessary to distinguish between digital non-phase-difference image data and digital phase-difference image data, the term "digital image data" will be used. Similarly, in the following, when it is not necessary to distinguish between analog image data and digital image data, the term "image data" will be used.

[0078] The mechanical shutter 72 is a focal-plane shutter and is positioned between the aperture 40C and the light-receiving surface 30A. The mechanical shutter 72 has a front curtain (not shown) and a rear curtain (not shown). Each of the front and rear curtains has multiple blades. The front curtain is positioned closer to the subject than the rear curtain.

[0079] The mechanical shutter actuator 64 is an actuator having a front curtain solenoid (not shown) and a rear curtain solenoid (not shown). The front curtain solenoid is the drive source for the front curtain and is mechanically connected to the front curtain. The rear curtain solenoid is the drive source for the rear curtain and is mechanically connected to the rear curtain. The mechanical shutter driver 62 controls the mechanical shutter actuator 64 according to instructions from the CPU 48A.

[0080] The front curtain solenoid generates power under the control of the mechanical shutter driver 62 and selectively winds up and down the front curtain by applying the generated power to the front curtain. The rear curtain solenoid generates power under the control of the mechanical shutter driver 62 and selectively winds up and down the rear curtain by applying the generated power to the rear curtain. In the imaging device 10, the opening and closing of the front curtain and the opening and closing of the rear curtain are controlled by the CPU 48A, thereby controlling the amount of exposure to the photoelectric conversion element 30.

[0081] In the imaging device 10, imaging for live view images and imaging for recording still images and / or moving images are performed using a sequential exposure readout method (rolling shutter method). The image sensor 16 has an electronic shutter function, and imaging for live view images is achieved by activating the electronic shutter function without operating the mechanical shutter 72 while it is fully open.

[0082] In contrast, imaging with full exposure, that is, imaging for still images (hereinafter also referred to as "full exposure imaging"), is achieved by activating the electronic shutter function and operating the mechanical shutter 72 to transition from the front curtain closed state to the rear curtain closed state. The image obtained by performing full exposure imaging with the imaging device 10 (hereinafter also referred to as "full exposure image") is an example of an "imaging image" related to the technology of this disclosure.

[0083] Digital image data is stored in the image memory 50. Specifically, the non-phase difference image data processing circuit 74A causes the image memory 50 to store non-phase difference image data, and the phase difference image data processing circuit 74B causes the image memory 50 to store phase difference image data. The CPU 48A retrieves the digital image data from the image memory 50 and performs various processes using the retrieved digital image data.

[0084] The UI device 52 includes a display 26, and the CPU 48A displays various information on the display 26. The UI device 52 also includes a reception device 80. The reception device 80 includes a touch panel 28 and a hard key section 82. The hard key section 82 consists of multiple hard keys, including an instruction key 24 (see Figure 2). The CPU 48A operates according to the various instructions received by the touch panel 28. Although the hard key section 82 is included in the UI device 52 here, the technology of this disclosure is not limited to this, and for example, the hard key section 82 may be connected to an external I / F 54.

[0085] External I / F 54 is responsible for the exchange of various types of information between the imaging device 10 and devices located outside of it (hereinafter also referred to as "external devices"). An example of an external I / F 54 is a USB interface. External devices such as smart devices, personal computers, servers, USB memory, memory cards, and / or printers (not shown) can be directly or indirectly connected to the USB interface.

[0086] The motor driver 58 is connected to the motor 44 and controls the motor 44 according to instructions from the CPU 48A. By controlling the motor 44, the position of the focus lens 40B on the optical axis OA is controlled via the slide mechanism 42. The focus lens 40B moves according to instructions from the CPU 48A, avoiding the period of main exposure by the image sensor 16.

[0087] The motor driver 60 is connected to the motor 46 and controls the motor 46 according to instructions from the CPU 48A. The size of the aperture 40C is controlled by the control of the motor 46.

[0088] As an example, as shown in Figure 7, the storage 48B stores the imaging support processing program 84. The CPU 48A reads the imaging support processing program 84 from the storage 48B and executes the read imaging support processing program 84 on the memory 48C. The CPU 48A performs imaging support processing according to the imaging support processing program 84 executed on the memory 48C (see also Figures 20A and 20B).

[0089] The CPU 48A, by executing imaging support processing, first obtains frequency information indicating the frequency of subject features classified into categories based on the subject features identified from the exposure image obtained by imaging by the imaging device 10 (hereinafter also referred to as "subject features"). Next, the CPU 48A performs support processing (hereinafter simply referred to as "support processing") to assist imaging by the imaging device 10 based on the acquired frequency information. The contents of the imaging support processing will be explained in more detail below.

[0090] As an example, as shown in Figure 8, the CPU 48A operates as an acquisition unit 48A1, a subject recognition unit 48A2, a feature extraction unit 48A3, a classification unit 38A4, and a control unit 48A5 by executing the imaging support processing program 84. The classification unit 38A4 is an example of a "classifier" related to the technology of this disclosure.

[0091] As an example, as shown in Figure 9, the acquisition unit 48A1 acquires non-phase difference image data from the image memory 50 as live view image data. The live view image data is acquired by the acquisition unit 48A1 from the image memory 50 at a predetermined frame rate (e.g., 60 fps). The live view image data is image data that represents a live view image. The live view image data is obtained when the imaging area is imaged by the image sensor 16. In the example shown in Figure 9, the live view image data is obtained by imaged an imaging area containing multiple people. Here, the multiple people are an example of "multiple objects" related to the technology of this disclosure.

[0092] In this embodiment, for the sake of explanation, a person is used as an example of the subject of the imaging device 10, but the technology of this disclosure is not limited to this, and the subject may be something other than a person. Examples of subjects other than people include small animals, insects, plants, buildings, landscapes, organs of living organisms, and / or cells of living organisms. In other words, the imaging area does not have to include a person, but it does not need to include a subject that can be imaged by the image sensor 16.

[0093] Each time the acquisition unit 48A1 acquires one frame of live view image data, the control unit 48A5 displays the live view image shown by the live view image data acquired by the acquisition unit 48A1 on the display 26. The live view image includes multiple person images representing multiple people, which in turn represent multiple subject images representing multiple subjects.

[0094] As an example, as shown in Figure 10, with a live view image displayed on the display 26, a specified imaging range is determined according to instructions given from an external source (for example, instructions received by the reception device 80). For example, the specified imaging range is determined by operating the touch panel 28 with the user's finger. One purpose of determining the specified imaging range is, for example, to define the imaging range to be focused on.

[0095] In the example shown in Figure 10, the designated imaging range is the rectangular frame (in the example shown in Figure 10, the frame with a dashed line) formed by the user sliding their finger on the touch panel 28 while it is in contact with the touch panel 28. The control unit 48A5 stores designated imaging range information 86, which indicates the designated imaging range determined according to instructions given from an external source, in the storage 48B. The designated imaging range information 86 stored in the storage 48B is updated each time a new designated imaging range is determined.

[0096] As an example, as shown in Figure 11, the acquisition unit 48A1 acquires non-phase-difference image data from the image memory 50 as the main exposure image data. The subject recognition unit 48A2 recognizes the subject in the imaging area based on the main exposure image data acquired by the acquisition unit 48A1. In the example shown in Figure 11, a trained model 92 is stored in the storage 48B, and the subject recognition unit 48A2 recognizes the subject in the imaging area using the trained model 92.

[0097] One example of a pre-trained model 92 is a pre-trained model using a cascade classifier. A pre-trained model using a cascade classifier is constructed as a pre-trained model for image recognition by, for example, performing supervised machine learning on a neural network. Note that the pre-trained model 92 is not limited to a pre-trained model using a cascade classifier; it may also be a dictionary for pattern matching. In other words, the pre-trained model 92 may be any pre-trained model used in image analysis performed when an object is recognized.

[0098] The subject recognition unit 48A2 recognizes a person included in the imaging area as a subject by performing image analysis on the exposure image data. The subject recognition unit 48A2 also recognizes the person's features as subject features by performing image analysis on the exposure image data, such as the person's face (expression), posture, eye opening and closing, and whether or not the person is present within the specified imaging range.

[0099] The subject recognition unit 48A2 recognizes a person's face as one of the subject features. A person's face can be, for example, a smiling face, a crying face, an angry face, or a neutral face. The subject recognition unit 48A2 recognizes, for example, a smiling face, a crying face, an angry face, and a neutral face as subject features belonging to the category of "person's face". Here, a smiling face refers to a person's expression when they are laughing, a crying face refers to a person's expression when they are crying, an angry face refers to a person's expression when they are angry, and a neutral face refers to an expression that does not fall under any of the categories of smiling face, crying face, or angry face.

[0100] Furthermore, the subject recognition unit 48A2 recognizes the person's posture as one of the subject features. The person's posture can be, for example, either facing forward or not facing forward. The subject recognition unit 48A2 recognizes, for example, both facing forward and not facing forward as subject features belonging to the category of "person's posture". Here, "facing forward" means that the person is facing directly towards the light-receiving surface 30A (see Figure 6). "Not facing forward" means that the person is facing in a direction other than facing forward.

[0101] Furthermore, the subject recognition unit 48A2 recognizes a person's eyes as one of the subject features. A person's eyes can be either open or closed, for example. The subject recognition unit 48A2 recognizes both open and closed eyes as subject features belonging to the category "person's eyes." Here, "open eyes" refers to a state where a person's eyes are open, and "closed eyes" refers to a state where a person's eyes are closed.

[0102] Furthermore, the subject recognition unit 48A2 recognizes the presence or absence of a person within the designated imaging range as one of the subject features. The subject recognition unit 48A2 recognizes both the area within and outside the designated imaging range as subject features belonging to the category "designated imaging range". Here, "within the designated imaging range" refers to the state in which a person is present within the designated imaging range, and "outside the designated imaging range" refers to the state in which a person is present outside the designated imaging range.

[0103] Furthermore, the subject recognition unit 48A2 stores recognition result information 94 in memory 48C, which indicates the result of recognizing a subject (in this case, a person as an example) included in the imaging area. The recognition result information 94 is overwritten and saved in memory 48C on a frame-by-frame basis. The recognition result information 94 includes the subject name, subject characteristics, and recognition area specific coordinates, and is stored in memory 48C with the subject name, subject characteristics, and recognition area specific coordinates associated with each subject included in the imaging area.

[0104] Here, the recognition area identification coordinates refer to the coordinates indicating the position within the live view image of a rectangular frame (hereinafter also referred to as the "subject frame") surrounding a characteristic region of the subject image (for example, a face region indicating a person's face) that represents a subject recognized by the subject recognition unit 48A2. An example of recognition area identification coordinates is the coordinates of two vertices on the diagonal of the subject frame within the live view image (for example, the coordinates of the upper left corner and the coordinates of the lower right corner). If the shape of the subject frame is rectangular, the recognition area identification coordinates may also be the coordinates of three vertices or four vertices. Furthermore, the shape of the subject frame is not limited to a rectangle; it may be other shapes. In this case as well, it is sufficient to use coordinates that can identify the position of the subject image within the live view image as the recognition area identification coordinates.

[0105] As an example, as shown in Figure 12, the feature extraction unit 48A3 extracts subject-specific feature information from the recognition result information 94 stored in the memory 48C. Subject-specific feature information is information in which the subject name and subject features are associated on a one-to-one basis for each subject.

[0106] As an example, as shown in Figure 13, a category database 96 is constructed in storage 48B. The category database 96 contains multiple subject-specific category groups 98. One subject-specific category group 98 is assigned to each subject. In the example shown in Figure 13, the multiple subject-specific category groups 98 include a category group for person A, a category group for person B, a category group for person C, a category group for person D, and a category group for person E (each category group is assigned to a specific person). Each of the multiple subject-specific category groups 98 is a category that can identify each of the multiple people themselves. The subject-specific category group 98 is an example of an "object target category" related to the technology of this disclosure.

[0107] The classification unit 48A4 identifies a subject-specific category group 98 corresponding to a subject name from the subject name included in the subject-specific feature information, and classifies the subject features corresponding to the subject name into the identified subject-specific category group 98.

[0108] The subject-specific category group 98 contains multiple categories. Each of the multiple categories included in the subject-specific category group 98 is a category created for each independent unit. A unit refers to a unit of subject characteristics. In the example shown in Figure 13, examples of multiple categories included in the subject-specific category group 98 are shown, such as face category, posture category, eye category, and specified imaging range category. The face category is a category defined by the subject characteristics of the unit "person's face". The posture category is a category defined by the subject characteristics of the unit "person's posture". The eye category is a category defined by the subject characteristics of the unit "person's eye". The specified imaging range category is a category defined by the subject characteristics of the unit "specified imaging range" applied to a person.

[0109] As an example, as shown in Figure 14, when the face category, posture category, eye category, and specified imaging range category are defined as major categories, multiple subcategories are subordinate to each major category. A subcategory is a category into which subject features that are subordinate to the subject features indicated by the major category are classified. Each of the multiple subcategories is associated with a classification count. Subject features are classified into the subcategory by the classification unit 48A4. Each time a subject feature is classified and added to a subcategory, "1" is added to the classification count to indicate that the subject feature has been classified.

[0110] In the example shown in Figure 14, the face category within the subject-specific category group 98 used for person A includes several categories: smiling face category, crying face category, angry face category, and neutral face category. The classification unit 48A4 classifies person A's subject feature "smiling" into the smiling face category. Then, each time the classification unit 48A4 classifies the subject feature "smiling" into the smiling face category, it adds "1" to the classification count for the smiling face category. Similarly to the subject features "crying face," "angry face," and "neutral face" of person A, the classification unit 48A4 classifies them into their respective categories, and "1" is added to the corresponding classification count.

[0111] The number of classifications is also associated with the face category. The number of classifications for the face category is the sum of the number of classifications for the smiling face category, the crying face category, the angry face category, and the neutral face category.

[0112] Furthermore, in the example shown in Figure 14, the frontal category and the non-frontal category are shown as multiple categories included in the posture category within the subject-specific category group 98 used for person A. The classification unit 48A4 classifies the subject feature "frontal" of person A into the frontal category. Then, each time the classification unit 48A4 classifies the subject feature "frontal" into the frontal category, it adds "1" to the classification count for the frontal category. Similarly, the subject feature "non-frontal" of person A is also classified into the non-frontal category by the classification unit 48A4, and "1" is added to the corresponding classification count.

[0113] The number of classifications is also associated with the posture category. The number of classifications for a posture category is the sum of the number of classifications for the frontal category and the number of classifications for the non-frontal category.

[0114] Furthermore, in the example shown in Figure 14, the eye category within the subject category group 98 used for person A includes multiple categories such as the open-eyes category and the closed-eyes category. The classification unit 48A4 classifies the subject feature "open eyes" of person A into the open-eyes category. Then, each time the classification unit 48A4 classifies the subject feature "open eyes" into the open-eyes category, it adds "1" to the classification count for the open-eyes category. Similarly, the subject feature "closed eyes" of person A is also classified into the closed-eyes category by the classification unit 48A4, and "1" is added to the corresponding classification count.

[0115] The number of classifications is also associated with the eye category. The number of classifications for the eye category is the sum of the number of classifications for the open-eye category and the number of classifications for the closed-eye category.

[0116] Furthermore, in the example shown in Figure 14, the categories included in the specified imaging range category within the subject-specific category group 98 used for person A are shown as categories within the specified imaging range and categories outside the specified imaging range. The classification unit 48A4 classifies the subject feature "within the specified imaging range" into the specified imaging range category. Then, each time the classification unit 48A4 classifies the subject feature "within the specified imaging range" into the specified imaging range category, it adds "1" to the classification count for the specified imaging range category. Similarly, for the subject feature "outside the specified imaging range," the classification unit 48A4 classifies it into the category outside the specified imaging range, and "1" is added to the corresponding classification count.

[0117] The number of classifications is also associated with the specified imaging range category. The number of classifications for a specified imaging range category is the sum of the number of classifications for categories within the specified imaging range and the number of classifications for categories outside the specified imaging range.

[0118] The subject-specific category group 98 is a category defined by subject characteristics in the unit of "subject name," and the subject characteristic "subject name" is classified into the subject-specific category group 98. A classification count is also associated with the subject-specific category group 98. The classification count associated with the subject-specific category group 98 is the sum of the classification counts of the multiple major categories belonging to the subject-specific category group 98.

[0119] As an example, as shown in Figure 15, the acquisition unit 48A1 acquires non-phase-difference image data from the image memory 50 as live view image data. The subject recognition unit 48A2 recognizes a subject within the imaging area using the trained model 92 based on the live view image data acquired by the acquisition unit 48A1.

[0120] The subject recognition unit 48A2 recognizes a person included in the imaging area as a subject by performing image analysis on the live view image data. The subject recognition unit 48A2 also recognizes the person's features as subject features by performing image analysis on the live view image data, such as the person's face, posture, whether the person's eyes are open or closed, and whether the person is present or absent within the specified imaging range.

[0121] Specifically, the subject recognition unit 48A2 performs image analysis on the live view image data to recognize subject features belonging to the face category, such as smiling, crying, angry, and neutral faces. The subject recognition unit 48A2 also performs image analysis on the live view image data to recognize frontal and non-frontal views as subject features belonging to the posture category. The subject recognition unit 48A2 also performs image analysis on the live view image data to recognize open eyes and closed eyes as subject features belonging to the eye category. Furthermore, the subject recognition unit 48A2 performs image analysis on the live view image data to recognize subjects within and outside the specified imaging range as subject features belonging to the specified imaging range category. The subject recognition unit 48A2 stores recognition result information 94, which indicates the result of recognizing a person included in the imaging area, in the memory 48C. The recognition result information 94 is overwritten and saved in the memory 48C on a frame-by-frame basis.

[0122] As an example, as shown in Figure 16, the control unit 48A5 displays the live view image shown by the live view image data acquired by the acquisition unit 48A1 on the display 26. The control unit 48A5 also refers to the recognition result information 94 stored in the memory 48C and obtains the number of classifications for each category of the person recognized as a subject by the subject recognition unit 48A2 (hereinafter also referred to as "recognized person") based on the live view image data, from each category included in the category database 96. In other words, the control unit 48A5 obtains the number of classifications for each recognized person from each category of the subject-specific category group 98 corresponding to the recognized person.

[0123] The control unit 48A5 generates an imaging support screen 100 based on the number of classifications obtained from each category in the category database 96, and overlays the generated imaging support screen 100 onto the live view image. In the example shown in Figure 16, the imaging support screen 100 is displayed in the upper left of the live view image.

[0124] As an example, as shown in Figure 17, the imaging support screen 100 displays a bubble chart 100A and a category selection screen 100B. The bubble chart 100A is a chart in which bubbles indicating the number of classifications are plotted on two axes: an axis indicating multiple recognized individuals and an axis indicating categories.

[0125] In the example shown in Figure 17, a bubble chart relating to face categories (hereinafter also referred to as the "face category bubble chart") is shown as an example of bubble chart 100A. The face category bubble chart is a chart in which bubbles representing the number of times each of the face categories belonging to the smile category, crying face category, angry face category, and neutral face category are classified are plotted for each of the multiple people, including people A to E.

[0126] In addition to the face category bubble chart, the imaging support screen 100 selectively displays bubble charts related to posture categories (hereinafter also referred to as the "posture category bubble chart"), eye categories (hereinafter also referred to as the "eye category bubble chart"), and specified imaging range categories (hereinafter also referred to as the "specified imaging range category bubble chart") according to instructions given from an external source. Here, instructions given from an external source include, for example, instructions received by the reception device 80. In the example shown in Figure 17, multiple soft keys indicating the names of each category are displayed in a scrolling manner within the category selection screen 100B. When any of the soft keys are turned on via the touch panel by the user's finger, the bubble chart related to the category corresponding to the turned-on soft key is displayed as a new bubble chart 100A within the imaging support screen 100.

[0127] Furthermore, when a person in the bubble chart 100A is selected via the touch panel by the user's finger, the number of classifications for each category in the currently displayed bubble chart 100A for the selected person is displayed as a histogram 100C (see Figure 18) on the imaging support screen 100. In the example shown in Figure 17, person A in the bubble chart 100A is selected via the touch panel by the user's finger. In this case, as shown in Figure 18 as an example, a histogram showing the imaging trend of face categories for person A (hereinafter also referred to as the "face category histogram") is displayed as histogram 100C. In the face category histogram, the horizontal axis represents the categories of smiling, crying, angry, and neutral, and the vertical axis represents the number of classifications.

[0128] In the example shown in Figure 18, the face category histogram is shown as histogram 100C. However, different types of histograms are displayed as histogram 100C depending on the type of bubble chart 100A and the person selected by the user. In addition to the face category histogram, other types of histogram 100C include, for example, a histogram showing the imaging trend of the posture category for each selected person (hereinafter also referred to as the "posture category histogram"), a histogram showing the imaging trend of the eye category (hereinafter also referred to as the "eye category histogram"), and a histogram showing the imaging trend of the specified imaging range category (hereinafter also referred to as the "specified imaging range category histogram"). Furthermore, the histogram is not limited to individual histograms; it may also be a histogram where the horizontal axis represents people A to E and the vertical axis represents the number of times each subject was classified into category group 98. Thus, the horizontal and vertical axes of the histogram may be axes of any element that can constitute a histogram.

[0129] In the example shown in Figure 18, the display switching instruction screen 100D is displayed within the imaging support screen 100. The display switching instruction screen 100D is a screen that accepts instructions to switch from the display of the histogram 100C to the display of the bubble chart 100A. The display switching instruction screen 100D displays a message guiding the user to switch to the bubble chart 100A (in the example shown in Figure 18, the message is "Do you want to return to the bubble chart screen?") and a switching soft key (in the example shown in Figure 18, a soft key labeled "Yes"). When the user turns on the switching soft key via the touch panel using their finger, the display switches from the histogram 100C to the display of the bubble chart 100A.

[0130] Here, the reception device 80 (touch panel 28 in the example shown in Figure 18) specifies the number of classifications for any category in the histogram 100C. In the example shown in Figure 18, the number of classifications for the smile category is specified by the user selecting the smile category via the touch panel 28 with their finger. When the number of classifications for any category is specified in this way, the control unit 48A5 performs support processing for the category corresponding to the specified number of classifications (hereinafter also referred to as the "target category").

[0131] Furthermore, in the example shown in Figure 18, the "smile" category is selected as the target category from among multiple categories (smile category, crying face category, angry face category, and neutral face category) in the histogram 100C for person A. The target category is a category determined based on the number of classifications (in the example shown in Figure 18, the category specified based on the number of classifications according to the instructions received by the touch panel 28). In this case, as a support process, processing is performed to support imaging using the image sensor 16 of person A (hereinafter also referred to as "target category subject S" (see Figure 19)), who has the subject feature "smile" which belongs to the smile category.

[0132] When a subject of a target category is imaged by the imaging device 10, the target category is a category determined according to the state of the subject S of the target category (for example, facial expression, posture, eye opening / closing state, and / or the positional relationship between the person and the designated imaging range), and is a category into which the subject characteristics of the subject S of the target category are classified. In the example shown in Figure 18, when person A is imaged by the imaging device 10 as the subject S of the target category, the user has designated the smile category as the target category, which is the subject characteristic of person A's smile, crying face, angry face, and neutral face, as the smile is the expression that is least often captured.

[0133] From histogram 100C, the smile category, designated as a target category by the user, has the lowest classification count compared to the crying face category, angry face category, and neutral face category. This means that the number of main exposure images in which person A is smiling is less than the number of main exposure images in which person A is smiling with other expressions. In histogram 100C, the smile category is an example of a "low-frequency category" related to the technology disclosed herein.

[0134] When a target category is specified from the histogram 100C in this way, as shown in Figure 19 as an example, the control unit 48A5 displays the live view image obtained by the imaging device 10 capturing the imaging area including the target category subject S on the display 26, and as a support process, identifies the smiling target category subject S based on the recognition result information 94 and performs display processing to display a message recommending that the identified target category subject S be captured. The display processing includes displaying an arrow on the live view image that points to the target category subject image S1 indicating the smiling target category subject S, and displaying information recommending that the target category subject S indicated by the target category subject image S1 pointed to by the arrow be captured (hereinafter also referred to as "imaging recommendation information"). In the example shown in Figure 19, the message "We recommend that you take a picture of this subject" is displayed as an example of the imaging recommendation information.

[0135] Furthermore, the display process includes a process for displaying the target category subject image S1 in a manner that makes it distinguishable from other image areas within the live view image. In this case, for example, the control unit 48A5 detects the face area (the image area showing the face of the target category subject S) of the target category subject image based on the recognition result information 94 stored in the memory 48C, and displays the detection frame 102 surrounding the detected face area within the live view image, thereby displaying the target category subject image S1 in a manner that makes it distinguishable from other image areas within the live view image.

[0136] The method of displaying the target category subject image S1 in the live view image in a manner that allows it to be distinguished from other image areas is not limited to this; for example, only the target category subject image S1 may be displayed in the live view image using a peaking method.

[0137] Next, the operation of the imaging device 10 will be explained with reference to Figures 20A and 20B.

[0138] Figures 20A and 20B show an example of the flow of imaging support processing executed by the CPU 48A when an imaging mode is set for the imaging device 10. The flow of imaging support processing is an example of an "imaging support method" related to the technology of this disclosure. For the sake of explanation, the following explanation assumes that the imaging device 10 is capturing images for live view at a predetermined frame rate. Also, for the sake of explanation, the following explanation assumes that the specified imaging range information 86 is stored in the storage 48B and that the category database 96 has been constructed.

[0139] In the imaging support process shown in Figure 20A, first, in step ST100, the acquisition unit 48A1 acquires live view image data from the image memory 50.

[0140] In the next step, ST102, the control unit 48A5 displays the live view image, which is shown by the live view image data acquired in step ST100, on the display 26.

[0141] In the next step, ST104, the subject recognition unit 48A2 recognizes a subject within the imaging area using the trained model 92 based on the live view image data acquired in step ST100.

[0142] In the next step, ST106, the control unit 48A5 obtains the number of classifications for each category of the subject recognized in step ST104 from the category database 96.

[0143] In the next step, ST108, the control unit 48A5 creates an imaging support screen 100 based on the classification count acquired in step ST106, and displays the created imaging support screen 100 in a portion of the live view image.

[0144] In the next step, ST110, the control unit 48A5 determines from the histogram 100C in the imaging support screen 100 whether or not a classification count has already been specified. If, in step ST110, a classification count has not yet been specified from the histogram 100C in the imaging support screen 100, the determination is denied, and the imaging support process proceeds to step ST116 shown in Figure 20B. If, in step ST110, a classification count has already been specified from the histogram 100C in the imaging support screen 100, the determination is affirmed, and the imaging support process proceeds to step ST112.

[0145] In step ST112, the control unit 48A5 displays a detection frame 102 in the live view image so as to surround the face region of the target category subject image S1, which shows a target category subject S having subject characteristics belonging to a specified target category based on the specified number of classifications.

[0146] In the next step, ST114, the control unit 48A5 displays imaging recommendation information within the live view image. After the processing in step ST114 is completed, the imaging support process proceeds to step ST116, shown in Figure 20B.

[0147] In step ST116, the control unit 48A5 determines whether the conditions for starting the main exposure (hereinafter referred to as "main exposure start conditions") have been met. An example of a main exposure start condition is that the fully pressed state described above has been reached by an instruction received by the reception device 80. If the main exposure start conditions are not met in step ST116, the determination is denied, and the imaging support process proceeds to step ST130. If the main exposure start conditions are met in step ST116, the determination is affirmed, and the imaging support process proceeds to step ST118.

[0148] In step ST118, the control unit 48A5 instructs the image sensor 16 to perform a main exposure image for the imaging area including the target category subject S. As a result of this main exposure image, the image memory 50 stores main exposure image data showing the main exposure image of the imaging area including the target category subject S.

[0149] In the next step, ST120, the acquisition unit 48A1 acquires the exposure image data from the image memory 50.

[0150] In the next step, ST122, the subject recognition unit 48A2 recognizes a subject within the imaging area using the trained model 92 based on the exposure image data acquired in step ST120, and stores the recognition result information 94 in the memory 48C.

[0151] In the next step, ST124, the feature extraction unit 48A3 extracts subject-specific feature information for each subject from the recognition result information 94 stored in the memory 48C.

[0152] In the next step, ST126, the classification unit 48A4 identifies a group of subject categories 98 corresponding to the subject name from the subject name included in the subject-specific feature information extracted in step ST124, and classifies the subject features into the corresponding category within the identified group of subject categories 98.

[0153] In the next step, ST128, the classification unit 48A4 updates the classification count by adding "1" to the classification count of the category into which the subject features have been classified.

[0154] In the next step, ST130, the control unit 48A5 erases the live view image and other data (for example, the live view image, the imaging support screen 100, the detection frame 102, and the imaging recommendation information) from the display 26.

[0155] In the next step, ST132, the control unit 48A5 determines whether the conditions for terminating the imaging support process (hereinafter also referred to as "imaging support process termination conditions") have been met. Examples of imaging support process termination conditions include the condition that the imaging mode set for the imaging device 10 has been released, or the condition that an instruction to terminate the imaging support process has been received by the receiving device 80. If the imaging support process termination conditions are not met in step ST120, the determination is denied, and the imaging support process proceeds to step ST100. If the imaging support process termination conditions are met in step ST132, the determination is affirmed, and the imaging support process ends.

[0156] As explained above, in the imaging device 10, the control unit 48A5 acquires the number of classifications of subject features that have been categorized based on the subject features identified from the exposure image. The control unit 48A5 then performs support processing to assist imaging by the imaging device 10 based on the classification count. Therefore, with this configuration, imaging by the imaging device 10 can be supported according to the number of classifications in which subject features are categorized.

[0157] Furthermore, in the imaging device 10, categories are classified into multiple categories, including the target category. The target category is a category determined based on the number of classifications. In the example shown in Figure 18, the target category is specified by the user selecting the number of classifications. Then, the control unit 48A5 performs processing to support imaging of the target category subject S that has subject characteristics belonging to the target category. In the example shown in Figure 19, processing is performed to support imaging of the target category subject S that is smiling.

[0158] In the example shown in Figure 19, the smile category is specified by the user selecting the number of classifications for the smile category, and therefore imaging of the target subject S of the smile category is supported. However, if the user selects the number of classifications for other subject features (e.g., crying face, angry face, and / or neutral face), and a subcategory other than the smile category is specified among the multiple subcategories within the face category, the system will support imaging of the target subject S of the specified target category that has subject features belonging to that category.

[0159] When a subcategory is specified by the user, not only by subcategories belonging to the face category, but also by selecting the number of times subject features classified into subcategories belonging to other major categories are classified, the system performs image acquisition support processing for the target category subject S that has subject features belonging to the subcategory specified as the target category.

[0160] Furthermore, if the subject category group 98 is specified by the user selecting the number of times a subject name classified as a subject characteristic has been classified in the subject category group 98, then the process to support imaging is performed for the target category subject S that has subject characteristics belonging to the subject category group 98 specified as the target category, that is, the target category subject S whose subject name corresponds to the subject category group 98 specified as the target category.

[0161] Therefore, with this configuration, it is possible to efficiently capture the target category subject S desired by the user, compared to a case where the image is taken based solely on the imager's intuition to determine whether or not the subject S is in the target category desired by the user.

[0162] Furthermore, in this embodiment, as a display recommending the imaging of the target category subject S, imaging recommendation information (in the example shown in Figure 19, the message "We recommend taking a picture of this subject") is displayed. Therefore, with this configuration, compared to a case where imaging is performed based solely on the photographer's intuition to determine whether or not the target category subject S has the subject characteristics desired by the user, it is possible to efficiently image the target category subject with the expression desired by the user.

[0163] Furthermore, in this embodiment, the control unit 48A5 displays the live view image on the display 26, and the target category subject image S1 is displayed within the live view image in a manner that allows it to be distinguished from other image areas. Therefore, this configuration allows the user to visually recognize the target category subject.

[0164] Furthermore, in this embodiment, the control unit 48A5 displays the live view image on the display 26, and the detection frame 102 is displayed for the face region of the target category subject image S1 within the live view image. Therefore, with this configuration, the user can recognize that the subject corresponding to the display area where the detection frame 102 is displayed is a target category subject.

[0165] Furthermore, in this embodiment, the user specifies a target category that has a relatively low classification count among multiple categories (in the example shown in Figure 18, the "smile" category). Therefore, with this configuration, compared to a case where imaging is performed based solely on the imager's intuition to determine whether or not a subject belongs to a category with a relatively low classification count among multiple categories, it is possible to efficiently image subjects belonging to categories with a relatively low classification count among multiple categories.

[0166] Furthermore, in this embodiment, the target category is a category determined according to the state of the target category subject S (for example, a person's facial expression). In the example shown in Figure 19, the target category is specified as "smile," which is a smile category determined according to the state of the subject S. The control unit 48A5 then performs processing to support imaging of the target category subject S, which is a subject having subject characteristics classified into the specified category. Therefore, with this configuration, it is possible to increase the number of times an image is taken of a subject in the same state, or conversely, to decrease the number of times an image is taken of a subject in the same state.

[0167] Furthermore, in this embodiment, multiple subject-specific category groups 98 are included in the category database 96 as categories that can identify each of multiple individuals. Therefore, the control unit 48A5 performs processing to support imaging of the target category subject S, which is a subject corresponding to the subject-specific category group 98 specified by the user. Accordingly, with this configuration, it is possible to increase the number of times the same subject is imaged, or conversely, to decrease the number of times the same subject is imaged.

[0168] Furthermore, in this embodiment, categories into which subject features are classified are created for each of several units. Examples of these units include "subject name (e.g., a person's name or an identifier that can identify a person)," "person's face (expression)," "person's posture," "person's eyes," and "specified imaging range." Therefore, with this configuration, imaging by the imaging device 10 can be supported according to the number of classifications in which subject features are classified into the category of the specified unit.

[0169] Furthermore, in this embodiment, the control unit 48A5 performs a process that includes displaying the number of classifications on the display 26 as support processing for imaging by the imaging device 10. In the example shown in Figure 17, the number of classifications is represented using a bubble chart 100A, and in the example shown in Figure 18, the number of classifications is represented using a histogram 100C. Therefore, with this configuration, the user can understand the number of times imaging has been performed for each of the various subject features.

[0170] Furthermore, in this embodiment, when the histogram 100C is displayed on the display 26 and the number of classifications in the histogram 100C is specified by the reception device 80, the control unit 48A5 performs processing to support imaging related to the category corresponding to the specified number of classifications. Therefore, this configuration makes it possible to support imaging of subjects that have the subject characteristics intended by the user.

[0171] In the above embodiment, an example was given in which the target category subject S is located within the designated imaging range, but the technology of this disclosure is not limited to this. For example, if the target category subject S is located outside the designated imaging range, as shown in Figure 21 as an example, the target category subject image S1 is displayed in the live view image at a position outside the object Ob indicating the designated imaging range. In this case, the control unit 48A5 displays the object Ob indicating the designated imaging range and the target category subject image S1 (an example of the "object indicating a target category subject" related to the technology of this disclosure) in different display modes in the live view image. In the example shown in Figure 21, the target category subject image S1 is displayed using a peaking method, and the object Ob is colored with a semi-transparent color (for example, semi-transparent gray). This allows the user to visually distinguish between the designated imaging range intended as the target of imaging and the target category subject S.

[0172] Furthermore, in the above embodiment, the number of classifications shown in the histogram 100C is specified by the user via the touch panel 28, thereby supporting imaging related to the category corresponding to the specified number of classifications (in the example shown in Figure 18, the smiling category). However, the technology of this disclosure is not limited thereto. For example, the control unit 48A5 may identify a low-frequency category with a relatively low number of classifications among a plurality of categories as a target category, and perform processing to support imaging related to the identified low-frequency category. For example, a low-frequency category refers to the category with the fewest classifications (e.g., the smiling category) among a plurality of subcategories (smiling category, crying face category, angry face category, and neutral face category) belonging to a large category (e.g., face category) specified by the user via the reception device 80. In the live view image, the control unit 48A5 identifies a subject of interest (e.g., person A) that has subject characteristics classified as a low-frequency category among the subject of interest specified by the user via the reception device 80, and performs processing to support imaging of the identified subject of interest. In this case, selection (operation by the user) on the histogram 100C is not required, and the imaging support screen 100 does not need to be displayed.

[0173] Thus, when the control unit 48A5 performs processing to support imaging of a subject of interest having subject features classified as a low-frequency category, the imaging support processing is executed by the CPU 48A, as shown in Figure 22 as an example. The flowchart shown in Figure 22 differs from the flowchart shown in Figure 20A in that it includes steps ST200 and ST202 instead of steps ST106 to ST112.

[0174] In step ST200 of the imaging support processing shown in Figure 22, the control unit 48A5 identifies a low-frequency category from among multiple subcategories within the major category specified by the user for the subject of interest specified by the user. For example, according to the bubble chart 100A shown in Figure 22, if the user specifies the face category as the major category and person A as the subject of interest, the control unit 48A5 identifies the smile category as a low-frequency category.

[0175] In the next step, ST202, the control unit 48A5 identifies a subject image (hereinafter also referred to as the "subject of interest image") in the live view image that represents a subject of interest that has subject features classified as a low-frequency category among the subjects of interest specified by the user, and displays a detection frame 102 (see Figures 19 and 21) around the face region of the subject of interest image. For example, if person A is specified as the subject of interest by the user, and the smiling category is automatically identified as a low-frequency category in step ST200, first, the control unit 48A5 identifies a person image representing smiling person A within the live view image as the subject of interest image based on the recognition result information 94. Then, the control unit 48A5 displays a detection frame 102 surrounding the face region of the person image representing smiling person A within the live view image. In this embodiment, "automatic" means that the controller 48 is the primary operator and does not act in response to human operation (e.g., operation by the user).

[0176] Therefore, as shown in the example in Figure 22, imaging of subjects with subject characteristics classified into the category with the fewest classification counts is supported. This allows for more efficient imaging of subjects in low-frequency categories compared to cases where imaging is performed based solely on the imager's intuition as to whether or not a subject belongs to a low-frequency category.

[0177] In the above embodiment, an example was described in which the exposure imaging is performed when the condition that a user instruction is received by the receiving device 80 is satisfied as the exposure start condition in the imaging support process. However, the technology of this disclosure is not limited thereto. For example, the control unit 48A5 may detect a target category subject based on the imaging result from the imaging device 10, and on the condition that a target category subject has been detected, it may automatically acquire an image including an image corresponding to the target category subject.

[0178] In this case, for example, the control unit 48A5 first identifies a low-frequency category as the target category, similar to the example shown in Figure 22. Then, based on the recognition result information 94, the control unit 48A5 detects the presence of a low-frequency subject of interest image, which is a subject of interest image corresponding to a low-frequency category, in the live view image, and automatically starts the main exposure imaging. Here, a low-frequency subject of interest image corresponding to a low-frequency category refers to, for example, a subject of interest image of person A that has the subject feature "smiling," assuming that person A is designated as the subject of interest, when the control unit 48A5 identifies the smile category as a low-frequency category. When this exposure imaging is performed, the main exposure image data for one frame may be stored in a predetermined storage area (for example, image memory 50), or only the data related to the low-frequency subject of interest image from the main exposure image data for one frame may be stored in the predetermined storage area.

[0179] Thus, when an image containing an image corresponding to a target category subject is automatically acquired by the control unit 48A5, provided that a target category subject has been detected, the image acquisition support process is executed by the CPU 48A, as shown in Figure 23 as an example. The flowchart shown in Figure 23 differs from the flowcharts shown in Figures 20A and 20B in that it includes steps ST300 and ST302 instead of steps ST106 to ST116.

[0180] In step ST300 of the imaging support process shown in Figure 23, the control unit 48A5 performs the same processing as in step ST200 shown in Figure 22.

[0181] In the next step, ST302, the control unit 48A5 determines whether or not an image of a low-frequency subject of interest exists in the live view image based on the recognition result information 94. If, in step ST302, an image of a low-frequency subject of interest does not exist in the live view image, the determination is denied, and the imaging support process proceeds to step ST130. If, in step ST302, an image of a low-frequency subject of interest exists in the live view image, the determination is affirmed, and the imaging support process proceeds to step ST118.

[0182] Thus, since the control unit 48A5 starts the exposure imaging when it detects the presence of a low-frequency subject of interest in the live view image, the effort required to image the target category subject can be reduced compared to when imaging is started only after the target category subject is visually identified and a user instruction is received by the receiving device 80.

[0183] In the example shown in Figure 23, it is explained that in step ST302 of the imaging support processing, it is determined whether or not an image of a low-frequency subject of interest exists in the live view image, and if it is determined that an image of a low-frequency subject of interest exists in the live view image, the exposure imaging is started. However, the technology of this disclosure is not limited to this. For example, instead of the processing in step ST302 shown in Figure 23, the processing in step ST352 shown in Figure 24 may be executed by the control unit 48A5.

[0184] In step ST352 of the imaging support process shown in Figure 24, the control unit 48A5 determines whether or not the imaging range condition is satisfied. Here, the imaging range condition refers to the condition that the target category subject is included within the specified imaging range indicated by the specified imaging range information 86 (see Figures 10 and 11). The determination of whether or not the target category subject is included within the specified imaging range is performed, for example, by determining whether or not the subject of interest indicated by the low-frequency subject of interest image is included within the specified imaging range, based on the specified imaging range information 86 and the recognition result information 94. If the imaging range condition is not satisfied in step ST352, the determination is denied and the imaging support process proceeds to step ST130. If the imaging range condition is satisfied in step ST352, the determination is affirmed and the imaging support process proceeds to step ST118.

[0185] Thus, since this exposure imaging is performed on the condition that the target category subject is included within the specified imaging range, the effort required to image the target category subject can be reduced compared to the case where imaging is started only after visually detecting that the target category subject is included within the specified imaging range and receiving instructions from the user via the receiving device 80.

[0186] In the example shown in Figure 24, the main exposure imaging is performed in step ST118 of the imaging support processing, but imaging may be performed with focus on a target category subject within the specified imaging range. Here, "imaging" may refer to imaging for live view images or imaging for recorded images (e.g., still images or moving images).

[0187] In this case, the imaging support process shown in Figure 25 is executed by CPU 48A. The flowchart shown in Figure 25 differs from the flowchart shown in Figure 24 in that it has step ST400 instead of step ST118.

[0188] In step ST400 of the imaging support process shown in Figure 25, the control unit 48A5 focuses on the target category subject (for example, a subject of interest indicated by a low-frequency subject of interest image in the live view image) in the imaging area including the target category subject S, based on the result of the AF calculation, and then causes the image sensor 16 to perform the main exposure imaging. After the processing of step ST400 is completed, the imaging support process proceeds to step ST120.

[0189] In this way, the camera focuses on the target category subject within the specified imaging range before performing the main exposure image. This reduces the effort required to position the target category subject within the specified imaging range and then focus on it before taking the image.

[0190] In the example shown in Figure 25, an example of a configuration in which the main exposure imaging is performed when the imaging range conditions are satisfied was given, but the technology of this disclosure is not limited thereto. For example, the control unit 48A5 may perform a default processing when the degree of difference between the first imaging conditions given from an external source (for example, imaging conditions determined according to instructions received by the reception device 80) and the second imaging conditions given for the target category subject is greater than or equal to a predetermined difference.

[0191] In this case, for example, the imaging support process shown in Figure 26 is executed by CPU 48A. The flowchart shown in Figure 26 differs from the flowchart shown in Figure 25 in that it has steps ST450 to ST456 instead of steps ST352, ST400, and ST120.

[0192] In step ST450 of the imaging support process shown in Figure 26, the control unit 48A5 acquires the first imaging conditions and the second imaging conditions. Examples of the first and second imaging conditions will be described later.

[0193] In the next step, ST452, the control unit 48A5 calculates the degree of difference between the first imaging condition and the second imaging condition acquired in step ST450 (for example, a value indicating how much the first imaging condition and the second imaging condition are different).

[0194] In the next step, ST454, the control unit 48A5 determines whether the difference calculated in step ST452 is greater than or equal to a predetermined difference. If the difference calculated in step ST452 is less than the predetermined difference in step ST454, the determination is rejected, and the control unit 48A5 executes the processing corresponding to step ST400 and steps ST120 to ST132 (see Figure 25) before proceeding to step ST100. If the difference calculated in step ST452 is greater than or equal to the predetermined difference in step ST454, the determination is affirmed, and the imaging support process proceeds to step ST456.

[0195] In step ST456, the control unit 48A5 executes default processing. As will be described in detail later, default processing includes depth-of-field adjustment imaging processing, which performs main exposure imaging after adjusting the depth of field, and / or focus bracket imaging processing, which performs main exposure imaging using the focus bracketing method. After the processing in step ST456 is executed, the imaging support processing moves to step ST122.

[0196] As shown in the example in Figure 26, default processing is performed when the difference between the first imaging condition given externally and the second imaging condition given for the target category subject is greater than or equal to a predetermined difference, thus contributing to imaging under the imaging conditions intended by the user.

[0197] The examples shown in Figures 27 and 28 illustrate the post-depth-of-field imaging process. As an example, as shown in Figure 27, if the first imaging condition is the position within the specified imaging range and the second imaging condition is the position of the target category subject, and the position of the target category subject is not within the specified imaging range, the control unit 48A5 determines that the degree of difference is greater than or equal to a predetermined degree of difference. If the degree of difference is greater than or equal to a predetermined degree of difference, the control unit 48A5 controls the imaging device 10 to bring the specified imaging range and the target category subject into the depth of field, and causes the image sensor 16 to perform the main exposure imaging with the specified imaging range and the target category subject in the depth of field.

[0198] If the degree of difference is greater than or equal to the predetermined degree of difference, as shown in Figure 28 as an example, the acquisition unit 48A1 first calculates the focus position for each of the multiple subjects, namely, at least one subject within the specified imaging range (hereinafter also referred to as "subject within the specified imaging range") and the target category subject, based on the results of the AF calculation for each of the subject within the specified imaging range and the target category subject. For example, the acquisition unit 48A1 calculates the focus position for each of the subject within the specified imaging range and the target category subject based on the phase difference image data corresponding to each position of the subject image within the specified imaging range (image showing the subject within the specified imaging range) and the target category subject image S1 in the live view image. Note that the method for calculating the focus position is merely an example, and the focus position may also be calculated using a TOF method or a contrast AF method.

[0199] The acquisition unit 48A1 calculates the depth of field that encompasses the subject within the specified imaging range and the subject within the target category, based on multiple focus positions calculated for each subject within the specified imaging range and the subject within the target category. The depth of field is calculated using a first calculation formula. The first calculation formula is, for example, a formula in which multiple focus positions are independent variables and the depth of field is the dependent variable. Alternatively, a first table in which multiple focus positions and depth of field are associated may be used instead of the first calculation formula.

[0200] The acquisition unit 48A1 calculates the F-number that achieves the calculated depth of field. The acquisition unit 48A1 calculates the F-number using a second calculation formula. The second calculation formula used here is, for example, a formula in which depth of field is the independent variable and F-number is the dependent variable. Alternatively, a second table in which values ​​representing depth of field and F-numbers are associated may be used instead of the second calculation formula.

[0201] The control unit 48A5 operates the aperture 40C by controlling the motor 46 via the motor driver 60 according to the F value calculated by the acquisition unit 48A1.

[0202] As shown in the examples in Figures 27 and 28, when the position of the target category subject S is not within the specified imaging range, the aperture 40C is activated so that the subject within the specified imaging range and the target category subject S are within the depth of field, and the main exposure imaging is performed with the subject within the specified imaging range and the target category subject within the depth of field. As a result, compared to when the subject within the specified imaging range and the target category subject S are not within the depth of field, images with higher contrast can be obtained as images of the subject within the specified imaging range and the target category subject without having to take multiple images.

[0203] Incidentally, due to the structure of the imaging device 10, there may be situations where both the subject within the specified imaging range and the target category subject S are not within the depth of field. In such a situation, the control unit 48A5 instructs the imaging device 10 to image both the subject within the specified imaging range and the target category subject using a focus bracketing method.

[0204] In this case, as shown in Figure 29 as an example, the acquisition unit 48A1 first calculates the first and second focus positions based on the phase difference image data corresponding to each position of the subject image and the target category subject image S1 within the specified imaging range in the live view image. The first focus position is the focus position relative to the center of the subject within the specified imaging range, and the second focus position is the focus position relative to the target category subject S. The first and second focus positions are the focus positions used when the main exposure imaging is performed using the focus bracketing method.

[0205] Once the acquisition unit 48A1 calculates the first and second focus positions, as shown in Figure 30 as an example, the control unit 48A5 moves the focus lens 40B to the first focus position and instructs the image sensor to start the main exposure imaging at the timing when the focus lens 40B reaches the first focus position. In response, the image sensor 16 performs the main exposure imaging. After the main exposure imaging is performed with the focus lens 40B aligned to the first focus position, the control unit 48A5 moves the focus lens 40B to the second focus position and instructs the image sensor to start the main exposure imaging at the timing when the focus lens 40B reaches the second focus position. In response, the image sensor 16 performs the main exposure imaging. Here, the first and second focus positions are used as examples of focus positions used in the main exposure imaging using the focus bracketing method, but this is only an example, and three or more focus positions may be used as focus positions for the main exposure imaging using the focus bracketing method.

[0206] As shown in the examples in Figures 29 and 30, when both the subject within the specified imaging range and the target category subject S are not within the depth of field due to the structure of the imaging device 10, the main exposure imaging is performed on both the subject within the specified imaging range and the target category subject S using the focus bracketing method. This makes it possible to obtain images with higher contrast as both the subject within the specified imaging range and the target category subject images, even when both the subject within the specified imaging range and the target category subject S are not within the depth of field due to the structure of the imaging device 10, compared to when only one frame of imaging is performed under the condition that both the subject within the specified imaging range and the target category subject S are not within the depth of field.

[0207] In the example shown in Figure 27, the position of the specified imaging range is exemplified as the first imaging condition, and the position of the target category subject S is exemplified as the second imaging condition, but the technology of this disclosure is not limited thereto. For example, the first imaging condition may be the brightness of a reference subject (for example, a subject within the specified imaging range), and the second imaging condition may be the brightness of the target category subject S. In this case, the control unit 48A5 executes the exposure bracketing imaging process as the default processing described above when the difference between the brightness of the reference subject and the brightness of the target category subject S (hereinafter also referred to as "brightness difference") is greater than or equal to a predetermined brightness difference. The exposure bracketing imaging process is a process that causes the imaging device 10 to image the reference subject and the target category subject S using the exposure bracketing method.

[0208] Furthermore, if the brightness difference is less than the predetermined brightness difference, the control unit 48A5 instructs the imaging device 10 to image the reference subject and the target category subject S with an exposure determined based on the reference subject.

[0209] Exposure bracketing imaging processing is implemented by the CPU 48A executing the imaging support processing shown in Figure 31, as an example. The flowchart shown in Figure 31 differs from the flowchart shown in Figure 26 in that it includes steps ST500 to ST508 instead of steps ST450 to ST456.

[0210] In step ST500 of the imaging support process shown in Figure 31, the acquisition unit 48A1 acquires photometric values ​​for subjects within the specified imaging range and for subjects S of the target category. The photometric values ​​may be calculated based on live view image data or detected by a photometric sensor (not shown).

[0211] In the next step, ST502, the control unit 48A5 calculates the brightness difference using the two photometric values ​​acquired in step ST500 for the subject within the specified imaging range and the subject S of the target category. The brightness difference is, for example, the absolute value of the difference between the two photometric values.

[0212] In the next step, ST504, the control unit 48A5 determines whether the brightness difference calculated in step ST502 is greater than or equal to a predetermined brightness difference. The predetermined brightness difference may be a fixed value or a variable value that is changed according to a given instruction and / or given conditions. In step ST504, if the brightness difference is less than the predetermined brightness difference, the determination is denied and the imaging support process proceeds to step ST508. In step ST504, if the brightness difference is greater than or equal to the predetermined brightness difference, the determination is affirmed and the imaging support process proceeds to step ST506.

[0213] In step ST506, the control unit 48A5 controls the imaging device 10 to perform exposure bracketing imaging for each subject within the specified imaging range and each of the target category subjects S. After the processing in step ST506 is completed, the imaging support process moves to step ST122. The exposure image data for each frame obtained by performing exposure bracketing imaging may be stored individually in a predetermined storage area, or they may be stored in a predetermined storage area as a composite image data of one frame obtained by combining them.

[0214] In step ST508, the control unit 48A5 controls the imaging device 10 to perform the main exposure imaging on the subject within the specified imaging range and the subject in the target category S with an exposure determined based on the subject within the specified imaging range. After the processing in step ST508 is completed, the control unit 48A5 performs the processing corresponding to steps ST120 to ST132 (see Figure 25) and then proceeds to step ST100.

[0215] As shown in the example in Figure 31, when the brightness difference is greater than or equal to a predetermined brightness difference, the exposure bracketing method is used to perform the main exposure imaging on the subject within the specified imaging range and the target category subject S. This makes it possible to obtain images with less brightness unevenness as the subject image within the specified imaging range and the target category subject image S1, compared to the case where only one frame is captured when there is brightness unevenness between the subject within the specified imaging range and the target category subject S.

[0216] Furthermore, as shown in the example in Figure 31, when the brightness difference is less than the predetermined brightness difference, the main exposure imaging is performed on the subject within the specified imaging range and the target category subject S with an exposure determined based on the reference subject. This reduces the effort required to eliminate brightness unevenness between the subject within the specified imaging range and the target category subject S before performing the main exposure imaging on the subject within the specified imaging range and the target category subject S.

[0217] In the above embodiment, examples of major categories included in the subject-specific category group 98 are the face category, posture category, eye category, and specified imaging range category, but the technology of this disclosure is not limited thereto. For example, as shown in Figure 32, the subject-specific category group 98 may include, as major categories, a period category which is a category defined by a subject characteristic in the unit of "period," and / or a position category which is a category defined by a subject characteristic in the unit of "position." Here, "period" refers to the period during which the subject was imaged, and "position" refers to the position in which the subject was imaged.

[0218] Each period category contains multiple date categories as subcategories, each with different dates. Each of these date categories has a corresponding classification count. Similarly, each period category also has a corresponding classification count. The classification count for a period category is the sum of the classification counts for all its date categories.

[0219] Each location category contains multiple sub-location categories, each with a different location. Each of these sub-location categories has a corresponding classification count. The location category itself also has a corresponding classification count. The classification count of a location category is the sum of the classification counts of its sub-location categories.

[0220] In the example shown in Figure 32, map data 104 is stored in storage 48B. Map data 104 is data that associates location coordinates, such as latitude, longitude, and height, with addresses on a map. Map data 104 is referenced by the classification unit 48A4. Note that sub-location categories are not limited to addresses on a map, but may also be location coordinates.

[0221] In the example shown in Figure 32, the classification unit 48A4 is connected to the RTC 106 and a GPS receiver 108, which is an example of a GNSS receiver. The RTC 106 acquires the current time. The RTC 106 receives power for operation from a power supply system that is disconnected from the power supply system for the controller 48, and continues to keep track of the current time (year, month, day, hour, minute, second) even when the controller 48 is shut down.

[0222] The GPS receiver 108 receives radio waves from multiple GPS satellites (not shown), which are an example of multiple GNSS satellites, and calculates position coordinates that can determine the current position of the imaging device 10 based on the reception results.

[0223] The classification unit 48A4 obtains the current time from the RTC 106 each time a full exposure image is taken for one frame, and uses the obtained current time as the imaging time to classify it into the corresponding year, month, and day category among the multiple year, month, and day categories included in the period category. Each time the classification unit 48A4 classifies the imaging time into a year, month, and day category, it adds "1" to the classification count of the year, month, and day category to which the imaging time was classified. In this case, the classification unit 48A4 obtains the current time from the RTC 106, but it may also obtain the current time via a communication network such as the Internet.

[0224] Each time a full exposure image for one frame is captured, the classification unit 48A4 obtains position coordinates from the GPS receiver 108 as the location where the image was captured (hereinafter also referred to as the "image location"). The classification unit 48A4 identifies the address corresponding to the acquired image location from the map data 104. Then, the classification unit 48A4 classifies the image location into a sub-location category corresponding to the identified address. Each time the classification unit 48A4 classifies the image location into a sub-location category, it adds "1" to the classification count of the sub-location category to which the image location was classified.

[0225] In the example shown in Figure 33, a bubble chart related to the period category (hereinafter also referred to as the "period category bubble chart") is displayed as a bubble chart 100A within the imaging support screen 100. Similar to the face category bubble chart (see Figure 17), the period category bubble chart plots bubbles indicating the number of classifications on two axes: an axis representing multiple recognized individuals and an axis representing the year, month, and day category.

[0226] Furthermore, when a location category is selected from the category selection screen 100B of the reception device 80 (for example, the touch panel 28), a bubble chart related to the location category (hereinafter referred to as the location category bubble chart) is displayed on the imaging support screen 100 as a bubble chart 100A. Similar to the bubble chart related to the face category (see Figure 17), the location category bubble chart plots bubbles indicating the number of classifications on two axes: an axis indicating multiple recognized individuals and an axis indicating sub-location categories.

[0227] In the example shown in Figure 34, a histogram related to the period category (hereinafter also referred to as the "period category histogram") is displayed as histogram 100C within the imaging support screen 100. The horizontal axis of the period category histogram represents the year, month, and day category, and the vertical axis represents the number of classifications. In addition, a histogram related to the location category (hereinafter also referred to as the "location category histogram") is also displayed as histogram 100C within the imaging support screen 100, and can be switched with the histograms of other categories (not shown). The horizontal axis of the location category histogram represents the sub-location category, and the vertical axis represents the number of classifications.

[0228] Thus, the subject-specific category group 98 includes, as a major category, a period category defined by subject characteristics in the unit of "period". Furthermore, each period category includes multiple date categories with different dates. Each time a full exposure image for one frame is taken, the imaging time is classified into a date category, and a period category bubble chart and a period category histogram corresponding to the number of classifications are displayed on the display 26 along with the live view image. The period category bubble chart and period category histogram are used in the same manner as the face category bubble chart and face category histogram described in the above embodiment. Therefore, with this configuration, imaging by the imaging device 10 can be supported according to the number of classifications counted by classifying the imaging time into a date category. Note that here, a date category separated by year, month, and day is given as an example, but this is merely an example, and categories may be divided by year, month, day, hour, minute, or second.

[0229] Furthermore, the subject-specific category group 98 includes a major category, which is a position category defined by a subject characteristic called "position." Each position category also contains multiple sub-position categories that differ in their positions. Each time a full exposure image is taken for one frame, the imaging position is classified into a sub-position category, and a position category bubble chart and a position category histogram corresponding to the number of classifications are displayed on the display 26 along with the live view image. The position category bubble chart and position category histogram are used in the same manner as the face category bubble chart and face category histogram described in the above embodiment. Therefore, with this configuration, imaging by the imaging device 10 can be supported according to the number of classifications counted by classifying the imaging position into a sub-position category.

[0230] In the above embodiment, an example was given in which the imaging support process is assumed to be executed continuously while the imaging mode is set. However, the technology of this disclosure is not limited thereto, and the imaging support process may be executed intermittently according to time and / or location. For example, as shown in Figure 35, the imaging support process may be executed only for a specified time (e.g., 10 minutes) at time checkpoints divided into predetermined time intervals (e.g., 1 hour). The predetermined time intervals that define the time checkpoints may be fixed or may be changed according to given instructions and / or given conditions (e.g., imaging conditions). Also, as an example, as shown in Figure 35, the imaging support process may be executed only for a specified time at multiple location checkpoints divided by location. Location checkpoints can be identified, for example, by using map data 104 and GPS receiver 108.

[0231] Furthermore, although the above embodiment described an example in which subject features are classified into categories by the classification unit 48A4 regardless of the imaging scene captured by the imaging device 10, the technology of this disclosure is not limited thereto. For example, the classification unit 48A4 may classify subject features into categories when the scene to be captured by the imaging device 10 matches a specific scene (for example, a sports day scene, a beach scene, and a concert scene). The specific scene may be a scene that has been captured in the past.

[0232] In this case, the imaging support process shown in Figure 36 is executed by the CPU 48A as an example. The flowchart shown in Figure 36 differs from the flowchart shown in Figure 23 in that it has a step ST550 between steps ST118 and ST120.

[0233] In step ST550 of the imaging support process shown in Figure 36, the subject recognition unit 48A2 identifies the current imaging scene by recognizing a subject within the imaging area based on the latest main exposure image data obtained when the main exposure imaging is performed in step ST118. The subject recognition unit 48A2 also identifies past imaging scenes based on past main exposure image data (for example, main exposure image data obtained within a period specified by the user). The subject recognition unit 48A2 then determines whether the current imaging scene and the past imaging scene match. In step ST550, if the current imaging scene and the past imaging scene do not match, the determination is denied, and the imaging support process proceeds to step ST130. In step ST550, if the current imaging scene and the past imaging scene match, the determination is affirmed, and the imaging support process proceeds to step ST120. As a result, in step ST126, the classification unit 48A4 classifies the subject features into categories for each subject.

[0234] In this way, the classification unit 48A4 categorizes subject features for each subject only when the current imaging scene matches a specific scene. Therefore, subject features identified from the exposure image data obtained when the exposure imaging is performed on a scene unintended by the user are not categorized.

[0235] Furthermore, the classification unit 48A4 categorizes subject features for each subject only when the current imaging scene matches a past imaging scene. Therefore, subject features identified from the main exposure image data obtained by performing main exposure imaging on the current imaging scene that matches a past imaging scene can be categorized.

[0236] Furthermore, although the above embodiment describes an example in which subject features are classified into multiple categories, the technology of this disclosure is not limited thereto. For example, as shown in Figure 37, the exposed image obtained by performing the exposure imaging may be classified into each of the multiple categories by the classification unit 48A4. In this case, each time the exposed image is classified into a category, "1" is added to the classification count of the category to which the exposed image was classified. The subject-specific category group 98 constructed in this way is used in the same manner as the subject-specific category group 98 described in the above embodiment. Therefore, with this configuration, imaging by the imaging device 10 can be supported according to the classification count in which the exposed image has been classified into a category.

[0237] Furthermore, while the above embodiment illustrates a face category histogram for person A (see Figure 18), the technology of this disclosure is not limited thereto. For example, as shown in Figure 38, a four-quadrant face category bubble chart may be used instead of a face category histogram. In the four-quadrant face category bubble chart, the smile category is assigned to the first quadrant, the crying face category to the second quadrant, the angry face category to the third quadrant, and the neutral face category to the fourth quadrant. The number of classifications corresponding to each category is represented by the size of the bubble. Note that when displaying categories in four quadrants as shown in Figure 38, facial expressions may be classified more finely, and a scatter plot may be displayed instead of a bubble chart. In this case, subtle facial expressions can be represented by adjusting the plotting position of the points based on the classified facial expressions. For example, even within the same smile category, images showing smiles that are closer to crying faces are plotted on the left side of the first quadrant. Also, images showing smiles that are closer to neutral faces are plotted on the lower side of the first quadrant. Such scatter plots allow users to understand in more detail which parts of a person's face were captured in the image.

[0238] Furthermore, although the above embodiment illustrates a bubble chart 100A and a histogram 100C, the technology of this disclosure is not limited thereto, and other graphs may be used, or the numerical values ​​indicating the number of classifications may be displayed in a form that is divided for each subject and for each category.

[0239] Furthermore, although the above embodiment describes an example in which imaging is supported for categories corresponding to the number of classifications selected by the user from the histogram 100C, the technology of this disclosure is not limited thereto, and the categories for which imaging is supported may be directly selected by the user from the histogram 100C, etc., via a reception device 80 (e.g., touch panel 28).

[0240] Furthermore, although the above embodiment described an example in which the exposure image data is stored in the image memory 50, the data including the exposure image data obtained by performing the above-described support processing may be used as training data for machine learning of the trained model 92. This makes it possible to create a trained model 92 based on the exposure image data obtained by performing the support processing.

[0241] Furthermore, although the above embodiment describes an example in which imaging support processing is performed by a controller 48 within the imaging device 10, the technology of this disclosure is not limited thereto. For example, as shown in Figure 40, imaging support processing may be performed by a computer 114 in an external device 112 that is communicably connected to the imaging device 10 via a network 110 such as a LAN or WAN. In the example shown in Figure 40, the computer 114 includes a CPU 116, storage 118, and memory 120. A category database 96 is constructed in the storage 118 and stores the imaging support processing program 84.

[0242] The imaging device 10 requests the external device 112 to perform imaging support processing via the network 110. In response, the CPU 116 of the external device 112 reads the imaging support processing program 84 from the storage 118 and executes the imaging support processing program 84 on the memory 120. The CPU 116 performs imaging support processing according to the imaging support processing program 84 executed on the memory. The CPU 116 then provides the processing results obtained from the execution of the imaging support processing to the imaging device 10 via the network 110.

[0243] Furthermore, the imaging device 10 and the external device 112 may perform the imaging support processing in a distributed manner, or multiple devices including the imaging device 10 and the external device 112 may perform the imaging support processing in a distributed manner. When implementing distributed processing, for example, the CPU 48A of the imaging device 10 may be operated as the acquisition unit 48A1 and the control unit 48A5, and the CPU of a device other than the imaging device 10 (for example, the external device 112) may be operated as the subject recognition unit 48A2, the feature extraction unit 48A3, and the classification unit 48A4. In other words, the processing load on the imaging device 10 may be reduced by having an external device with higher computing power than the imaging device 10 handle processing that has a relatively large processing load.

[0244] Furthermore, although a still image was used as an example of the exposure image in the above embodiment, the technology of this disclosure is not limited thereto, and a moving image may also be used as the exposure image. The moving image may be a recording moving image, or a display moving image, i.e., a live view image or a post-view image.

[0245] Furthermore, although the above embodiment displays the recommended imaging information within the live view image, it is not necessary to display it within the live view image. For example, when capturing a moving image with a full exposure, a display indicating the target category subject (at least one of an arrow, a face frame, and a message) may be displayed in the same way as the recommended imaging information. By displaying the target category subject while capturing a moving image with a full exposure in this way, the user can use the imaging device 10 to recognize that the moving image contains a target category subject. In addition, the user can use the imaging device 10 to cut off frames containing the target category subject after capturing a moving image with a full exposure. By doing so, still images containing subjects of the target category can be obtained. In this case, the imaging device 10 may classify subjects of the target category in the same way as above each time a subject of the target category is identified by full exposure imaging of a moving image, and update and display the histogram and / or bubble chart. In this way, the user can understand what subjects are included simply by using the imaging device 10 to perform imaging for moving images and obtain moving images. In this case, the values ​​based on still images and values ​​based on moving images may be displayed in different ways in the histogram and / or bubble chart. For example, in the histogram, the histogram based on still images and the histogram based on moving images may be displayed in different colors using a stacked bar graph. In this way, the user can understand whether each classification count is based on still images or moving images. Alternatively, the histogram and / or bubble chart may be created based only on the classification of subjects indicated by the subject areas contained in a single moving image. In this way, the user can understand what categories the subject areas contained in that single moving image belong to. Such histograms and / or bubble charts may be displayed on the display 26 based on user operations, such as after capturing still or moving images, or during playback mode when a live view image is not displayed.

[0246] Furthermore, although the above embodiments describe an example in which the number of classifications continues to increase, the technology of this disclosure is not limited to this. For example, the number of classifications corresponding to at least one category in the subject category group 98 may be reset periodically or at specified intervals. For example, it may be reset according to time and / or location. Specifically, it may be reset once a day, once an hour, or every time the location changes by 100 meters.

[0247] Furthermore, although the smile category was used as an example of a target category in the above embodiment, the technology of this disclosure is not limited to this, and other categories may be used as target categories, or multiple categories may be used as target categories. In this case, for example, in the face category histogram shown in Figure 18, the number of classifications for multiple categories (e.g., the smile category and the crying face category), or multiple categories may be selected by the user.

[0248] Furthermore, although Person A was used as an example of a target category subject in the above embodiment, the technology of this disclosure is not limited thereto, and there may be multiple target category subjects. In this case, for example, in the face category bubble chart shown in Figure 17, multiple people (e.g., Person A, Person B, and Person C) can be selected by the user. As a result, for example, if the target category is the smile category, the various support processes described above will be performed in scenes where at least one of the multiple people selected by the user is smiling.

[0249] Furthermore, while the above embodiment exemplified the classification count, which is simply the number of times the subject features were classified into a category, the technology of this disclosure is not limited thereto. For example, it may be the classification count per unit time.

[0250] Furthermore, although the above embodiment illustrates a detection frame 102 (see Figures 19 and 21), the technology of this disclosure is not limited thereto, and information about the subject surrounded by the detection frame 102 (e.g., name and / or category name, etc.) may also be displayed along with the detection frame 102.

[0251] Furthermore, although the above embodiment exemplifies a physical camera (hereinafter also referred to as "physical camera") as the imaging device 10, the technology of this disclosure is not limited thereto. Instead of a physical camera, a virtual camera may be applied that generates virtual viewpoint image data by virtually imaging a subject from a virtual viewpoint based on captured image data obtained by capturing images from multiple physical cameras set at different positions. In this case, the image shown by the virtual viewpoint image data, i.e., the virtual viewpoint image, is an example of the "captured image" related to the technology of this disclosure.

[0252] In the above embodiment, an example of a configuration in which a non-phase difference pixel partitioning region 30N and a phase difference pixel partitioning region 30P are used in combination was described, but the technology of this disclosure is not limited thereto. For example, instead of the non-phase difference pixel partitioning region 30N and the phase difference pixel partitioning region 30P, an area sensor may be used in which phase difference image data and non-phase difference image data are selectively generated and read out. In this case, the area sensor has a plurality of photosensitive pixels arranged in two dimensions. For example, a pair of independent photodiodes without light-shielding members are used for the photosensitive pixels included in the area sensor. When non-phase difference image data is generated and read out, photoelectric conversion is performed by the entire area of ​​the photosensitive pixels (the pair of photodiodes), and when phase difference image data is generated and read out (for example, when performing a passive distance measurement), photoelectric conversion is performed by one of the photodiodes of the pair. Here, one of the photodiodes of the pair is a photodiode corresponding to the first phase difference pixel L described in the above embodiment, and the other of the photodiodes of the pair is a photodiode corresponding to the second phase difference pixel R described in the above embodiment. Furthermore, while it is possible to selectively generate and read out phase-difference image data and non-phase-difference image data using all the photosensitive pixels included in the area sensor, the system is not limited to this configuration; it is also possible to selectively generate and read out phase-difference image data and non-phase-difference image data using only some of the photosensitive pixels included in the area sensor.

[0253] In the above embodiment, image plane phase difference pixels are exemplified as phase difference pixels P, but the technology of this disclosure is not limited thereto. For example, non-phase difference pixels N may be arranged in place of the phase difference pixels P included in the photoelectric conversion element 30, and a phase difference AF plate containing a plurality of phase difference pixels P may be provided separately from the photoelectric conversion element 30 in the imaging device body 12.

[0254] In the above embodiment, an AF method utilizing distance measurement results based on phase-difference image data, i.e., a phase-difference AF method, was exemplified, but the technology of this disclosure is not limited thereto. For example, a contrast AF method may be used instead of a phase-difference AF method. Alternatively, an AF method based on distance measurement results using the parallax of a pair of images obtained from a stereo camera, or an AF method utilizing distance measurement results of a TOF method using laser light or the like, may be used.

[0255] In the above embodiment, a focal-plane shutter was given as an example of a mechanical shutter 72, but the technology of this disclosure is not limited to this, and the technology of this disclosure can be applied even if other types of mechanical shutters, such as lens shutters, are used instead of a focal-plane shutter.

[0256] In the above embodiment, an example was described in which the imaging support processing program 84 is stored in storage 48B, but the technology of this disclosure is not limited thereto. For example, as shown in Figure 41, the imaging support processing program 84 may be stored in a storage medium 200. The storage medium 200 is a non-temporary storage medium. An example of the storage medium 200 is any portable storage medium such as an SSD or a USB memory.

[0257] The imaging support processing program 84 stored in the storage medium 200 is installed on the controller 48. The CPU 48A executes imaging support processing according to the imaging support processing program 84.

[0258] Alternatively, the imaging support processing program 84 may be stored in the memory of another computer or server device connected to the controller 48 via a communication network (not shown), so that the imaging support processing program 84 is downloaded and installed on the controller 48 in response to a request from the imaging device 10.

[0259] Furthermore, it is not necessary to store the entire imaging support processing program 84 in the memory unit of another computer or server device connected to the controller 48, or in the storage 48B; it is acceptable to store only a portion of the imaging support processing program 84.

[0260] In the example shown in Figure 41, the controller 48 is built into the imaging device 10. However, the technology of this disclosure is not limited to this, and for example, the controller 48 may be provided outside the imaging device 10.

[0261] In the example shown in Figure 41, CPU48A is a single CPU, but it could be multiple CPUs. Alternatively, a GPU could be used instead of CPU48A.

[0262] In the example shown in Figure 41, a controller 48 is illustrated, but the technology of this disclosure is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs may be used instead of the controller 48. Alternatively, a combination of hardware and software configurations may be used instead of the controller 48.

[0263] The hardware resources used to perform the imaging support processing described in the above embodiment include the following types of processors. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for performing imaging support processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform particular processing, such as FPGAs, PLDs, or ASICs. Each processor has built-in or connected memory, and each processor uses memory to perform imaging support processing.

[0264] The hardware resources that perform the imaging support processing may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform the imaging support processing may consist of a single processor.

[0265] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs imaging support processing. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform imaging support processing, on a single IC chip, as exemplified by SoCs. In this way, imaging support processing is realized using one or more of the above types of processors as hardware resources.

[0266] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits combining circuit elements such as semiconductor devices. Also, the above imaging support processing is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0267] The description content and the illustrated content shown above are detailed descriptions of the part related to the technology of the present disclosure, and are only examples of the technology of the present disclosure. For example, the descriptions regarding the above-described configuration, function, operation, and effect are descriptions of an example of the configuration, function, operation, and effect of the part related to the technology of the present disclosure. Therefore, it goes without saying that within the scope not departing from the gist of the technology of the present disclosure, the description content and the illustrated content shown above may be deleted of unnecessary parts, new elements may be added, or replacements may be made. Also, in order to avoid complication and facilitate the understanding of the part related to the technology of the present disclosure, in the description content and the illustrated content shown above, descriptions regarding common technical knowledge and the like that do not particularly require explanation for implementing the technology of the present disclosure are omitted.

[0268] In this specification, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0269] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually described as being incorporated by reference.

[0270] Regarding the above embodiments, the following supplementary notes are further disclosed.

[0271] (Supplementary Note 1) A processor, a memory connected to or built in the above processor, and the above processor acquires frequency information indicating the frequency of the above features classified into categories based on the features of the subject specified from the captured image obtained by the imaging device, and performs a support process for assisting the imaging by the above imaging device based on the above frequency information Imaging support device.

[0272] (Note 2) The above categories are further classified into multiple categories, each containing at least one target category. The above target categories are categories determined based on the above frequency information. The imaging support device described in Appendix 1, wherein the above support processing includes processing to support imaging of a subject belonging to the above target category and having the above characteristics.

[0273] (Note 3) The above support processing is an imaging support device as described in Appendix 2, which includes a display process that displays a recommendation to image subjects of the above target category.

[0274] (Note 4) The above display process is the process of displaying a display image on the above display and displaying a frame within the display image that surrounds at least a portion of the subject image of the target category, as described in Appendix 3 of the imaging support device.

[0275] (Note 5) The above processor is Based on the imaging results from the above imaging device, the above-mentioned target category subject is detected. An imaging support device according to any one of the appendices 2 to 4, which, on the condition that it has detected a subject in the above-mentioned target category, acquires an image containing an image corresponding to the subject in the above-mentioned target category.

[0276] (Note 6) The imaging support device described in Appendix 5, wherein the processor detects the subject of the target category and, on the condition that the subject of the target category is included in the predetermined imaging range, causes the imaging device to perform the imaging accompanied by the exposure.

[0277] (Note 7) The above processor controls the imaging device when the subject of the target category is located outside the designated imaging range determined according to instructions given from an external source, thereby bringing the designated imaging range and the subject of the target category into the depth of field, as described in any one of Appendix 2 to Appendix 6.

[0278] (Note 8) The above processor is an imaging support device as described in Appendix 7, which causes the imaging device to image the specified imaging range and the target category subject in a focus bracketing manner when both the specified imaging range and the target category subject do not fall within the depth of field due to the structure of the imaging device.

[0279] (Note 9) The above processor is an imaging support device according to Appendix 7 or Appendix 8, which causes the imaging device to image the above-mentioned subject in focus when the subject in the target category is located within the above-specified imaging range.

[0280] (Note 10) The above processor is an imaging support device according to any one of the appendices 2 to 9, which causes the imaging device to image the reference subject and the target category subject in an exposure bracketing manner when the difference between the brightness of the reference subject and the brightness of the target category subject is greater than or equal to a predetermined difference.

[0281] (Note 11) The above processor is an imaging support device as described in Appendix 10, which causes the imaging device to image the reference subject and the target category subject with an exposure determined based on the target category subject when the difference is less than the predetermined difference.

[0282] (Note 12) The image obtained by the above-mentioned imaging supported by the above-mentioned support processing is used for learning with the imaging support device described in any one of the appendices 1 to 11.

Claims

1. Processor and The processor comprises, The aforementioned processor, Based on the characteristics of each of the multiple subjects identified from the captured images obtained by the imaging device, the characteristics are classified into categories. Based on the aforementioned categories, the imaging conditions for the multiple subjects are controlled. The aforementioned category is further classified into multiple categories, each containing at least one target category. The aforementioned target category is a category determined based on frequency information indicating the frequency of the features classified into the aforementioned category. The processor performs support processing to assist imaging by the imaging device based on the frequency information. The support process includes a process that supports imaging of a subject belonging to the target category having the characteristics of the target category, The processor performs default processing when the degree of difference between the first imaging condition given externally and the second imaging condition given for the target category subject is greater than or equal to a predetermined difference. Imaging support device.

2. The aforementioned category is a category relating to the posture of the subject. The imaging support device according to claim 1.

3. The aforementioned imaging conditions include the focus position. The imaging support device according to claim 1 or claim 2.

4. The types of categories mentioned above are a major category and multiple subcategories that are subordinate to the major category. The aforementioned subcategories include features that are subordinate to the features classified in the aforementioned major categories. The aforementioned major category and the aforementioned plurality of minor categories include at least one target category, The aforementioned target category is a category determined based on frequency information indicating the frequency of the features classified into the aforementioned category. The imaging support device according to any one of claims 1 to 3.

5. The aforementioned frequency includes the frequency of the features classified into the aforementioned category for each of the plurality of subjects identified from the captured images previously obtained by imaging the imaging device. The imaging support device according to claim 4.

6. The aforementioned support process includes a display process that displays a recommendation to take images of the subject in the target category. The imaging support device according to claim 1.

7. The display process involves displaying the display image obtained by the imaging device on a display, and displaying the target category subject image representing the target category subject within the display image in a manner that allows it to be distinguished from other image areas. The imaging support device according to claim 6.

8. The aforementioned processor, Based on the imaging results from the aforementioned imaging device, the target category subject is detected. On the condition that the aforementioned target category subject is detected, an image containing an image corresponding to the aforementioned target category subject is acquired. The imaging support device according to any one of claims 1 to 7.

9. The processor displays, in a different display mode, an object indicating a specified imaging range determined according to instructions given from an external source, and an object indicating the target category subject. The imaging support device according to any one of claims 1 to 8.

10. The aforementioned target category is the low-frequency category, which is the category with the lowest frequency among the multiple categories. The imaging support device according to any one of claims 1 to 9.

11. When the subject of the aforementioned target category is imaged by the imaging device, the target category is a category determined according to the state of the subject of the aforementioned target category, and is a category into which the characteristics of the subject of the aforementioned target category are classified. The imaging support device according to any one of claims 1 to 10.

12. When multiple objects are imaged by the imaging device, the target category is an object target category that can identify each of the multiple objects themselves. The imaging support device according to any one of claims 1 to 11.

13. The aforementioned categories are created for at least one unit. The imaging support device according to any one of claims 1 to 12.

14. One of the aforementioned units is a period of time. The imaging support device according to claim 13.

15. One of the aforementioned units is position. The imaging support device according to claim 14.

16. The processor causes the classifier to classify the features, The classifier classifies the features when the scene captured by the imaging device matches a specific scene. The imaging support device according to any one of claims 1 to 15.

17. The aforementioned specific scene is a scene that was captured in the past. The imaging support device according to claim 16.

18. The aforementioned support process is a process that includes a process for displaying the frequency information. The imaging support device according to any one of claims 1 to 12.

19. The support process includes a process that supports imaging related to the category corresponding to the specified frequency information when the frequency information is displayed and the frequency information is specified by the receiving device. The imaging support device according to claim 18.

20. An imaging support device according to any one of claims 1 to 19, Equipped with an image sensor, The processor supports the imaging using the image sensor by controlling the imaging conditions. Imaging device.

21. The process involves classifying the characteristics of multiple subjects identified from images captured by an imaging device into categories, and This includes controlling imaging conditions for the plurality of subjects based on the aforementioned categories, The aforementioned category is further classified into multiple categories, each containing at least one target category. The aforementioned target category is a category determined based on frequency information indicating the frequency of the features classified into the aforementioned category. This includes performing support processing to assist imaging by the imaging device based on the frequency information, The support process includes a process that supports imaging of a subject belonging to the target category having the characteristics of the target category, The process includes performing a default action when the degree of difference between the first imaging condition given externally and the second imaging condition given for the target category subject is greater than or equal to a predetermined difference. Imaging support method.

22. A program that causes a computer to perform image processing, The aforementioned image processing is, The process involves classifying the characteristics of multiple subjects identified from images captured by an imaging device into categories, and This includes controlling imaging conditions for the plurality of subjects based on the aforementioned categories, The aforementioned category is further classified into multiple categories, each containing at least one target category. The aforementioned target category is a category determined based on frequency information indicating the frequency of the features classified into the aforementioned category. The image processing includes performing support processing to assist imaging by the imaging device based on the frequency information, The support process includes a process that supports imaging of a subject belonging to the target category having the characteristics of the target category, The image processing includes performing a default process when the degree of difference between the first imaging condition given externally and the second imaging condition given for the target category subject is greater than or equal to a predetermined difference. program.

Images