Imaging support device, imaging device, imaging support method, and program

Abstract

To provide an imaging support device that can support imaging by an imaging device according to a frequency with which the features of a subject are classified into categories.SOLUTION: An imaging support device includes a processor and a memory connected to or built into the processor, and the processor acquires frequency information indicating a frequency of features classified into categories on the basis of the features of a subject contained in an image obtained by imaging using the imaging device, and performs support processing to support imaging by the imaging device on the basis of the frequency information.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The technology disclosed herein relates to an imaging support device, an imaging device, an imaging support method, and a program. [Background technology]

[0002] Japanese Patent Application Publication No. 2007-006033 discloses an object determination device that selects a face to be processed from a plurality of faces contained in an image. The object determination device described in Japanese Patent Application Publication No. 2007-006033 includes a face detection means for detecting a face from an image, a face information recording means for recording a face previously detected by the face detection means in association with a detection history related to the detection, and a face selection means for selecting a face to be processed from the faces contained in the image based on the detection history.

[0003] Japanese Patent Application Laid-Open No. 2009-252069 discloses an image processing device including a face recognition dictionary, an image acquisition means, a face area detection means, a feature extraction means, a discrimination means, a face recognition dictionary correction means, and a classification means. In the face recognition dictionary, facial features for discriminating whether or not the faces are the same person are registered for each person. The image acquisition means acquires an image including the person. The face area detection means detects a face area from the image acquired by the image acquisition means. The feature extraction means extracts facial features of the face area based on the face area detected by the face area detection means. The discrimination means discriminates whether or not the facial features of the same person are registered in the face recognition dictionary based on the facial features extracted by the feature extraction means and the facial features registered in the face recognition dictionary. The face recognition dictionary correction means corrects the registered facial features based on the extracted facial features when the discrimination means discriminates that the facial features of the same person are registered in the face recognition dictionary, and registers the extracted facial features as facial features of a new person when the discrimination means discriminates that the facial features of the same person are not registered in the face recognition dictionary. The classification means classifies the face of the person in the image acquired by the image acquisition means as a known face when the discrimination means discriminates that the facial features of the same person are registered in the face recognition dictionary, and classifies the face of the person in the image acquired by the image acquisition means as an unknown face when the discrimination means discriminates that the facial features of the same person are not registered in the face recognition dictionary.

[0004] Japanese Patent Application Laid-Open No. 2012-099943 discloses an image processing device that includes a storage means for storing face image data, a face detection means for detecting a face from a video signal, a face recognition means for determining whether or not a face detected by the face detection means is included in the face image data stored in the storage means, and an image processing means. When a face determined by the face recognition means to be included in the face image data stored in the storage means and a face determined not to be included are detected from the video signal, the image processing means performs image processing to improve the image quality of the face determined to be included in the face image data and the face determined not to be included in the face image data compared to other areas.

[0005] Japanese Patent Laid-Open Publication No. 2009-003012 discloses an imaging device including an imaging lens capable of changing a lens focal position by driving at least some of a plurality of lenses arranged along an optical axis direction along the optical axis direction, an image acquisition means, a subject detection means, a focus evaluation value calculation means, a selection means, and a recording means. The image acquisition means performs continuous shooting while changing the lens focal position of the imaging lens, and acquires a plurality of image data. The subject detection means detects a main subject region according to a movement state of the subject among the plurality of image data acquired by the image acquisition means. The focus evaluation value calculation means calculates a focus evaluation value of the main subject region acquired by the subject detection means for each of the plurality of image data. The selection means selects at least one of the plurality of image data based on the focus evaluation value acquired by the focus evaluation value calculation means. The recording means records the image data selected by the selection means on a recording medium. Summary of the Invention

[0006] One embodiment of the technique disclosed herein provides an imaging support device, an imaging device, an imaging support method, and a program that can support imaging by an imaging device according to the frequency with which features of a subject are classified into categories. [Means for solving the problem]

[0007] A first aspect of the technology of the present disclosure is an imaging support device that includes a processor and a memory connected to or built into the processor, where the processor acquires frequency information indicating the frequency of features classified into categories based on features of a subject identified from an image obtained by capturing an image using an imaging device, and performs support processing to support imaging by the imaging device based on the frequency information.

[0008] A second aspect of the technology disclosed herein is an imaging support device related to the first aspect, in which the categories are classified into a plurality of categories including at least one target category, the target categories are categories determined based on frequency information, and the support processing is processing including processing for supporting imaging of target category subjects having characteristics belonging to the target category.

[0009] A third aspect according to the technology of the present disclosure is the imaging support device according to the second aspect, in which the support process includes a display process for displaying a recommendation to capture an image of a target category subject.

[0010] A fourth aspect of the technology disclosed herein is an imaging support device relating to the third aspect, in which the display processing is a process of displaying on a display a display image obtained by imaging using an imaging device, and displaying a target category subject image showing a target category subject within the display image in a manner distinguishable from other image areas.

[0011] A fifth aspect of the technology disclosed herein is an imaging support device relating to any one of the second to fourth aspects, in which a processor detects a target category subject based on an imaging result by an imaging device, and, on condition that a target category subject has been detected, acquires an image including an image corresponding to the target category subject.

[0012] A sixth aspect of the technology disclosed herein is an imaging support device relating to any one of the second to fifth aspects, in which a processor displays an object indicating a designated imaging range determined in accordance with instructions given from outside and an object indicating a target category subject in different display modes.

[0013] A seventh aspect of the technology disclosed herein is an imaging support device relating to any one of the second to sixth aspects, in which a processor performs a default processing when the degree of difference between a first imaging condition given from outside and a second imaging condition given for a target category subject is equal to or greater than a default degree of difference.

[0014] An eighth aspect of the technology disclosed herein is an imaging support device according to any one of the second to seventh aspects, in which the target category is a low-frequency category having a relatively low frequency among multiple categories.

[0015] A ninth aspect of the technology disclosed herein is an imaging support device relating to any one of the second to eighth aspects, in which, when a target category subject is imaged by an imaging device, the target category is a category determined according to the state of the target category subject and is a category into which characteristics of the target category subject are classified.

[0016] A tenth aspect of the technology disclosed herein is an imaging support device relating to any one of the second to ninth aspects, in which, when multiple objects are imaged by the imaging device, the target category is an object target category that can identify each of the multiple objects themselves.

[0017] An eleventh aspect of the technique of the present disclosure is an imaging support device according to any one of the first to tenth aspects, in which the categories are created for at least one unit.

[0018] A twelfth aspect of the technology of the present disclosure is an imaging support device according to the eleventh aspect, in which one of the units is a period.

[0019] A thirteenth aspect of the technique of the present disclosure is the imaging support device according to the eleventh or twelfth aspect, in which one of the units is a position.

[0020] A 14th aspect of the technology disclosed herein is an imaging support device relating to any one of the first to thirteenth aspects, in which a processor classifies features using a classifier, and the classifier classifies the features when a scene to be imaged by the imaging device matches a specific scene.

[0021] A fifteenth aspect related to the technique of the present disclosure is an imaging support device related to the fourteenth aspect, in which the specific scene is a scene that was captured in the past.

[0022] A sixteenth aspect of the technique of the present disclosure is an imaging support device according to any one of the first to fifteenth aspects, in which the support process includes a process of displaying frequency information.

[0023] A 17th aspect of the technology disclosed herein is an imaging support device relating to the 16th aspect, in which the support processing includes processing for supporting imaging related to a category corresponding to the specified frequency information when frequency information is specified by a receiving device while the frequency information is displayed.

[0024] An 18th aspect of the technology of the present disclosure is an imaging support device that includes a processor and a memory connected to or built into the processor, where the processor acquires frequency information indicating the frequency of captured images classified into categories based on characteristics of subjects contained in captured images obtained by imaging using an imaging device, and performs support processing to support imaging by the imaging device based on the frequency information.

[0025] A 19th aspect of the technology of the present disclosure is an imaging device comprising an imaging support device relating to any one of the first to eighteenth aspects and an image sensor, in which a processor performs support processing to support imaging using the image sensor.

[0026] A twentieth aspect of the technology of the present disclosure is an imaging support method that includes acquiring frequency information indicating the frequency of features classified into categories based on features of a subject identified from an image obtained by capturing an image using an imaging device, and performing support processing to support imaging by the imaging device based on the frequency information.

[0027] A 21st aspect of the technology of the present disclosure is an imaging support method that includes acquiring frequency information indicating a frequency of captured images classified into categories based on characteristics of a subject identified from an captured image obtained by capturing an image using an imaging device, and performing support processing to support imaging by the imaging device based on the frequency information.

[0028] A 22nd aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring frequency information indicating a frequency of features classified into categories based on features of a subject identified from an image obtained by capturing an image using an imaging device, and performing support processing for supporting imaging by the imaging device based on the frequency information.

[0029] A 23rd aspect of the technology of the present disclosure is a program for causing a computer to execute processing including acquiring frequency information indicating a frequency of captured images classified into categories based on characteristics of a subject identified from an captured image obtained by capturing an image using an imaging device, and performing support processing for supporting imaging by the imaging device based on the frequency information. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a perspective view showing an example of the appearance of an imaging device. [Diagram 2] 2 is a rear view showing an example of the appearance of the rear side of the imaging device shown in FIG. [Diagram 3] 2 is a schematic diagram showing an example of an arrangement of pixels included in a photoelectric conversion element of an imaging device. FIG. [Figure 4] 4 is a conceptual diagram showing an example of incident characteristics of subject light with respect to a first phase difference pixel and a second phase difference pixel included in the photoelectric conversion element shown in FIG. 3. [Diagram 5] 4 is a schematic configuration diagram showing an example of the configuration of a non-phase difference pixel included in the photoelectric conversion element shown in FIG. 3. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of an imaging apparatus. [Figure 7] 2 is a block diagram showing an example of a configuration of a controller included in the imaging device. FIG. [Figure 8] 2 is a block diagram showing an example of main functions of a CPU included in the imaging device. FIG. [Figure 9] 11 is a conceptual diagram showing an example of processing contents when a CPU operates as an acquisition unit and a control unit. FIG. [Figure 10]FIG. 4 is a conceptual diagram showing an example of processing contents when a CPU operates as a control unit. [Figure 11] 11 is a conceptual diagram showing an example of processing contents when a CPU operates as a subject recognition unit and a control unit. FIG. [Figure 12] FIG. 13 is a conceptual diagram showing an example of processing contents when a CPU operates as a feature extraction unit. [Figure 13] FIG. 13 is a conceptual diagram showing an example of processing contents when a CPU operates as a feature extraction unit and a classification unit. [Figure 14] 10 is a conceptual diagram showing an example of how subject features are classified into a plurality of categories by a classification unit. FIG. [Figure 15] 11 is a conceptual diagram showing an example of processing contents in which a subject recognition unit recognizes a subject based on live view image data. FIG. [Figure 16] 11 is a conceptual diagram showing an example of processing contents when an imaging support screen is displayed within a live view image. FIG. [Figure 17] 11 is a screen diagram showing an example of an aspect in which a bubble chart or the like is displayed on the imaging support screen. FIG. [Figure 18] 11 is a screen diagram showing an example of an aspect in which a histogram or the like is displayed on the imaging support screen. FIG. [Figure 19] 11A and 11B are screen diagrams showing an example of a display mode in a live view image when imaging support processing is executed. [Figure 20A] 13 is a flowchart showing an example of the flow of an imaging support process. [Figure 20B] This is a continuation of the flowchart shown in FIG. 20A. [Figure 21] 11A and 11B are screen diagrams showing an example of display contents in a live view image in which an object indicating a designated imaging area and a target category subject image are displayed in different display modes. [Figure 22] 13 is a flowchart showing a first modified example of the flow of the imaging support process. [Diagram 23] 13 is a flowchart showing a second modified example of the flow of the imaging support process. [Figure 24] 13 is a flowchart showing a third modified example of the flow of imaging support processing. [Diagram 25] 13 is a flowchart showing a fourth modified example of the flow of imaging support processing. [Figure 26] 13 is a flowchart showing a fifth modified example of the flow of the imaging support process. [Figure 27] 10 is a conceptual diagram showing an example of calculation of a degree of difference between a first imaging condition and a second imaging condition. FIG. [Figure 28] FIG. 11 is a conceptual diagram showing an example of processing content performed when deepening the depth of field. [Figure 29] 11A and 11B are conceptual diagrams showing examples of calculation of in-focus positions for subjects within a designated image capture range and for subjects of a target category. [Diagram 30] FIG. 11 is a block diagram showing an example of processing contents when main exposure imaging using the focus bracketing method is performed. [Diagram 31] 13 is a flowchart showing a sixth modified example of the flow of imaging support processing. [Diagram 32] FIG. 2 is a block diagram showing an example of a configuration of subject category groups. [Diagram 33] 13 is a screen diagram showing an example of an aspect in which a period category bubble chart, etc. are displayed on the imaging support screen. [Diagram 34] 13 is a screen diagram showing an example of an aspect in which a period category histogram and the like are displayed on the imaging support screen. FIG. [Diagram 35] 11 is a time chart showing an example of a manner in which an imaging support process is executed at a preset time interval, and an example of a manner in which an imaging support process is executed at each position checkpoint. [Diagram 36] 13 is a flowchart showing a seventh modified example of the flow of the imaging support process. [Figure 37] 11 is a conceptual diagram showing an example of how a classification unit classifies a main exposure image into a plurality of categories. FIG. [Figure 38] FIG. 13 is a conceptual diagram showing an example of a four-quadrant face category bubble chart. [Figure 39] A block diagram showing an example of an embodiment in which training data including actual exposure image data obtained by actual exposure imaging subjected to assisted processing is used for machine learning of a trained model. [Diagram 40] 1 is a block diagram showing an example of a manner in which an imaging apparatus causes an external apparatus to execute imaging support processing. [Diagram 41] 11 is a block diagram showing an example of a manner in which an imaging support processing program is installed in a controller within an imaging apparatus from a storage medium on which the imaging support processing program is stored. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Hereinafter, examples of embodiments of an imaging support device, an imaging device, an imaging support method, and a program according to the technology of the present disclosure will be described with reference to the attached drawings.

[0032] First, the terms used in the following description will be explained.

[0033] CPU is an abbreviation for "Central Processing Unit". RAM is an abbreviation for "Random Access Memory". IC is an abbreviation for "Integrated Circuit". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". TOF is an abbreviation for "Time of Flight". fps is an abbreviation for "frame per second". MF is an abbreviation for "Manual Focus". AF is an abbreviation for "Auto Focus". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". RTC is an abbreviation for "Real Time Clock". GPS is an abbreviation for "Global Positioning System". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". GNSS is an abbreviation for "Global Navigation Satellite System".For ease of explanation, a CPU is exemplified below as an example of a "processor" according to the technology of the present disclosure, but the "processor" according to the technology of the present disclosure may be a combination of multiple processing devices such as a CPU and a GPU. When a combination of a CPU and a GPU is applied as an example of a "processor" according to the technology of the present disclosure, the GPU operates under the control of the CPU and is responsible for executing image processing.

[0034] In the description of this specification, "vertical" refers to vertical in the sense of including a perfectly vertical error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not go against the spirit of the technology of the present disclosure. In the description of this specification, "matching" refers to a perfectly matching error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and that does not go against the spirit of the technology of the present disclosure.

[0035] As an example, as shown in Fig. 1, the imaging device 10 is a digital camera with interchangeable lenses and no reflex mirror. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 that is replaceably attached to the imaging device body 12. Note that, although a digital camera with interchangeable lenses and no reflex mirror is given here as an example of the imaging device 10, the technology of the present disclosure is not limited thereto, and may be a digital camera with a fixed lens, a digital camera without a reflex mirror, or a digital camera built into various electronic devices such as a smart device, a wearable terminal, a cell observation device, an ophthalmic observation device, or a surgical microscope.

[0036] The imaging device body 12 is provided with an image sensor 16. The image sensor 16 is a CMOS image sensor. The image sensor 16 captures an imaging area including a group of subjects. When an interchangeable lens 14 is attached to the imaging device body 12, subject light indicative of a subject passes through the interchangeable lens 14 and is focused on the image sensor 16, and image data indicative of the image of the subject is generated by the image sensor 16.

[0037] In this embodiment, a CMOS image sensor is used as the image sensor 16, but the technology of the present disclosure is not limited to this. For example, the technology of the present disclosure can be applied even if the image sensor 16 is another type of image sensor, such as a CCD image sensor.

[0038] A release button 18 and a dial 20 are provided on the top surface of the imaging device body 12. The dial 20 is operated when setting an operation mode of the imaging system and an operation mode of the playback system, and by operating the dial 20, the imaging device 10 is selectively set to an imaging mode or a playback mode as an operation mode.

[0039] The release button 18 functions as an imaging preparation instruction section and an imaging instruction section, and is capable of detecting two-stage pressing operations, an imaging preparation instruction state and an imaging instruction state. The imaging preparation instruction state refers to a state in which the button is pressed from a standby position to an intermediate position (half-pressed position), for example, and the imaging instruction state refers to a state in which the button is pressed beyond the intermediate position to a final pressed position (fully-pressed position). In the following, the "state in which the button is pressed from the standby position to the half-pressed position" is referred to as the "half-pressed state," and the "state in which the button is pressed from the standby position to the fully-pressed position" is referred to as the "fully-pressed state." Depending on the configuration of the imaging device 10, the imaging preparation instruction state may be a state in which the user's finger is in contact with the release button 18, and the imaging instruction state may be a state in which the operating user's finger has moved from a state in which the button is in contact with the release button 18 to a state in which the finger is released.

[0040] As an example, as shown in FIG. 2, a touch panel display 22 and instruction keys 24 are provided on the rear surface of the imaging device body 12.

[0041] The touch panel display 22 includes a display 26 and a touch panel 28 (see also FIG. 3). An example of the display 26 is an organic electroluminescence (EL) display. The display 26 may be any other type of display, such as a liquid crystal display or an inorganic EL display, instead of an organic EL display.

[0042] The display 26 displays images and / or text information, etc. When the imaging device 10 is in imaging mode, the display 26 is used for capturing images for live view images, i.e., for displaying live view images obtained by performing continuous imaging. Capturing images for live view images (hereinafter also referred to as "capturing images for live view images") is performed according to a frame rate of, for example, 60 fps. 60 fps is merely one example, and the frame rate may be less than 60 fps or may be more than 60 fps.

[0043] Here, the term "live view image" refers to a moving image for display based on image data obtained by capturing an image using the image sensor 16. A live view image is also generally referred to as a through image. The live view image is an example of a "display image" according to the technology of the present disclosure.

[0044] The display 26 is also used to display a still image obtained by capturing an image for a still image when an instruction to capture an image for a still image is given to the imaging device 10 via the release button 18. Furthermore, the display 26 is also used to display a playback image when the imaging device 10 is in playback mode, a menu screen, and the like.

[0045] The touch panel 28 is a transmissive touch panel, and is overlaid on the surface of the display area of ​​the display 26. The touch panel 28 receives instructions from the user by detecting contact with a pointing object such as a finger or a stylus pen. For ease of explanation, the above-mentioned "full press state" hereinafter also includes a state in which the user turns on a soft key for starting imaging via the touch panel 28.

[0046] In addition, in the present embodiment, an out-cell type touch panel display in which the touch panel 28 is overlaid on the surface of the display area of ​​the display 26 is given as an example of the touch panel display 22, but this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be applied as the touch panel display 22.

[0047] The instruction keys 24 accept various instructions. Here, "various instructions" refers to, for example, an instruction to display a menu screen from which various menus can be selected, an instruction to select one or more menus, an instruction to confirm the selection, an instruction to erase the selection, various instructions such as zooming in, zooming out, and frame-by-frame advance. These instructions may also be given by the touch panel 28.

[0048] As an example, as shown in FIG. 3, the image sensor 16 includes a photoelectric conversion element 30. The photoelectric conversion element 30 has a light receiving surface 30A. The photoelectric conversion element 30 is disposed in the imaging device body 12 (see FIG. 1) so that the center of the light receiving surface 30A coincides with the optical axis OA (see FIG. 1). The photoelectric conversion element 30 has a plurality of photosensitive pixels arranged in a matrix, and the light receiving surface 30A is formed by the plurality of photosensitive pixels. The photosensitive pixels are pixels having a photodiode PD, which photoelectrically convert received light and output an electrical signal according to the amount of received light. There are two types of photosensitive pixels included in the photoelectric conversion element 30: phase difference pixels P, which are so-called image plane phase difference pixels, and non-phase difference pixels N, which are pixels different from the phase difference pixels P.

[0049] The photodiode PD is provided with color filters, including a G filter corresponding to the G (green) wavelength region that contributes most to obtaining a luminance signal, an R filter corresponding to the R (red) wavelength region, and a B filter corresponding to the B (blue) wavelength region.

[0050] In general, the non-phase difference pixel N is also referred to as a normal pixel. The photoelectric conversion element 30 has three types of photosensitive pixels, R pixels, G pixels, and B pixels, as the non-phase difference pixel N. The R pixels, G pixels, B pixels, and phase difference pixel P are regularly arranged with a predetermined periodicity in each of the row direction (for example, the horizontal direction in a state where the bottom surface of the imaging device body 12 is in contact with a horizontal surface) and the column direction (for example, the vertical direction that is perpendicular to the horizontal direction). The R pixels are pixels corresponding to the photodiode PD in which the R filter is arranged, the G pixels and the phase difference pixel P are pixels corresponding to the photodiode PD in which the G filter is arranged, and the B pixels are pixels corresponding to the photodiode PD in which the B filter is arranged.

[0051] A plurality of phase difference pixel lines 32A and a plurality of non-phase difference pixel lines 32B are arranged on the light receiving surface 30A. The phase difference pixel line 32A is a horizontal line including phase difference pixels P. Specifically, the phase difference pixel line 32A is a horizontal line including a mixture of phase difference pixels P and non-phase difference pixels N. The non-phase difference pixel line 32B is a horizontal line including only a plurality of non-phase difference pixels N.

[0052] On the light receiving surface 30A, phase difference pixel lines 32A and non-phase difference pixel lines 32B corresponding to a preset number of lines are alternately arranged along the column direction. The "preset number of lines" referred to here refers to, for example, two lines. Note that, although two lines are exemplified as the preset number of lines here, the technology of the present disclosure is not limited thereto, and the preset number of lines may be a few lines, such as three or more lines, a dozen lines, several tens of lines, or several hundreds of lines.

[0053] The phase difference pixel line 32A is arranged every two rows from the first row to the last row in the column direction. Some pixels in the phase difference pixel line 32A are phase difference pixels P. Specifically, the phase difference pixel line 32A is a horizontal line in which the phase difference pixels P and the non-phase difference pixels N are periodically arranged. The phase difference pixels P are roughly classified into first phase difference pixels L and second phase difference pixels R. In the phase difference pixel line 32A, the first phase difference pixels L and the second phase difference pixels R are alternately arranged as G pixels at intervals of several pixels in the line direction.

[0054] The first phase difference pixel L and the second phase difference pixel R are arranged so as to alternate in the column direction. In the example shown in FIG. 3, in the fourth column, the first phase difference pixel L, the second phase difference pixel R, the first phase difference pixel L, and the second phase difference pixel R are arranged in this order along the column direction from the first row. That is, the first phase difference pixel L and the second phase difference pixel R are arranged alternately along the column direction from the first row. Also, in the example shown in FIG. 3, in the tenth column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are arranged in this order along the column direction from the first row. That is, the second phase difference pixel R and the first phase difference pixel L are arranged alternately along the column direction from the first row.

[0055] The photoelectric conversion element 30 is divided into two regions. That is, the photoelectric conversion element 30 has a non-phase difference pixel division region 30N and a phase difference pixel division region 30P. The phase difference pixel division region 30P is a phase difference pixel group consisting of a plurality of phase difference pixels P, which receives subject light and generates phase difference image data as an electrical signal according to the amount of received light. The phase difference image data is used, for example, for distance measurement. The non-phase difference pixel division region 30N is a non-phase difference pixel group consisting of a plurality of non-phase difference pixels N, which receives subject light and generates non-phase difference image data as an electrical signal according to the amount of received light. The non-phase difference image data is displayed, for example, as a visible light image on the display 26 (see FIG. 2).

[0056] 4, the first phase difference pixel L includes a light shielding member 34A, a microlens 36, and a photodiode PD. In the first phase difference pixel L, the light shielding member 34A is disposed between the microlens 36 and the light receiving surface of the photodiode PD. The left half of the light receiving surface of the photodiode PD in the row direction (the left side when the light receiving surface faces an object (in other words, the right side when the light receiving surface faces an object)) is shielded from light by the light shielding member 34A.

[0057] The second phase difference pixel R includes a light shielding member 34B, a microlens 36, and a photodiode PD. In the second phase difference pixel R, the light shielding member 34B is disposed between the microlens 36 and the light receiving surface of the photodiode PD. The right half of the light receiving surface of the photodiode PD in the row direction (the right side when the light receiving surface faces the subject (in other words, the left side when the light receiving surface faces the subject)) is shielded by the light shielding member 34B. In the following description, for convenience of explanation, when it is not necessary to distinguish between the light shielding members 34A and 34B, they will be referred to as "light shielding members" without reference numerals.

[0058] The interchangeable lens 14 includes an imaging lens 40. A light beam passing through the exit pupil of the imaging lens 40 is roughly divided into a left region passing light 38L and a right region passing light 38R. The left region passing light 38L refers to the left half of the light beam passing through the exit pupil of the imaging lens 40 when facing the subject side from the phase difference pixel P side, and the right region passing light 38R refers to the right half of the light beam passing through the exit pupil of the imaging lens 40 when facing the subject side from the phase difference pixel P side. The light beam passing through the exit pupil of the imaging lens 40 is divided into left and right by the microlens 36, the light shielding member 34A, and the light shielding member 34B that function as a pupil division section, and the first phase difference pixel L receives the left region passing light 38L as the subject light, and the second phase difference pixel R receives the right region passing light 38R as the subject light. As a result, the photoelectric conversion element 30 generates first phase-contrast image data corresponding to the subject image corresponding to the left area passing light 38L and second phase-contrast image data corresponding to the subject image corresponding to the right area passing light 38R.

[0059] In the imaging device 10, for example, in the same phase difference pixel line 32A, the distance to the subject, that is, the subject distance is measured based on the shift amount α between one line of the first phase difference image data and one line of the second phase difference image data (hereinafter, also simply referred to as the "shift amount α"). Note that the method of deriving the subject distance from the shift amount α is a known technique, and therefore a detailed description thereof will be omitted here.

[0060] 5, the non-phase difference pixel N is different from the phase difference pixel P in that it does not have a light blocking member. The photodiode PD of the non-phase difference pixel N receives the left area passing light 38L and the right area passing light 38R as the subject light.

[0061] 6, the imaging lens 40 includes an objective lens 40A, a focus lens 40B, and an aperture 40C. The objective lens 40A, the focus lens 40B, and the aperture 40C are arranged in this order along the optical axis OA from the subject side (object side) to the imaging device body 12 side (image side).

[0062] The interchangeable lens 14 also includes a slide mechanism 42, a motor 44, and a motor 46. A focus lens 40B is attached to the slide mechanism 42 so as to be slidable along the optical axis OA. A motor 44 is connected to the slide mechanism 42, and the slide mechanism 42 moves the focus lens 40B along the optical axis OA by receiving power from the motor 44 and operating. The diaphragm 40C is an diaphragm with a variable opening size. A motor 46 is connected to the diaphragm 40C, and the diaphragm 40C adjusts the exposure by receiving power from the motor 46. The components and / or operation method of the interchangeable lens 14 can be changed as necessary.

[0063] The motors 44 and 46 are connected to the imaging device body 12 via a mount (not shown), and the driving is controlled according to a command from the imaging device body 12. In this embodiment, a stepping motor is used as an example of the motors 44 and 46. Therefore, the motors 44 and 46 operate in synchronization with a pulse signal according to a command from the imaging device body 12. In the example shown in FIG. 6, the motors 44 and 46 are provided in the interchangeable lens 14, but this is not limiting, and one of the motors 44 and 46 may be provided in the imaging device body 12, or both of the motors 44 and 46 may be provided in the imaging device body 12.

[0064] In the imaging device 10, in the imaging mode, the MF mode and the AF mode are selectively set according to an instruction given to the imaging device body 12. The MF mode is an operation mode in which the focus is adjusted manually. In the MF mode, for example, when the user operates the focus ring of the interchangeable lens 14, the focus lens 40B moves along the optical axis OA by an amount corresponding to the amount of operation of the focus ring, thereby adjusting the focus.

[0065] In the AF mode, the imaging device body 12 calculates the in-focus position according to the subject distance, and adjusts the focus by moving the focus lens 40B toward the calculated in-focus position. Here, the in-focus position refers to the position on the optical axis OA of the focus lens 40B when the subject is in focus.

[0066] For ease of explanation, the control for adjusting the focus lens 40B to the in-focus position is also referred to as "AF control" below. For ease of explanation, the calculation of the in-focus position is also referred to as "AF calculation" below. In the imaging device 10, the CPU 48A described below performs AF calculation to detect the focus of a plurality of subjects. Then, the CPU 48A described below focuses on the subjects based on the result of the AF calculation, i.e., the focus detection result.

[0067] The imaging device body 12 includes an image sensor 16, a controller 48, an image memory 50, a UI device 52, an external I / F 54, a photoelectric conversion element driver 56, a motor driver 58, a motor driver 60, a mechanical shutter driver 62, and a mechanical shutter actuator 64. The imaging device body 12 also includes a mechanical shutter 72. The image sensor 16 also includes a signal processing circuit 74.

[0068] The input / output interface 70 is connected to the controller 48 , the image memory 50 , the UI device 52 , the external I / F 54 , the photoelectric conversion element driver 56 , the motor driver 58 , the motor driver 60 , the mechanical shutter driver 62 , and a signal processing circuit 74 .

[0069] The controller 48 includes a CPU 48A, a storage 48B, and a memory 48C. The CPU 48A is an example of a "processor" according to the technology of the present disclosure, the memory 48C is an example of a "memory" according to the technology of the present disclosure, and the controller 48 is an example of an "imaging support device" and a "computer" according to the technology of the present disclosure.

[0070] The CPU 48A, the storage 48B, and the memory 48C are connected via a bus 76, which is connected to the input / output interface 70.

[0071] 6, for convenience of illustration, one bus is shown as the bus 76, but multiple buses may be used. The bus 76 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, and the like.

[0072] The storage 48B stores various parameters and various programs. The storage 48B is a non-volatile storage device. Here, an EEPROM is used as an example of the storage 48B. The EEPROM is merely an example, and instead of the EEPROM, or together with the EEPROM, an HDD and / or an SSD may be applied as the storage 48B. The memory 48C temporarily stores various information and is used as a work memory. An example of the memory 48C is a RAM, but is not limited to this, and may be another type of storage device.

[0073] Various programs are stored in the storage 48B. The CPU 48A reads out necessary programs from the storage 48B and executes the read programs on the memory 48C. The CPU 48A controls the entire imaging device body 12 according to the programs executed on the memory 48C. In the example shown in Fig. 6, the image memory 50, the UI device 52, the external I / F 54, the photoelectric conversion element driver 56, the motor driver 58, the motor driver 60, and the mechanical shutter driver 62 are controlled by the CPU 48A.

[0074] A photoelectric conversion element driver 56 is connected to the photoelectric conversion element 30. The photoelectric conversion element driver 56 supplies an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 30 to the photoelectric conversion element 30 in accordance with an instruction from the CPU 48A. The photoelectric conversion element 30 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 56. Examples of the imaging timing signal include a vertical synchronization signal and a horizontal synchronization signal.

[0075] When the interchangeable lens 14 is attached to the imaging device body 12, the subject light incident on the imaging lens 40 is imaged on the light receiving surface 30A by the imaging lens 40. Under the control of the photoelectric conversion element driver 56, the photoelectric conversion element 30 photoelectrically converts the subject light received by the light receiving surface 30A, and outputs an electrical signal according to the amount of subject light to the signal processing circuit 74 as analog image data indicating the subject light. Specifically, the signal processing circuit 74 reads out the analog image data from the photoelectric conversion element 30 in units of frames and for each horizontal line by an exposure sequential readout method. The analog image data is roughly divided into analog phase difference image data generated by the phase difference pixel P and analog non-phase difference image data generated by the non-phase difference pixel N.

[0076] The signal processing circuit 74 generates digital image data by digitizing the analog image data input from the photoelectric conversion element 30. The signal processing circuit 74 includes a non-phase difference image data processing circuit 74A and a phase difference image data processing circuit 74B. The non-phase difference image data processing circuit 74A generates digital non-phase difference image data by digitizing the analog non-phase difference image data. The phase difference image data processing circuit 74B generates digital phase difference image data by digitizing the analog phase difference image data.

[0077] In the following, for convenience of explanation, when there is no need to distinguish between digital non-phase contrast image data and digital phase contrast image data, they will be referred to as "digital image data." In addition, in the following, for convenience of explanation, when there is no need to distinguish between analog image data and digital image data, they will be referred to as "image data."

[0078] The mechanical shutter 72 is a focal plane shutter, and is disposed between the aperture 40C and the light receiving surface 30A. The mechanical shutter 72 includes a front curtain (not shown) and a rear curtain (not shown). Each of the front curtain and the rear curtain includes a plurality of blades. The front curtain is disposed closer to the subject than the rear curtain.

[0079] The mechanical shutter actuator 64 is an actuator having a front curtain solenoid (not shown) and a rear curtain solenoid (not shown). The front curtain solenoid is a drive source for the front curtain and is mechanically linked to the front curtain. The rear curtain solenoid is a drive source for the rear curtain and is mechanically linked to the rear curtain. The mechanical shutter driver 62 controls the mechanical shutter actuator 64 according to instructions from the CPU 48A.

[0080] The first curtain solenoid generates power under the control of the mechanical shutter driver 62, and selectively winds up and pulls down the first curtain by applying the generated power to the first curtain. The second curtain solenoid generates power under the control of the mechanical shutter driver 62, and selectively winds up and pulls down the second curtain by applying the generated power to the second curtain. In the imaging device 10, the opening and closing of the first curtain and the opening and closing of the second curtain are controlled by the CPU 48A, thereby controlling the amount of exposure to the photoelectric conversion element 30.

[0081] In the imaging device 10, imaging for live view images and imaging for recording images for recording still images and / or moving images are performed by an exposure sequential readout method (rolling shutter method). The image sensor 16 has an electronic shutter function, and imaging for live view images is realized by activating the electronic shutter function without operating the mechanical shutter 72 in a fully open state.

[0082] In contrast, imaging involving main exposure, that is, imaging for a still image (hereinafter also referred to as "main exposure imaging"), is realized by activating the electronic shutter function and operating the mechanical shutter 72 so as to transition the mechanical shutter 72 from a front curtain closed state to a rear curtain closed state. Note that an image obtained by performing main exposure imaging by the imaging device 10 (hereinafter also referred to as a "main exposure image") is an example of a "captured image" according to the technology of the present disclosure.

[0083] Digital image data is stored in the image memory 50. That is, the non-phase difference image data processing circuit 74A stores non-phase difference image data in the image memory 50, and the phase difference image data processing circuit 74B stores phase difference image data in the image memory 50. The CPU 48A acquires the digital image data from the image memory 50 and executes various processes using the acquired digital image data.

[0084] The UI device 52 includes a display 26, and the CPU 48A causes the display 26 to display various pieces of information. The UI device 52 also includes a reception device 80. The reception device 80 includes a touch panel 28 and a hard key unit 82. The hard key unit 82 is a plurality of hard keys including the instruction keys 24 (see FIG. 2). The CPU 48A operates in accordance with various instructions received by the touch panel 28. Note that, although the hard key unit 82 is included in the UI device 52 here, the technology of the present disclosure is not limited to this, and for example, the hard key unit 82 may be connected to the external I / F 54.

[0085] The external I / F 54 controls the exchange of various information with devices (hereinafter also referred to as "external devices") that exist outside the imaging device 10. An example of the external I / F 54 is a USB interface. To the USB interface, external devices (not shown), such as a smart device, a personal computer, a server, a USB memory, a memory card, and / or a printer, are directly or indirectly connected.

[0086] Motor driver 58 is connected to motor 44 and controls motor 44 according to instructions from CPU 48A. By controlling motor 44, the position of focus lens 40B on optical axis OA is controlled via slide mechanism 42. Focus lens 40B moves to avoid the period of main exposure by image sensor 16 according to instructions from CPU 48A.

[0087] The motor driver 60 is connected to the motor 46, and controls the motor 46 according to instructions from the CPU 48A. By controlling the motor 46, the size of the opening of the diaphragm 40C is controlled.

[0088] 7, the storage 48B stores an imaging support processing program 84. The CPU 48A reads out the imaging support processing program 84 from the storage 48B, and executes the read imaging support processing program 84 on the memory 48C. The CPU 48A performs imaging support processing in accordance with the imaging support processing program 84 executed on the memory 48C (see also FIGS. 20A and 20B).

[0089] By executing the imaging support process, the CPU 48A first acquires frequency information indicating the frequency of subject features classified into categories based on the subject features (hereinafter also referred to as "subject features") identified from the main exposure image obtained by imaging by the imaging device 10. Next, the CPU 48A performs a support process (hereinafter also simply referred to as "support process") for supporting imaging by the imaging device 10 based on the acquired frequency information. The contents of the imaging support process will be described in more detail below.

[0090] 8, the CPU 48A operates as an acquisition unit 48A1, an object recognition unit 48A2, a feature extraction unit 48A3, a classification unit 38A4, and a control unit 48A5 by executing an imaging support processing program 84. The classification unit 38A4 is an example of a "classifier" according to the technology of the present disclosure.

[0091] As an example, as shown in FIG. 9, the acquisition unit 48A1 acquires non-phase difference image data as live view image data from the image memory 50. The live view image data is acquired from the image memory 50 by the acquisition unit 48A1 at a default frame rate (for example, 60 fps). The live view image data is image data showing a live view image. The live view image data is obtained by capturing an image of an imaging area by the image sensor 16. In the example shown in FIG. 9, live view image data obtained by capturing an image of an imaging area including multiple people is shown as the live view image data. Here, the multiple people are an example of "multiple objects" according to the technology of the present disclosure.

[0092] In this embodiment, for convenience of explanation, a person is exemplified as the subject of the imaging device 10, but the technology of the present disclosure is not limited thereto, and the subject may be something other than a person. Examples of subjects other than a person include small animals, insects, plants, buildings, landscapes, organs of living things, and / or cells of living things. In other words, the imaging area does not need to include a person, and it is sufficient to include a subject that can be imaged by the image sensor 16.

[0093] Each time one frame of live view image data is acquired by acquisition unit 48A1, control unit 48A5 causes a live view image indicated by the live view image data acquired by acquisition unit 48A1 to be displayed on display 26. The live view image includes a plurality of person images showing a plurality of people as a plurality of subject images showing a plurality of subjects.

[0094] 10 as an example, with a live view image being displayed on display 26, a designated imaging range is determined in accordance with an externally applied instruction (e.g., an instruction accepted by accepting device 80). For example, the designated imaging range is determined by a user's finger operating touch panel 28. One purpose of determining the designated imaging range is, for example, to determine an imaging range to be focused on.

[0095] 10, the designated imaging range is within a rectangular frame (in the example shown in FIG. 10, a frame indicated by a two-dot chain line) formed by a user sliding a finger on touch panel 28 while touching touch panel 28. Control unit 48A5 stores designated imaging range information 86 indicating the designated imaging range determined in accordance with an externally applied instruction in storage 48B. The designated imaging range information 86 stored in storage 48B is updated each time a new designated imaging range is determined.

[0096] 11, the acquisition unit 48A1 acquires non-phase difference image data as main exposure image data from the image memory 50. The object recognition unit 48A2 recognizes the object in the imaging area based on the main exposure image data acquired by the acquisition unit 48A1. In the example shown in FIG. 11, a trained model 92 is stored in the storage 48B, and the object recognition unit 48A2 recognizes the object in the imaging area using the trained model 92.

[0097] An example of the trained model 92 is a trained model using a cascade classifier. The trained model using a cascade classifier is constructed as a trained model for image recognition by performing supervised machine learning on a neural network, for example. Note that the trained model 92 does not need to be limited to a trained model using a cascade classifier, and may be a dictionary for pattern matching. In other words, the trained model 92 may be any trained model that is used in image analysis performed when a subject is recognized.

[0098] The subject recognition unit 48A2 performs image analysis on the main exposure image data to recognize a person included in the imaging area as a subject. The subject recognition unit 48A2 also performs image analysis on the main exposure image data to recognize features of a person, such as the face (facial expression), posture, whether the eyes are open or closed, and the presence or absence of a person within the designated imaging range, as subject features.

[0099] The subject recognition unit 48A2 recognizes a person's face as one of the subject features. The person's face is, for example, any one of a smiling face, a crying face, an angry face, and a straight face. The subject recognition unit 48A2 recognizes, for example, a smiling face, a crying face, an angry face, and a straight face as subject features belonging to a category of "person's face". Note that, here, a smiling face refers to a person's smiling expression, a crying face refers to a person's crying expression, an angry face refers to a person's angry expression, and a straight face refers to an expression that does not correspond to any of a smiling face, a crying face, and an angry face.

[0100] Furthermore, the subject recognition unit 48A2 recognizes the posture of the person as one of the subject characteristics. The posture of the person is, for example, either a frontal or non-frontal orientation. The subject recognition unit 48A2 recognizes, for example, a frontal orientation and a non-frontal orientation as subject characteristics belonging to a category of "posture of the person." Note that, here, a frontal orientation means that the person faces the front with respect to the light receiving surface 30A (see FIG. 6). Also, a non-frontal orientation means that the person faces in a direction other than a frontal orientation.

[0101] Furthermore, the subject recognition unit 48A2 recognizes the eyes of a person as one of the subject features. The eyes of a person are, for example, either open or closed. The subject recognition unit 48A2 recognizes, for example, open and closed eyes as subject features belonging to a category of "person's eyes." Note that here, open eyes refer to a state in which a person has their eyes open, and closed eyes refer to a state in which a person has their eyes closed.

[0102] Furthermore, the subject recognition unit 48A2 recognizes the presence or absence of a person within the designated imaging range as one of the subject features. The subject recognition unit 48A2 recognizes inside the designated imaging range and outside the designated imaging range as subject features belonging to a category called "designated imaging range." Note that, here, "inside the designated imaging range" refers to a state in which a person is present within the designated imaging range, and "outside the designated imaging range" refers to a state in which a person is present outside the designated imaging range.

[0103] Furthermore, the subject recognition unit 48A2 stores in the memory 48C recognition result information 94 indicating the result of recognizing the subject (here, a person as an example) included in the imaging area. The recognition result information 94 is overwritten and saved in the memory 48C on a frame-by-frame basis. The recognition result information 94 is information including the subject name, subject characteristics, and recognition area specifying coordinates, and is stored in the memory 48C in a state in which the subject name, subject characteristics, and recognition area specifying coordinates are associated with each other for each subject included in the imaging area.

[0104] Here, the recognition area specifying coordinates refer to coordinates indicating the position in the live view image of a rectangular frame (hereinafter also referred to as "subject frame") surrounding a characteristic area (for example, a face area indicating a person's face) of the subject image indicating the subject recognized by the subject recognition unit 48A2. An example of the recognition area specifying coordinates is the coordinates of two vertices on the diagonal line of the subject frame in the live view image (for example, the coordinates of the upper left corner and the coordinates of the lower right corner). If the shape of the subject frame is rectangular, the recognition area specifying coordinates may be the coordinates of three vertices or may be the coordinates of four vertices. The shape of the subject frame is not limited to a rectangle and may be another shape. In this case, too, it is sufficient that the coordinates capable of specifying the position of the subject image in the live view image are used as the recognition area specifying coordinates.

[0105] 12, feature extraction unit 48A3 extracts subject-specific feature information from recognition result information 94 stored in memory 48C. Subject-specific feature information is information in which subject names and subject features are associated one-to-one with each other for each subject.

[0106] As an example, as shown in FIG. 13, a category database 96 is constructed in the storage 48B. The category database 96 includes a plurality of subject-specific category groups 98. One subject-specific category group 98 is assigned to each subject. In the example shown in FIG. 13, a plurality of category groups (category groups assigned to each person) including a category group for person A, a category group for person B, a category group for person C, a category group for person D, and a category group for person E are shown as the plurality of subject-specific category groups 98. Each of the plurality of subject-specific category groups 98 is a category capable of identifying each of the plurality of people themselves. Note that the subject-specific category group 98 is an example of an "object target category" according to the technology of the present disclosure.

[0107] The classification unit 48A4 identifies a subject category group 98 corresponding to the subject name from the subject name included in the subject characteristic information, and classifies the subject characteristic corresponding to the subject name into the identified subject category group 98.

[0108] The subject-specific category group 98 includes a plurality of categories. Each of the plurality of categories included in the subject-specific category group 98 is a category created for each unit independent of each other. A unit refers to a unit of subject features. In the example shown in FIG. 13, a face category, a posture category, an eye category, and a designated imaging range category are shown as examples of the plurality of categories included in the subject-specific category group 98. The face category is a category defined by a subject feature in the unit of "person's face". The posture category is a category defined by a subject feature in the unit of "person's posture". The eye category is a category defined by a subject feature in the unit of "person's eyes". The designated imaging range category is a category defined by a subject feature in the unit of "designated imaging range" applied to a person.

[0109] As an example, as shown in FIG. 14, if the face category, posture category, eye category, and specified imaging range category are taken as major categories, multiple small categories are subordinate to the major categories. The small categories are categories into which subject features subordinate to the subject features indicated by the major categories are classified. The multiple small categories are associated with a classification count. Subject features are classified into the small categories by classification unit 48A4. Each time a subject feature is classified and added to a small category, "1" is added to the classification count as the number of times the subject feature has been classified.

[0110] In the example shown in FIG. 14, a smiling face category, a crying face category, an angry face category, and a neutral face category are shown as a number of categories included in the face category in the subject-specific category group 98 used for person A. The classification unit 48A4 classifies the subject feature of "smiling face" of person A into the smiling face category. Then, each time the classification unit 48A4 classifies the subject feature of "smiling face" into the smiling face category, it adds "1" to the number of classifications for the smiling face category. Similarly to the subject feature of "smiling face", each of the subject features of person A, "crying face", "angry face", and "neutral face", is classified into the corresponding category by the classification unit 48A4, and "1" is added to the corresponding number of classifications.

[0111] The number of classifications is also associated with each face category. The number of classifications for a face category is the sum of the number of classifications for the smiling face category, the crying face category, the angry face category, and the neutral face category.

[0112] 14, a frontal category and a non-frontal category are shown as a plurality of categories included in the posture category in the subject-specific category group 98 used for person A. The classification unit 48A4 classifies the subject feature "front" of person A into the frontal category. Then, each time the classification unit 48A4 classifies the subject feature "front" into the frontal category, it adds "1" to the classification count for the frontal category. Similarly to the subject feature "front", the subject feature "non-front" of person A is also classified into the non-frontal category by the classification unit 48A4, and "1" is added to the corresponding classification count.

[0113] The number of classifications is also associated with each posture category. The number of classifications for a posture category is the sum of the number of classifications for the frontal category and the number of classifications for the non-frontal category.

[0114] 14, an eyes open category and an eyes closed category are shown as a plurality of categories included in the eye category in the subject-specific category group 98 used for person A. Classification unit 48A4 classifies the subject feature "eyes open" of person A into the eyes open category. Then, each time classification unit 48A4 classifies the subject feature "eyes open" into the eyes open category, it adds "1" to the classification count for the eyes open category. Similarly to the subject feature "eyes open," classification unit 48A4 also classifies the subject feature "closed eyes" of person A into the eyes closed category, and adds "1" to the corresponding classification count.

[0115] The eye category is also associated with a classification count, which is the sum of the classification count for the eyes open category and the classification count for the eyes closed category.

[0116] 14, a category within the designated imaging range and a category outside the designated imaging range are shown as categories included in the designated imaging range category in the subject-specific category group 98 used for person A. The classification unit 48A4 classifies the subject feature "within the designated imaging range" into the designated imaging range category. Then, each time the classification unit 48A4 classifies the subject feature "within the designated imaging range" into the designated imaging range category, it adds "1" to the classification count for the designated imaging range category. Similarly to the subject feature "within the designated imaging range", the subject feature "outside the designated imaging range" is also classified into the outside the designated imaging range category by the classification unit 48A4, and "1" is added to the corresponding classification count.

[0117] The designated imaging range category is also associated with a classification count. The classification count for a designated imaging range category is the sum of the classification count for the inside-designated imaging range category and the classification count for the outside-designated imaging range category.

[0118] The subject-specific category group 98 is a category defined by a subject feature in units of "subject name", and subject features called subject names are classified into the subject-specific category group 98. A classification count is also associated with the subject-specific category group 98. The classification count associated with the subject-specific category group 98 is the sum of the classification counts of multiple large categories belonging to the subject-specific category group 98.

[0119] 15 as an example, the acquisition unit 48A1 acquires non-phase contrast image data as live view image data from the image memory 50. The object recognition unit 48A2 recognizes the object in the imaging area using the trained model 92, based on the live view image data acquired by the acquisition unit 48A1.

[0120] The subject recognition unit 48A2 performs image analysis on the live view image data to recognize a person included in the imaging area as a subject. The subject recognition unit 48A2 also performs image analysis on the live view image data to recognize features of a person, such as the face of the person, the posture of the person, whether the person's eyes are open or closed, and the presence or absence of a person within the designated imaging range, as subject features.

[0121] That is, the subject recognition unit 48A2 recognizes a smiling face, a crying face, an angry face, and a straight face as subject features belonging to the face category by performing image analysis on the live view image data. The subject recognition unit 48A2 also recognizes a front and a non-front as subject features belonging to the posture category by performing image analysis on the live view image data. The subject recognition unit 48A2 also recognizes an open eye and a closed eye as subject features belonging to the eye category by performing image analysis on the live view image data. The subject recognition unit 48A2 also recognizes an inside of the designated imaging range and an outside of the designated imaging range as subject features belonging to the designated imaging range category by performing image analysis on the live view image data. The subject recognition unit 48A2 stores recognition result information 94 indicating the result of recognizing a person included in the imaging area in the memory 48C. The recognition result information 94 is overwritten and saved in the memory 48C on a frame-by-frame basis.

[0122] 16 as an example, control unit 48A5 causes display 26 to display a live view image indicated by live view image data acquired by acquisition unit 48A1. Control unit 48A5 also refers to recognition result information 94 stored in memory 48C, and acquires the number of times of classification for each category for a person recognized as a subject by subject recognition unit 48A2 based on the live view image data (hereinafter also referred to as a "recognized person") from each category included in category database 96. That is, control unit 48A5 acquires the number of times of classification for each recognized person from each category of subject-specific category group 98 corresponding to the recognized person.

[0123] The control unit 48A5 generates an imaging support screen 100 based on the classification count acquired from each category included in the category database 96, and superimposes and displays the generated imaging support screen 100 on a live view image. In the example shown in Fig. 16, the imaging support screen 100 is displayed in the upper left of the live view image.

[0124] 17, a bubble chart 100A and a category selection screen 100B are displayed in an imaging support screen 100. The bubble chart 100A is a chart in which bubbles indicating the number of classifications are plotted on two axes, an axis indicating a plurality of recognized persons and an axis indicating a category.

[0125] 17, a bubble chart related to face categories (hereinafter, also referred to as a "face category bubble chart") is shown as an example of bubble chart 100A. The face category bubble chart is a chart in which bubbles are plotted for each of a plurality of persons including persons A to E, representing the number of times each face falls into the smiling face category, crying face category, angry face category, and neutral face category, which belong to the face category.

[0126] In addition to the face category bubble chart, the imaging support screen 100 selectively displays a bubble chart related to the posture category (hereinafter also referred to as a "posture category bubble chart"), a bubble chart related to the eye category (hereinafter also referred to as an "eye category bubble chart"), and a bubble chart related to the designated imaging range category (hereinafter also referred to as a "designated imaging range category bubble chart") according to an instruction given from outside. Here, an instruction given from outside includes, for example, an instruction received by the reception device 80. In the example shown in FIG. 17, a plurality of soft keys indicating the names of the categories are displayed in a scroll manner in the category selection screen 100B. When any of the soft keys is turned on via the touch panel by the user's finger operation, a bubble chart related to the category corresponding to the turned on soft key is displayed as a new bubble chart 100A in the imaging support screen 100.

[0127] Furthermore, when any person in bubble chart 100A is selected via the touch panel by the user's finger operation, the classification frequency of each category of bubble chart 100A currently displayed for the selected person is displayed as histogram 100C (see FIG. 18) on imaging support screen 100. In the example shown in FIG. 17, person A in bubble chart 100A is selected via the touch panel by the user's finger operation. In this case, as an example, as shown in FIG. 18, a histogram showing the imaging tendency of face categories for person A (hereinafter, also referred to as a "face category histogram") is displayed as histogram 100C. In the face category histogram, the horizontal axis indicates each category of a smiling face category, a crying face category, an angry face category, and a neutral face category, and the vertical axis indicates the classification frequency.

[0128] In the example shown in FIG. 18, a face category histogram is shown as the histogram 100C, but different types of histograms are displayed as the histogram 100C depending on the type of the bubble chart 100A and the person selected by the user. In addition to the face category histogram, the types of the histogram 100C include, for example, a histogram showing an imaging tendency of a posture category (hereinafter also referred to as a "posture category histogram"), a histogram showing an imaging tendency of an eye category (hereinafter also referred to as an "eye category histogram"), and a histogram showing an imaging tendency of a designated imaging range category (hereinafter also referred to as a "designated imaging range category histogram") for each selected person. In addition, the histogram is not limited to a histogram for each person, and may be a histogram in which the horizontal axis represents persons A to E and the vertical axis represents the number of times the subject is classified into the subject-specific category group 98. In this way, the horizontal and vertical axes of the histogram may be axes of any elements that can constitute a histogram.

[0129] In the example shown in FIG. 18, a display switching instruction screen 100D is displayed within the imaging support screen 100. The display switching instruction screen 100D is a screen for receiving an instruction to switch from the display of the histogram 100C to the display of the bubble chart 100A. The display switching instruction screen 100D displays a message (in the example shown in FIG. 18, a message saying "Do you want to return to the bubble chart screen?") instructing the user to switch to the bubble chart 100A and a switching soft key (in the example shown in FIG. 18, a soft key saying "Yes") on the screen. When the switching soft key is turned on via the touch panel by the user's finger operation, the display is switched from the histogram 100C to the bubble chart 100A.

[0130] Here, the classification count of any of the categories in histogram 100C is specified by reception device 80 (touch panel 28 in the example shown in FIG. 18). In the example shown in FIG. 18, the smiley face category is selected by the user's finger via touch panel 28, thereby specifying the classification count of the smiley face category. When the classification count of any of the categories is specified in this manner, control unit 48A5 performs a support process for the category corresponding to the specified classification count (hereinafter also referred to as the "target category").

[0131] 18, the smiling face category is specified as the target category from among multiple categories (smiling face category, crying face category, angry face category, and neutral face category) in histogram 100C for person A. The target category is a category determined based on the number of classifications (in the example shown in FIG. 18, the category is specified based on the number of classifications in accordance with an instruction received by touch panel 28). In this case, as a support process, a process is performed to support imaging using image sensor 16 of person A (hereinafter also referred to as "target category subject S" (see FIG. 19)) who has a subject characteristic of "smiling face" that belongs to the smiling face category.

[0132] When a target category subject is imaged by the imaging device 10, the target category is a category determined according to the state of the target category subject S (e.g., facial expression, posture, eye opening / closing state, and / or the positional relationship between the person and the designated imaging range, etc.) and is a category into which the subject features of the target category subject S are classified. In the example shown in Fig. 18, when a person A is imaged by the imaging device 10 as the target category subject S, the smile category into which the subject feature of a smile, which is the least frequently imaged expression among a smile, a crying face, an angry face, and a straight face, of the person A, is classified is specified as the target category by the user.

[0133] The smiling face category specified by the user as the target category from the histogram 100C is the category with the lowest classification frequency compared to the crying face category, the angry face category, and the neutral face category. This means that the number of actual-exposure images in which person A is smiling is smaller than the number of actual-exposure images in which person A is smiling with other facial expressions. Note that the smiling face category in the histogram 100C is an example of a "low-frequency category" according to the technology of the present disclosure.

[0134] When the target category is specified from the histogram 100C in this way, as shown in FIG. 19 as an example, the control unit 48A5 causes the display 26 to display a live view image obtained by capturing an image of an imaging area including the target category subject S by the imaging device 10, and performs a display process as a support process to identify a smiling target category subject S based on the recognition result information 94 and to display a display recommending capturing an image of the identified target category subject S. The display process includes a process of displaying an arrow pointing to a target category subject image S1 indicating a smiling target category subject S on the live view image, and displaying information recommending capturing an image of the target category subject S indicated by the target category subject image S1 pointed to by the arrow (hereinafter also referred to as "imaging recommendation information"). In the example shown in FIG. 19, a message saying "We recommend capturing an image of this subject" is displayed as an example of the imaging recommendation information.

[0135] The display process also includes a process of displaying the target category subject image S1 in a manner distinguishable from other image areas in the live view image. In this case, for example, the control unit 48A5 detects a face area (image area showing the face of the target category subject S) of the target category subject image based on the recognition result information 94 stored in the memory 48C, and displays a detection frame 102 surrounding the detected face area in the live view image, thereby displaying the target category subject image S1 in a manner distinguishable from other image areas in the live view image.

[0136] Note that the method of displaying the target category subject image S1 in a manner that allows it to be distinguished from other image areas within the live view image is not limited to this; for example, only the target category subject image S1 may be displayed within the live view image using a peaking method.

[0137] Next, the operation of the imaging device 10 will be described with reference to FIGS. 20A and 20B.

[0138] 20A and 20B show an example of the flow of imaging support processing executed by the CPU 48A when an imaging mode is set for the imaging device 10. The flow of the imaging support processing is an example of an "imaging support method" according to the technology of the present disclosure. For convenience of explanation, the following description will be given on the assumption that the imaging device 10 captures images for live view images at a default frame rate. For convenience of explanation, the following description will be given on the assumption that the designated imaging range information 86 is stored in the storage 48B and the category database 96 has been constructed.

[0139] In the imaging support process shown in FIG. 20A, first, the acquisition unit 48A1 acquires live view image data from the image memory 50 in step ST100.

[0140] In the next step ST102, the control unit 48A5 causes the display 26 to display a live view image indicated by the live view image data acquired in step ST100.

[0141] In the next step ST104, the object recognition unit 48A2 recognizes the object in the imaging area using the trained model 92 based on the live view image data acquired in step ST100.

[0142] In the next step ST106, the control unit 48A5 acquires from the category database 96 the number of times the subject recognized in step ST104 is classified into each category.

[0143] In the next step ST108, the control unit 48A5 creates an imaging support screen 100 based on the classification count acquired in step ST106, and displays the created imaging support screen 100 in a partial area within the live view image.

[0144] In the next step ST110, the control unit 48A5 determines whether or not any of the classification numbers has been specified from the histogram 100C in the imaging support screen 100. In step ST110, if any of the classification numbers has not yet been specified from the histogram 100C in the imaging support screen 100, the determination is negative, and the imaging support process proceeds to step ST116 shown in Fig. 20B. In step ST110, if any of the classification numbers has been specified from the histogram 100C in the imaging support screen 100, the determination is positive, and the imaging support process proceeds to step ST112.

[0145] In step ST112, the control unit 48A5 displays a detection frame 102 in the live view image so as to surround the face area of ​​a target category subject image S1 indicating a target category subject S having subject features belonging to a specified target category based on the specified classification count.

[0146] In the next step ST114, the control unit 48A5 displays the imaging recommendation information in the live view image. After the processing of step ST114 is executed, the imaging support processing proceeds to step ST116 shown in FIG.

[0147] In step ST116, the control unit 48A5 determines whether or not a condition for starting the main exposure (hereinafter, referred to as a "main exposure start condition") is satisfied. One example of the main exposure start condition is the condition that the above-mentioned full press state is reached by an instruction accepted by the accepting device 80. In step ST116, if the main exposure start condition is not satisfied, the determination is negative, and the imaging support process proceeds to step ST130. In step ST116, if the main exposure start condition is satisfied, the determination is positive, and the imaging support process proceeds to step ST118.

[0148] In step ST118, the control unit 48A5 causes the image sensor 16 to perform main exposure imaging of the imaging area including the target category subject S. By performing the main exposure imaging, main exposure image data indicating the main exposure image of the imaging area including the target category subject S is stored in the image memory 50.

[0149] In the next step ST120, the acquisition unit 48A1 acquires the main exposure image data from the image memory 50.

[0150] In the next step ST122, the object recognition unit 48A2 recognizes the object in the imaging area using the trained model 92 based on the actual exposure image data acquired in step ST120, and stores recognition result information 94 in the memory 48C.

[0151] In the next step ST124, the feature extraction unit 48A3 extracts object-specific feature information for each object from the recognition result information 94 stored in the memory 48C.

[0152] In the next step ST126, the classification unit 48A4 identifies a subject-specific category group 98 corresponding to the subject name from the subject name included in the subject-specific feature information extracted in step ST124, and classifies the subject features into the corresponding category within the identified subject-specific category group 98.

[0153] In the next step ST128, the classification unit 48A4 updates the classification count by adding "1" to the classification count of the category into which the subject features have been classified.

[0154] In the next step ST130, the control unit 48A5 erases the live view image and the like (for example, the live view image, the imaging support screen 100, the detection frame 102, and the imaging recommendation information) from the display 26.

[0155] In the next step ST132, the control unit 48A5 judges whether or not a condition for terminating the imaging support process (hereinafter also referred to as an "imaging support process termination condition") has been satisfied. Examples of the imaging support process termination condition include a condition that the imaging mode set for the imaging device 10 has been released, or a condition that an instruction to terminate the imaging support process has been accepted by the acceptance device 80. In step ST120, if the imaging support process termination condition is not satisfied, the judgment is negative, and the imaging support process proceeds to step ST100. In step ST132, if the imaging support process termination condition is satisfied, the judgment is positive, and the imaging support process ends.

[0156] As described above, in the imaging device 10, the control unit 48A5 acquires the classification count of the subject features classified into categories based on the subject features of the subject identified from the actual exposure image. Then, the control unit 48A5 performs a support process for supporting imaging by the imaging device 10 based on the classification count. Therefore, according to this configuration, it is possible to support imaging by the imaging device 10 according to the classification count into categories of the subject features.

[0157] In addition, in the imaging device 10, categories are classified into a plurality of categories including a target category. The target category is a category determined based on the number of classifications. In the example shown in FIG. 18, the target category is specified by selecting the number of classifications by the user. Then, the control unit 48A5 performs a process of supporting imaging of a target category subject S having subject characteristics belonging to the target category. In the example shown in FIG. 19, a process of supporting imaging of a smiling target category subject S is performed.

[0158] In the example shown in FIG. 19, the smile category is specified by selecting the classification count of the smile category, and thus image capture of the smiling target category subject S is supported; however, when the classification count of other subject features (e.g., crying face, angry face, and / or serious face) is selected by the user, and a subcategory other than the smile category is specified among multiple subcategories within the face category, a process of supporting image capture of the target category subject S having subject features belonging to the specified target category is performed.

[0159] When a small category is specified by the user selecting the classification frequency of subject features that are classified into small categories belonging to other large categories, not limited to small categories belonging to the face category, a process is performed to assist in imaging of a target category subject S having subject features that belong to the small category specified as the target category.

[0160] In addition, when a subject-specific category group 98 is specified by a user selecting the classification count of a subject name classified as a subject feature in the subject-specific category group 98, a process of assisting in image capture is performed for a target category subject S having a subject feature belonging to the subject-specific category group 98 specified as the target category, i.e., a target category subject S having a subject name corresponding to the subject-specific category group 98 specified as the target category.

[0161] Therefore, with this configuration, it is possible to efficiently capture a target category subject desired by a user, compared to a case where an image is captured based solely on the photographer's intuition as to whether or not the target category subject S is desired by a user.

[0162] Furthermore, in this embodiment, imaging recommendation information (in the example shown in FIG. 19, a message saying "We recommend taking a picture of this subject") is displayed as a display recommending imaging of the target category subject S. Therefore, according to this configuration, it is possible to efficiently image a target category subject with an expression desired by the user, compared to a case where imaging is performed based on a judgement of whether or not the target category subject S has the subject characteristics desired by the user based only on the photographer's intuition.

[0163] Furthermore, in this embodiment, the control unit 48A5 displays a live view image on the display 26, and displays the target category subject image S1 in the live view image in a manner that allows it to be distinguished from other image areas. Therefore, according to this configuration, the user can visually recognize the target category subject.

[0164] Furthermore, in this embodiment, the control unit 48A5 displays a live view image on the display 26, and also displays a detection frame 102 in the live view image for the face region of the target category subject image S1. Therefore, according to this configuration, it is possible for the user to recognize that the subject corresponding to the display region in which the detection frame 102 is displayed is the target category subject.

[0165] In this embodiment, a category that is relatively low in the number of times classified among multiple categories (the smile category in the example shown in FIG. 18) is specified by the user as the target category. Therefore, according to this configuration, it is possible to efficiently capture an image of a subject in a category that is relatively low in the number of times classified among multiple categories, compared to a case where an image is captured by determining only based on the photographer's intuition whether the subject is in a category that is relatively low in the number of times classified among multiple categories.

[0166] Furthermore, in this embodiment, the target category is a category determined according to the state of the target category subject S (for example, a person's facial expression). In the example shown in FIG. 19, a smile category determined according to the state of the subject S, "smile", is specified as the target category. Then, the control unit 48A5 performs a process of supporting image capture of the target category subject S, which is a subject having a subject characteristic classified into the specified category. Therefore, according to this configuration, it is possible to increase the number of times images are captured of a subject in the same state, or conversely, to decrease the number of times images are captured of a subject in the same state.

[0167] Furthermore, in this embodiment, a plurality of subject-specific category groups 98 are included in the category database 96 as categories capable of identifying each of a plurality of people themselves. Therefore, the control unit 48A5 performs a process of supporting image capture of a target category subject S, which is a subject corresponding to the subject-specific category group 98 specified by the user. Therefore, according to this configuration, it is possible to increase the number of times that the same subject is captured, or conversely, to decrease the number of times that the same subject is captured.

[0168] In this embodiment, categories into which subject features are classified are created for each of a plurality of units. Examples of the plurality of units include a unit called "subject name (e.g., a person's name or an identifier capable of identifying a person)", a unit called "person's face (expression)", a unit called "person's posture", a unit called "person's eyes", and a unit called "designated imaging range". Therefore, according to this configuration, imaging by the imaging device 10 can be supported according to the number of times that subject features are classified into categories of designated units.

[0169] In the present embodiment, the control unit 48A5 performs a process including a process of displaying the classification count on the display 26 as a support process for supporting imaging by the imaging device 10. In the example shown in Fig. 17, the classification count is represented using a bubble chart 100A, and in the example shown in Fig. 18, the classification count is represented using a histogram 100C. Therefore, according to this configuration, the user can know the number of times imaging has been performed for each of various subject features.

[0170] Furthermore, in this embodiment, when the number of classifications in histogram 100C is specified by the accepting device 80 while histogram 100C is being displayed on display 26, control unit 48A5 performs a process of supporting image capture for the category corresponding to the specified number of classifications. Therefore, according to this configuration, it is possible to support image capture of a subject having subject characteristics intended by the user.

[0171] In the above embodiment, an example in which the target category subject S is located within the designated image capture range has been described, but the technology of the present disclosure is not limited to this. For example, when the target category subject S is located outside the designated image capture range, as shown in FIG. 21 as an example, the target category subject image S1 is displayed in a position outside the object Ob indicating the designated image capture range in the live view image. In this case, the control unit 48A5 displays the object Ob indicating the designated image capture range and the target category subject image S1 (an example of the "object indicating the target category subject" according to the technology of the present disclosure) in a different display mode in the live view image. In the example shown in FIG. 21, the target category subject image S1 is displayed in a peaking manner, and the object Ob is colored in a translucent color (for example, translucent gray). This allows the user to visually distinguish the designated image capture range and the target category subject S that he or she intends to capture.

[0172] In the above embodiment, the number of classifications shown in the histogram 100C is specified by the user via the touch panel 28, and imaging related to the category corresponding to the specified number of classifications (smile category in the example shown in FIG. 18) is supported, but the technology of the present disclosure is not limited to this. For example, the control unit 48A5 may specify a low-frequency category with a relatively low number of classifications as a target category among a plurality of categories, and perform a process of supporting imaging related to the specified low-frequency category. For example, the low-frequency category refers to a category (e.g., smile category) with the smallest number of classifications among a plurality of small categories (smile category, crying face category, angry face category, and straight face category) belonging to a large category (e.g., face category) specified by the user via the reception device 80. The control unit 48A5 specifies a target subject having a subject feature classified into the low-frequency category among target subjects (e.g., person A) specified by the user via the reception device 80 in the live view image, and performs a process of supporting imaging of the specified target subject. In this case, selection (operation by the user) on the histogram 100C is not necessary, and the imaging support screen 100 may not be displayed.

[0173] In this way, when the control unit 48A5 performs the process of supporting imaging of a target subject having subject features classified into a low frequency category, the imaging support process is executed by the CPU 48A as shown in Fig. 22 as an example. The flowchart shown in Fig. 22 differs from the flowchart shown in Fig. 20A in that it has steps ST200 and ST202 instead of steps ST106 to ST112.

[0174] In step ST200 of the imaging support process shown in Fig. 22, control unit 48A5 specifies a low frequency category from among multiple small categories in a large category specified by the user for a target subject specified by the user. For example, according to bubble chart 100A shown in Fig. 22, when the user specifies a face category as the large category and a person A as the target subject, control unit 48A5 specifies a smile category as the low frequency category.

[0175] In the next step ST202, the control unit 48A5 specifies a subject image (hereinafter also referred to as a "subject of interest image") showing a subject of interest having a subject feature classified into a low frequency category among the subjects of interest specified by the user in the live view image, and displays a detection frame 102 (see Figs. 19 and 21) for a face region of the subject of interest image. For example, when a person A is specified as a subject of interest by the user and a smile category is automatically specified as a low frequency category in step ST200, the control unit 48A5 first specifies a person image showing a smiling person A among the persons A as a subject of interest image in the live view image based on the recognition result information 94. Then, the control unit 48A5 displays a detection frame 102 surrounding a face region of the person image showing the smiling person A in the live view image. Note that in this embodiment, "automatic" does not mean that the movement is triggered by an artificial operation (for example, an operation by the user) but that the movement is mainly performed by the controller 48.

[0176] Therefore, according to the example shown in FIG. 22, imaging of a subject having subject characteristics that are classified into the category with the least number of classifications is supported, so that subjects in the low-frequency category can be imaged more efficiently than when the subject is imaged based solely on the photographer's intuition as to whether or not it is in the low-frequency category.

[0177] In the above embodiment, an example has been described in which main exposure imaging is performed when the condition that an instruction from a user is accepted by the accepting device 80 is satisfied as the main exposure start condition in the imaging support process, but the technology of the present disclosure is not limited to this. For example, the control unit 48A5 may detect a target category subject based on an imaging result by the imaging device 10, and automatically acquire images including an image corresponding to the target category subject on the condition that the target category subject has been detected.

[0178] In this case, for example, the control unit 48A5 first specifies the low frequency category as the target category, as in the example shown in FIG. 22. Then, when the control unit 48A5 detects, based on the recognition result information 94, that a low frequency attention subject image, which is an attention subject image corresponding to the low frequency category, exists in the live view image, the main exposure imaging is automatically started. Here, the low frequency attention subject image corresponding to the low frequency category refers to, for example, an attention subject image showing person A having a subject characteristic of "smile" among person A when the control unit 48A5 specifies the smile category as the low frequency category on the premise that person A is designated as the attention subject. Note that, by performing the main exposure imaging, one frame of main exposure image data may be stored in a default storage area (for example, the image memory 50), or only data related to the low frequency attention subject image among one frame of main exposure image data may be stored in a default storage area.

[0179] In this way, when images including images corresponding to target category subjects are automatically acquired by control unit 48A5 on the condition that a target category subject has been detected, imaging support processing is executed by CPU 48A as shown in Fig. 23 as an example. The flowchart shown in Fig. 23 differs from the flowcharts shown in Fig. 20A and Fig. 20B in that it has steps ST300 and ST302 instead of steps ST106 to ST116.

[0180] In step ST300 of the imaging support process shown in FIG. 23, the control unit 48A5 executes the same process as the process in step ST200 shown in FIG.

[0181] In the next step ST302, the control unit 48A5 determines whether or not a low-frequency target subject image is present in the live view image based on the recognition result information 94. In step ST302, if a low-frequency target subject image is not present in the live view image, the determination is negative, and the imaging support process proceeds to step ST130. In step ST302, if a low-frequency target subject image is present in the live view image, the determination is positive, and the imaging support process proceeds to step ST118.

[0182] In this way, when the control unit 48A5 detects that a low-frequency target subject image is present in the live view image, the main exposure imaging is started, thereby reducing the effort required to image the target category subject compared to when imaging is started on the condition that the target category subject is visually found and an instruction from the user is accepted by the receiving device 80.

[0183] In the example shown in Fig. 23, in step ST302 of the imaging support process, it is determined whether or not a low-frequency target subject image is present in a live view image, and if it is determined that a low-frequency target subject image is present in the live view image, a form example in which main exposure imaging is started has been described, but the technology of the present disclosure is not limited to this. For example, instead of the process of step ST302 shown in Fig. 23, the process of step ST352 shown in Fig. 24 may be executed by the control unit 48A5.

[0184] In step ST352 of the imaging support process shown in FIG. 24, the control unit 48A5 judges whether or not the imaging range condition is satisfied. Here, the imaging range condition refers to a condition that the target category subject is included in the designated imaging range indicated by the designated imaging range information 86 (see FIG. 10 and FIG. 11). The judgment of whether or not the target category subject is included in the designated imaging range is performed, for example, by judging whether or not the attention subject indicated by the low-frequency attention subject image is included in the designated imaging range based on the designated imaging range information 86 and the recognition result information 94. In step ST352, if the imaging range condition is not satisfied, the judgment is negative, and the imaging support process proceeds to step ST130. In step ST352, if the imaging range condition is satisfied, the judgment is positive, and the imaging support process proceeds to step ST118.

[0185] In this way, main exposure imaging is performed on the condition that the target category subject is included within the designated imaging range, which reduces the effort required to image the target category subject compared to when imaging is started on the condition that the target category subject is visually found to be included within the designated imaging range and an instruction from the user is accepted by the receiving device 80.

[0186] 24, main exposure imaging is performed in step ST118 of the imaging support process, but imaging may be performed with the focus on a target category subject within a designated imaging range. The "imaging" here may be imaging for a live view image or imaging for a recorded image (e.g., a still image or a moving image).

[0187] In this case, the CPU 48A executes the imaging support process shown in Fig. 25. The flowchart shown in Fig. 25 differs from the flowchart shown in Fig. 24 in that it includes step ST400 instead of step ST118.

[0188] 25, the control unit 48A5 focuses on a target category subject (for example, a target subject indicated by a low-frequency target subject image in a live view image) based on the result of the AF calculation for an imaging area including the target category subject S, and then causes the image sensor 16 to perform main exposure imaging. After the processing of step ST400 is executed, the imaging support processing proceeds to step ST120.

[0189] In this way, the main exposure imaging is performed after the target category subject is focused on within the designated imaging range, thereby reducing the effort required to position the target category subject within the designated imaging range, focus on the target category subject, and then image the subject.

[0190] 25, an example of a form in which main exposure imaging is performed when the imaging range condition is satisfied has been described, but the technology of the present disclosure is not limited to this. For example, the control unit 48A5 may perform a default process when the degree of difference between the first imaging condition given from the outside (e.g., the imaging condition determined according to the instruction received by the receiving device 80) and the second imaging condition given to the target category subject is equal to or greater than a default degree of difference.

[0191] In this case, for example, the imaging support process shown in Fig. 26 is executed by the CPU 48A. The flowchart shown in Fig. 26 is different from the flowchart shown in Fig. 25 in that it includes steps ST450 to ST456 instead of steps ST352, ST400, and ST120.

[0192] In step ST450 of the imaging support process shown in Fig. 26, the control unit 48A5 acquires the first imaging condition and the second imaging condition. Examples of the first imaging condition and the second imaging condition will be described later.

[0193] In the next step ST452, control unit 48A5 calculates the degree of difference between the first and second imaging conditions acquired in step ST450 (for example, a value indicating the degree of difference between the first and second imaging conditions).

[0194] In the next step ST454, the control unit 48A5 judges whether or not the degree of dissimilarity calculated in step ST452 is equal to or greater than the default degree of dissimilarity. In step ST454, if the degree of dissimilarity calculated in step ST452 is less than the default degree of dissimilarity, the judgment is negative, and the control unit 48A5 executes processing equivalent to the processing of step ST400 and the processing of steps ST120 to ST132 (see FIG. 25), and then proceeds to step ST100. In step ST454, if the degree of dissimilarity calculated in step ST452 is equal to or greater than the default degree of dissimilarity, the judgment is positive, and the imaging support processing proceeds to step ST456.

[0195] In step ST456, the control unit 48A5 executes a default process. Although details will be described later, the default process is a process including a post-depth-of-field-adjusted imaging process for performing actual exposure imaging after adjusting the depth of field, and / or a focus bracketing imaging process for performing actual exposure imaging by the focus bracketing method. After the process of step ST456 is executed, the imaging support process proceeds to step ST122.

[0196] Thus, according to the example shown in FIG. 26, when the degree of difference between the first imaging condition given from outside and the second imaging condition given for the target category subject is equal to or greater than the default degree of difference, it is possible to contribute to capturing images under the imaging conditions intended by the user.

[0197] 27 and 28 show the contents of the imaging process after depth of field adjustment. As an example, as shown in FIG. 27, when the first imaging condition is the position of the designated imaging range, the second imaging condition is the position of the target category subject, and the position of the target category subject is not within the designated imaging range, the control unit 48A5 determines that the dissimilarity is equal to or greater than the default dissimilarity. When the dissimilarity is equal to or greater than the default dissimilarity, the control unit 48A5 controls the imaging device 10 to place the designated imaging range and the target category subject within the depth of field, and causes the image sensor 16 to perform main exposure imaging with the designated imaging range and the target category subject within the depth of field.

[0198] When the dissimilarity is equal to or greater than the default dissimilarity, as shown in FIG. 28 as an example, first, the acquisition unit 48A1 calculates the focus position for each of the subjects within the designated imaging range and the target category subject based on the result of the AF calculation for each of the multiple subjects, i.e., at least one subject within the designated imaging range (hereinafter also referred to as a "subject within the designated imaging range") and the target category subject. For example, the acquisition unit 48A1 calculates the focus position for each of the subjects within the designated imaging range and the target category subject based on phase difference image data corresponding to each position of the subject image within the designated imaging range (image showing the subject within the designated imaging range) and the target category subject image S1 in the live view image. Note that the calculation method of the focus position is merely an example, and the focus position may be calculated by a calculation method of the TOF method or the contrast AF method.

[0199] The acquisition unit 48A1 calculates a depth of field that contains the subjects within the designated image capture range and the subjects of the target category based on a plurality of focus positions calculated for each of the subjects within the designated image capture range and the subjects of the target category. The depth of field is calculated using a first arithmetic expression. The first arithmetic expression is, for example, an arithmetic expression in which the plurality of focus positions are independent variables and the depth of field is a dependent variable. Note that, instead of the first arithmetic expression, a first table in which a plurality of focus positions are associated with the depth of field may be used.

[0200] The acquisition unit 48A1 calculates an F-number that realizes the calculated depth of field. The acquisition unit 48A1 calculates the F-number using a second arithmetic expression. The second arithmetic expression used here is, for example, an arithmetic expression in which the depth of field is an independent variable and the F-number is a dependent variable. Note that, instead of the second arithmetic expression, a second table in which values ​​indicating the depth of field and F-numbers are associated with each other may be used.

[0201] Control unit 48A5 operates aperture 40C by controlling motor 46 via motor driver 60 in accordance with the F-number calculated by acquisition unit 48A1.

[0202] 27 and 28, when the position of the target category subject S is not within the designated image capture range, the aperture 40C is operated so that the subject within the designated image capture range and the target category subject S are contained within the depth of field, and main exposure imaging is performed with the subject within the designated image capture range and the target category subject contained within the depth of field. This makes it possible to obtain high-contrast images as the subject image within the designated image capture range and the target category subject image without performing imaging multiple times, compared to when the subject within the designated image capture range and the target category subject S are not contained within the depth of field.

[0203] However, there may be a situation where both the subject within the designated image capturing range and the target category subject S do not fall within the depth of field due to the structure of the image capturing device 10. When such a situation occurs, the control unit 48A5 causes the image capturing device 10 to capture the subject within the designated image capturing range and the target category subject by the focus bracketing method.

[0204] 29 as an example, first, the acquisition unit 48A1 calculates a first focus position and a second focus position based on phase difference image data corresponding to each position of the subject image within the designated imaging range and the target category subject image S1 in a live view image. The first focus position is a focus position relative to the center of the subject within the designated imaging range, and the second focus position is a focus position relative to the target category subject S. The first and second focus positions are focus positions used when main exposure imaging is performed using the focus bracketing method.

[0205] When the first and second focus positions are calculated by the acquisition unit 48A1, as shown in FIG. 30 as an example, the control unit 48A5 moves the focus lens 40B to the first focus position, and instructs the image sensor to start the main exposure imaging in accordance with the timing when the focus lens 40B reaches the first focus position. In response to this, the main exposure imaging is performed by the image sensor 16. After the main exposure imaging is performed with the focus lens 40B aligned to the first focus position in this way, the control unit 48A5 moves the focus lens 40B to the second focus position, and instructs the image sensor to start the main exposure imaging in accordance with the timing when the focus lens 40B reaches the second focus position. In response to this, the main exposure imaging is performed by the image sensor 16. Note that here, the first and second focus positions are exemplified as the focus positions used in the main exposure imaging in the focus bracketing method, but this is merely an example, and three or more focus positions may be used as the focus positions used in the main exposure imaging in the focus bracketing method.

[0206] 29 and 30, when both the subject within the designated image capture range and the target category subject S do not fall within the depth of field due to the structure of the image capture device 10, main exposure imaging is performed on the subject within the designated image capture range and the target category subject S by the focus bracketing method. As a result, even when both the subject within the designated image capture range and the target category subject S do not fall within the depth of field due to the structure of the image capture device 10, images with high contrast can be obtained as the subject image within the designated image capture range and the target category subject image, compared to a case where only one frame of imaging is performed under a situation where the subject within the designated image capture range and the target category subject S do not fall within the depth of field.

[0207] In the example shown in FIG. 27, the position of the designated imaging range is exemplified as the first imaging condition, and the position of the target category subject S is exemplified as the second imaging condition, but the technology of the present disclosure is not limited thereto. For example, the first imaging condition may be the brightness of the reference subject (e.g., a subject within the designated imaging range), and the second imaging condition may be the brightness of the target category subject S. In this case, when the difference between the brightness of the reference subject and the brightness of the target category subject S (hereinafter also referred to as "brightness difference") is equal to or greater than the default brightness difference, the control unit 48A5 executes the exposure bracketing imaging process as the default process described above. The exposure bracketing imaging process is a process that causes the imaging device 10 to capture images of the reference subject and the target category subject S by the exposure bracketing method.

[0208] Furthermore, when the brightness difference degree is less than the default brightness difference degree, the control unit 48A5 causes the imaging device 10 to capture an image of the reference subject and the target category subject S at an exposure determined based on the reference subject.

[0209] The exposure bracketing imaging process is realized by, for example, the imaging support process shown in Fig. 31 being executed by the CPU 48A. The flowchart shown in Fig. 31 differs from the flowchart shown in Fig. 26 in that it includes steps ST500 to ST508 instead of steps ST450 to ST456.

[0210] 31, the acquisition unit 48A1 acquires a photometric value for each of the subjects within the designated image capturing range and the target category subject S. The photometric value may be calculated based on the live view image data, or may be detected by a photometric sensor (not shown).

[0211] In the next step ST502, the control unit 48A5 calculates a brightness difference using the two photometric values ​​acquired in step ST500 for each of the subject within the designated image capturing range and the target category subject S. The brightness difference is, for example, an absolute value of the difference between the two photometric values.

[0212] In the next step ST504, the control unit 48A5 judges whether or not the brightness disparity calculated in step ST502 is equal to or greater than a default brightness disparity. The default brightness disparity may be a fixed value, or may be a variable value that is changed according to a given instruction and / or a given condition. In step ST504, if the brightness disparity is less than the default brightness disparity, the judgment is negative, and the imaging support process proceeds to step ST508. In step ST504, if the brightness disparity is equal to or greater than the default brightness disparity, the judgment is positive, and the imaging support process proceeds to step ST506.

[0213] In step ST506, the control unit 48A5 controls the imaging device 10 to perform main exposure imaging by the exposure bracketing method for each of the subjects within the designated imaging range and the target category subjects S. After the processing of step ST506 is performed, the imaging support processing proceeds to step ST122. Note that the main exposure image data of each frame obtained by performing main exposure imaging by the exposure bracketing method may be stored individually in a default storage area, or may be stored in a default storage area as composite image data of one frame obtained by combining the data.

[0214] In step ST508, control unit 48A5 controls imaging device 10 to perform main exposure imaging of subjects within the designated imaging range and target category subject S with exposure determined based on subjects within the designated imaging range. After the process of step ST508 is performed, control unit 48A5 performs processes equivalent to steps ST120 to ST132 (see FIG. 25) and then transitions to step ST100.

[0215] 31, when the brightness difference is equal to or greater than a preset brightness difference, main exposure imaging is performed on the subject within the designated capturing range and the target category subject S using the exposure bracketing method. This makes it possible to obtain images with less uneven brightness as the subject image within the designated capturing range and the target category subject image S1, compared to a case where only one frame of imaging is performed under conditions where there is uneven brightness between the subject within the designated capturing range and the target category subject S.

[0216] 31, when the brightness difference is less than a predetermined brightness difference, main exposure imaging is performed on the subjects within the designated imaging range and the target category subject S with an exposure determined based on the reference subject. This reduces the effort required to eliminate unevenness in brightness between the subjects within the designated imaging range and the target category subject S and then perform main exposure imaging on the subjects within the designated imaging range and the target category subject S.

[0217] In the above embodiment, the face category, posture category, eye category, and designated image capture range category are given as examples of major categories included in the subject-specific category group 98, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 32, the subject-specific category group 98 may include, as major categories, a period category, which is a category defined by subject features in units of "period", and / or a position category, which is a category defined by subject features in units of "position". Here, the period refers to the period during which the subject was imaged. Also, the position refers to the position at which the subject was imaged.

[0218] The period category includes, as subcategories, a plurality of date categories, each of which has a different date. A classification count is associated with each of the plurality of date categories. A classification count is also associated with the period category. The classification count of the period category is the sum of the classification counts of the plurality of date categories.

[0219] The location category includes a plurality of small location categories as small categories, each of which has a different location. A classification count is associated with each of the plurality of small location categories. A classification count is also associated with each location category. The classification count of a location category is the sum of the classification counts of the plurality of small location categories.

[0220] In the example shown in Fig. 32, map data 104 is stored in storage 48B. The map data 104 is data in which location coordinates, such as latitude, longitude, and altitude, are associated with addresses on a map. The map data 104 is referenced by classification unit 48A4. Note that the sub-location category is not limited to addresses on a map, and may be location coordinates.

[0221] 32, the classification unit 48A4 is connected to the RTC 106 and a GPS receiver 108, which is an example of a GNSS receiver. The RTC 106 acquires the current time. The RTC 106 receives a supply of driving power from a power supply system that is separated from the power supply system for the controller 48, and continues to keep track of the current time (year, month, day, hour, minute, and second) even when the controller 48 is shut down.

[0222] The GPS receiver 108 receives radio waves from a plurality of GPS satellites (not shown), which are an example of a plurality of GNSS satellites, and calculates position coordinates that enable identification of the current position of the imaging device 10 based on the reception results.

[0223] The classification unit 48A4 acquires the current time from the RTC 106 each time one frame of main exposure imaging is performed, and classifies the acquired current time as the imaging time into a corresponding date category among a plurality of date categories included in the period category. Each time the classification unit 48A4 classifies the imaging time into a date category, it adds "1" to the classification count of the date category into which the imaging time was classified. Note that, although the classification unit 48A4 acquires the current time from the RTC 106 here, the current time may be acquired via a communication network such as the Internet.

[0224] Each time main exposure imaging for one frame is performed, the classification unit 48A4 acquires position coordinates from the GPS receiver 108 as the position where imaging was performed (hereinafter also referred to as "imaging position"). The classification unit 48A4 identifies an address corresponding to the acquired imaging position from the map data 104. Then, the classification unit 48A4 classifies the imaging position into a small position category corresponding to the identified address. Each time the classification unit 48A4 classifies the imaging position into a small position category, it adds "1" to the classification count of the small position category into which the imaging position was classified.

[0225] 33, a bubble chart related to a period category (hereinafter also referred to as a "period category bubble chart") is displayed as a bubble chart 100A within the imaging support screen 100. In the period category bubble chart, similar to the face category bubble chart (see FIG. 17), bubbles indicating the number of classifications are plotted on two axes, an axis indicating a plurality of recognized persons and an axis indicating a date category.

[0226] Furthermore, when a location category is selected on category selection screen 100B of reception device 80 (for example, touch panel 28), a bubble chart related to the location category (hereinafter, location category bubble chart) is displayed as bubble chart 100A on imaging support screen 100. In the location category bubble chart, similar to the bubble chart related to the face category (see FIG. 17), bubbles indicating the number of classifications are plotted on two axes, an axis indicating multiple recognized persons and an axis indicating small location categories.

[0227] In the example shown in FIG. 34, a histogram related to a period category (hereinafter also referred to as a "period category histogram") is displayed as a histogram 100C in the imaging support screen 100. The horizontal axis of the period category histogram indicates the year / month / day category, and the vertical axis of the period category histogram indicates the number of classifications. In addition, a histogram related to a position category (hereinafter also referred to as a "position category histogram") is also displayed as a histogram 100C in the imaging support screen 100 so as to be switchable with histograms of other categories (not shown). The horizontal axis of the position category histogram indicates the small position category, and the vertical axis of the position category histogram indicates the number of classifications.

[0228] In this way, the subject-specific category group 98 includes a period category defined by subject characteristics in units of "period" as a major category. The period category also includes a plurality of date categories with different dates. Then, every time one frame of main exposure imaging is performed, the imaging time is classified into a date category, and a period category bubble chart and a period category histogram according to the number of classifications are displayed on the display 26 together with the live view image. The period category bubble chart and the period category histogram are used in the same manner as the face category bubble chart and the face category histogram described in the above embodiment. Therefore, according to this configuration, imaging by the imaging device 10 can be supported according to the number of classifications counted by classifying the imaging time into the date category. Note that here, the date category divided by date is exemplified, but this is merely an example, and the category may be a period divided by year, month, day, hour, minute, or second.

[0229] Moreover, the subject-specific category group 98 includes a position category defined by subject features in units of "position" as a large category. Moreover, the position category includes a plurality of small position categories having different positions from each other. Then, every time one frame of main exposure imaging is performed, the imaging position is classified into a small position category, and a position category bubble chart and a position category histogram according to the number of classifications are displayed on the display 26 together with the live view image. The position category bubble chart and the position category histogram are used in the same manner as the face category bubble chart and the face category histogram described in the above embodiment. Therefore, according to this configuration, imaging by the imaging device 10 can be supported according to the number of classifications counted by classifying the imaging position into a small position category.

[0230] In the above embodiment, an example of the embodiment is given assuming that the imaging support process is continuously executed while the imaging mode is set, but the technology of the present disclosure is not limited to this, and the imaging support process may be intermittently executed according to the time and / or the position. For example, as shown in FIG. 35, the imaging support process may be executed only at a specified time (for example, 10 minutes) at a time checkpoint separated by a preset time (for example, 1 hour) interval. The preset time interval that determines the time checkpoint may be fixed, or may be changed according to a given instruction and / or a given condition (for example, imaging condition). Also, as an example, as shown in FIG. 35, the imaging support process may be executed only at a specified time at a plurality of position checkpoints separated by position. The position checkpoints can be specified by using, for example, the map data 104 and the GPS receiver 108.

[0231] In the above embodiment, the classification unit 48A4 classifies the subject features into categories regardless of the imaging scene captured by the imaging device 10, but the technology of the present disclosure is not limited to this. For example, the classification unit 48A4 may classify the subject features into categories when the scene to be imaged by the imaging device 10 matches a specific scene (for example, an athletic meet scene, a beach scene, a concert scene, etc.). The specific scene may be a scene captured in the past.

[0232] In this case, as an example, the CPU 48A executes the imaging support process shown in Fig. 36. The flowchart shown in Fig. 36 differs from the flowchart shown in Fig. 23 in that step ST550 is provided between step ST118 and step ST120.

[0233] In step ST550 of the imaging support process shown in FIG. 36, the subject recognition unit 48A2 identifies a current imaging scene by recognizing a subject in an imaging area based on the latest main exposure image data obtained by performing main exposure imaging in step ST118. The subject recognition unit 48A2 also identifies a past imaging scene based on past main exposure imaging image data (for example, main exposure image data obtained within a period designated by a user). Then, the subject recognition unit 48A2 determines whether or not the current imaging scene matches the past imaging scene. In step ST550, if the current imaging scene does not match the past imaging scene, the determination is negative, and the imaging support process proceeds to step ST130. In step ST550, if the current imaging scene matches the past imaging scene, the determination is positive, and the imaging support process proceeds to step ST120. As a result, in step ST126, the subject features are classified into categories for each subject by the classification unit 48A4.

[0234] In this way, the classification unit 48A4 classifies the subject features for each subject into a category only when the current imaging scene and the specific scene match, so that it is possible to prevent subject features identified from the main exposure image data obtained by performing main exposure imaging on a scene not intended by the user from being classified into a category.

[0235] In addition, only when the current imaging scene matches a past imaging scene, the classification unit 48A4 classifies the subject features into categories for each subject, so that it is possible to classify the subject features identified from the main exposure image data obtained by performing main exposure imaging on a current imaging scene that matches a past imaging scene into categories.

[0236] In the above embodiment, an example in which subject features are classified into each of a plurality of categories has been described, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 37, a final exposure image obtained by performing final exposure imaging may be classified into each of a plurality of categories by the classification unit 48A4. In this case, each time a final exposure image is classified into a category, "1" is added to the classification count of the category into which the final exposure image is classified. The subject-specific category group 98 thus constructed is also used in the same manner as the subject-specific category group 98 described in the above embodiment. Therefore, according to this configuration, imaging by the imaging device 10 can be supported according to the classification count into which the final exposure image is classified into a category.

[0237] In the above embodiment, the face category histogram (see FIG. 18) for person A is illustrated, but the technology of the present disclosure is not limited thereto. For example, as shown in FIG. 38, a four-quadrant face category bubble chart may be used instead of the face category histogram. In the four-quadrant face category bubble chart, the smiling face category is assigned to the first quadrant, the crying face category is assigned to the second quadrant, the angry face category is assigned to the third quadrant, and the serious face category is assigned to the fourth quadrant. The classification frequency corresponding to each category is expressed by the size of the bubble. In addition, in the case of displaying in four quadrant categories as shown in FIG. 38, the facial expressions may be classified more finely, and a scatter diagram may be displayed instead of the bubble chart. In this case, subtle facial expressions can be expressed by adjusting the plot position of the points based on the classified facial expressions. For example, even in the same smiling face category, the image with a smiling face closer to a crying face is plotted to the left in the first quadrant in the figure. In addition, the image with a smiling face closer to a serious face is plotted to the lower side in the first quadrant in the figure. Such a scatter diagram allows the user to understand in more detail what kind of face of the person has been captured in the image.

[0238] In addition, in the above embodiment, bubble chart 100A and histogram 100C are exemplified, but the technology of the present disclosure is not limited to this and other graphs may be used, and the numerical values ​​indicating the number of classifications may be displayed in a form divided for each subject and for each category.

[0239] In addition, in the above embodiment, an example was given in which imaging support was provided for a category corresponding to the classification count selected by the user from the histogram 100C, but the technology disclosed herein is not limited to this, and the category for which imaging support is provided may be directly selected by the user from the histogram 100C, etc., via the reception device 80 (e.g., the touch panel 28).

[0240] Furthermore, in the above embodiment, an example has been described in which the final exposure image data is stored in the image memory 50, but data including the final exposure image data obtained by performing the final exposure imaging supported by carrying out the above-mentioned support process may be used as training data for machine learning of the trained model 92. This makes it possible to create a trained model 92 based on the final exposure image data obtained by performing the final exposure imaging supported by carrying out the support process.

[0241] In the above embodiment, an example in which the imaging support process is executed by the controller 48 in the imaging device 10 has been described, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 40, the imaging support process may be executed by a computer 114 in an external device 112 communicably connected to the imaging device 10 via a network 110 such as a LAN or a WAN. In the example shown in Fig. 40, the computer 114 includes a CPU 116, a storage 118, and a memory 120. A category database 96 is constructed in the storage 118, and an imaging support process program 84 is stored therein.

[0242] The imaging device 10 requests the external device 112 to execute imaging support processing via the network 110. In response to this, the CPU 116 of the external device 112 reads the imaging support processing program 84 from the storage 118, and executes the imaging support processing program 84 on the memory 120. The CPU 116 performs imaging support processing in accordance with the imaging support processing program 84 executed on the memory. Then, the CPU 116 provides the imaging device 10 via the network 110 with a processing result obtained by executing the imaging support processing.

[0243] Furthermore, the imaging support process may be distributed between the imaging device 10 and the external device 112, or may be distributed between a plurality of devices including the imaging device 10 and the external device 112. When the distributed process is performed, for example, the CPU 48A of the imaging device 10 may be operated as the acquisition unit 48A1 and the control unit 48A5, and a CPU of a device other than the imaging device 10 (for example, the external device 112) may be operated as the object recognition unit 48A2, the feature extraction unit 48A3, and the classification unit 48A4. In other words, the processing load on the imaging device 10 may be reduced by having an external device with a higher computing power than the imaging device 10 take on a process with a relatively large processing load.

[0244] In the above embodiment, a still image is exemplified as the main exposure image, but the technology of the present disclosure is not limited to this, and a moving image may be used as the main exposure image. The moving image may be a moving image for recording or a moving image for display, that is, a live view image or a post view image.

[0245] In addition, in the above embodiment, the recommended imaging information is displayed within the live view image, but it does not necessarily have to be displayed within the live view image. For example, when a moving image is being captured with a main exposure, a display indicating the target category subject (at least one of an arrow, a face frame, and a message) may be displayed similarly to the recommended imaging information. By displaying the target category subject while a moving image is being captured with a main exposure in this manner, the user can use the imaging device 10 to recognize that the moving image contains a target category subject. Also, the user can use the imaging device 10 to switch frames containing target category subjects after the moving image is captured with a main exposure. By extracting the still images, it is possible to obtain still images including the target category object. In this case, the imaging device 10 may classify the still images into categories similar to those described above each time the target category object is identified by the main exposure imaging of the moving image, and update and display the histogram and / or bubble chart. In this way, the user can grasp what kind of object is included in the moving image by simply capturing the moving image using the imaging device 10. In this case, the histogram and / or bubble chart may display values ​​based on the still images and values ​​based on the moving image in different modes. For example, in the histogram, the histogram based on the still images and the histogram based on the moving image are displayed in different colors using a stacked bar graph. In this way, the user can grasp whether each classification count is based on the still images or the moving image. In addition, the histogram and / or bubble chart may be created based only on the classification of the object indicated by the object area included in one moving image. In this way, the user can grasp what category the object area classified into is included in one moving image. In addition, such histograms and / or bubble charts may be displayed on the display 26 based on user operations, etc., after capturing still or moving images, or during playback mode when live view images are not being displayed.

[0246] In the above embodiment, the classification count continues to increase, but the technology of the present disclosure is not limited to this. For example, the classification count corresponding to at least one category included in the subject-specific category group 98 may be reset periodically or at a specified timing. For example, it may be reset according to time and / or position. Specifically, it may be reset once a day, once an hour, or every time the position changes by 100 meters.

[0247] In addition, in the above embodiment, the smiling face category is exemplified as the target category, but the technology of the present disclosure is not limited to this, and other categories may be the target category, or multiple categories may be the target categories. In this case, for example, in the face category histogram shown in Fig. 18, the number of classifications for multiple categories (for example, the smiling face category and the crying face category) or multiple categories may be selected by the user.

[0248] Furthermore, in the above embodiment, person A is exemplified as the target category subject, but the technology of the present disclosure is not limited to this, and there may be multiple target category subjects. In this case, for example, multiple people (e.g., person A, person B, and person C) may be selected by the user in the face category bubble chart shown in FIG. 17. As a result, for example, if the target category is a smile category, the various support processes described above are performed in a scene where at least one of the multiple people selected by the user is smiling.

[0249] In the above embodiment, the classification count, which is the simple number of times that the subject features are classified into categories, is exemplified, but the technology of the present disclosure is not limited to this. For example, the classification count per unit time may be used.

[0250] In addition, in the above embodiment, a detection frame 102 (see Figures 19 and 21) is exemplified, but the technology of the present disclosure is not limited to this, and information about the subject surrounded by the detection frame 102 (e.g., name and / or category name, etc.) may also be displayed together with the detection frame 102.

[0251] In addition, in the above embodiment, a physical camera (hereinafter also referred to as a "physical camera") is exemplified as the imaging device 10, but the technology of the present disclosure is not limited to this, and instead of a physical camera, a virtual camera that generates virtual viewpoint image data by virtually capturing an image of a subject from a virtual viewpoint based on captured image data obtained by capturing images using a plurality of physical cameras set at different positions may be applied. In this case, the image shown by the virtual viewpoint image data, i.e., the virtual viewpoint image, is an example of a "captured image" according to the technology of the present disclosure.

[0252] In the above embodiment, the non-phase difference pixel division region 30N and the phase difference pixel division region 30P are used in combination, but the technology of the present disclosure is not limited thereto. For example, instead of the non-phase difference pixel division region 30N and the phase difference pixel division region 30P, an area sensor in which phase difference image data and non-phase difference image data are selectively generated and read may be used. In this case, a plurality of photosensitive pixels are two-dimensionally arranged in the area sensor. For example, a pair of independent photodiodes without a light shielding member is used as the photosensitive pixel included in the area sensor. When non-phase difference image data is generated and read, photoelectric conversion is performed by the entire region (pair of photodiodes) of the photosensitive pixel, and when phase difference image data is generated and read (for example, when performing distance measurement by a passive method), photoelectric conversion is performed by one of the pair of photodiodes. Here, one of the pair of photodiodes is a photodiode corresponding to the first phase difference pixel L described in the above embodiment, and one of the pair of photodiodes is a photodiode corresponding to the second phase difference pixel R described in the above embodiment. In addition, the phase difference image data and the non-phase difference image data may be selectively generated and read out by all of the photosensitive pixels included in the area sensor, but this is not limited to the above, and the phase difference image data and the non-phase difference image data may be selectively generated and read out by a portion of the photosensitive pixels included in the area sensor.

[0253] In the above embodiment, the image plane phase difference pixel is exemplified as the phase difference pixel P, but the technology of the present disclosure is not limited thereto. For example, a non-phase difference pixel N may be arranged in place of the phase difference pixel P included in the photoelectric conversion element 30, and a phase difference AF plate including a plurality of phase difference pixels P may be provided in the imaging device body 12 separately from the photoelectric conversion element 30.

[0254] In the above embodiment, an AF method using a distance measurement result based on phase difference image data, i.e., a phase difference AF method, is exemplified, but the technology of the present disclosure is not limited to this. For example, instead of the phase difference AF method, a contrast AF method may be adopted. Also, an AF method based on a distance measurement result using the parallax of a pair of images obtained from a stereo camera, or an AF method using a distance measurement result of a TOF method using laser light or the like may be adopted.

[0255] In the above embodiment, a focal plane shutter was described as an example of mechanical shutter 72, but the technology of the present disclosure is not limited to this, and the technology of the present disclosure can also be applied by applying other types of mechanical shutters, such as a lens shutter, instead of a focal plane shutter.

[0256] In the above embodiment, an example in which the imaging support processing program 84 is stored in the storage 48B has been described, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 41, the imaging support processing program 84 may be stored in a storage medium 200. The storage medium 200 is a non-transitory storage medium. An example of the storage medium 200 is any portable storage medium such as an SSD or a USB memory.

[0257] The imaging support processing program 84 stored in the storage medium 200 is installed in the controller 48. The CPU 48A executes the imaging support processing in accordance with the imaging support processing program 84.

[0258] In addition, the image capture support processing program 84 may be stored in a memory unit of another computer or server device connected to the controller 48 via a communication network (not shown), and the image capture support processing program 84 may be downloaded and installed in the controller 48 in response to a request from the image capture device 10.

[0259] It is not necessary to store the entire image capture support processing program 84 in the memory unit of another computer or server device connected to the controller 48, or in the storage 48B; only a portion of the image capture support processing program 84 may be stored.

[0260] In the example shown in FIG. 41, an example is shown in which the controller 48 is built into the imaging device 10, but the technology of the present disclosure is not limited to this, and for example, the controller 48 may be provided outside the imaging device 10.

[0261] 41, the CPU 48A is a single CPU, but it may be a plurality of CPUs. Also, a GPU may be applied in place of the CPU 48A.

[0262] 41 illustrates a controller 48, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the controller 48. Also, instead of the controller 48, a combination of a hardware configuration and a software configuration may be used.

[0263] As the hardware resource for executing the imaging support processing described in the above embodiment, the following various processors can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the imaging support processing by executing software, i.e., a program. Another example of the processor is a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC. Each processor has a built-in or connected memory, and each processor executes the imaging support processing by using the memory.

[0264] The hardware resource for executing the imaging support process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource for executing the imaging support process may be a single processor.

[0265] As an example of a configuration using one processor, first, one or more CPUs and software are combined to configure one processor, and this processor functions as a hardware resource that executes the imaging support process. Second, as typified by SoC, a processor is used that realizes the functions of the entire system including multiple hardware resources that execute the imaging support process in one IC chip. In this way, the imaging support process is realized using one or more of the above various processors as hardware resources.

[0266] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. Also, the above-mentioned imaging support process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed, without departing from the spirit of the invention.

[0267] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0268] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0269] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0270] The following supplementary notes are further disclosed regarding the above embodiment.

[0271] (Appendix 1) A processor; A memory connected to or embedded in the processor, The processor acquires frequency information indicating a frequency of the features classified into categories based on features of a subject identified from a captured image obtained by capturing an image using an imaging device; A support process for supporting imaging by the imaging device is performed based on the frequency information. Imaging support device.

[0272] (Appendix 2) The categories are classified into a plurality of categories including at least one target category; the target category is a category determined based on the frequency information, The imaging support device according to claim 1, wherein the support process includes a process of supporting the imaging of a target category subject having the characteristic belonging to the target category.

[0273] (Appendix 3) 3. The imaging support device according to claim 2, wherein the support process includes a display process for displaying a recommendation to capture an image of the target category subject.

[0274] (Appendix 4) An imaging support device as described in Appendix 3, wherein the display process is a process of displaying a display image on the display and displaying a frame surrounding at least a portion of a target category subject image within the display image.

[0275] (Appendix 5) The processor is Detecting the target category subject based on an image capturing result by the imaging device; An imaging support device according to any one of Supplementary Note 2 to Supplementary Note 4, which acquires images including images corresponding to the target category subject on condition that the target category subject is detected.

[0276] (Appendix 6) The imaging support device described in Appendix 5, wherein the processor detects the target category subject and, on condition that the target category subject is included in a default imaging range, causes the imaging device to perform the imaging with a main exposure.

[0277] (Appendix 7) An imaging support device as described in any one of Appendix 2 to Appendix 6, wherein the processor controls the imaging device to include the designated imaging range and the target category subject within a depth of field when the target category subject is located outside a designated imaging range determined in accordance with instructions given from outside.

[0278] (Appendix 8) The imaging support device described in Appendix 7, wherein the processor causes the imaging device to capture the designated imaging range and the target category subject using a focus bracketing method when both the designated imaging range and the target category subject cannot fit within the depth of field due to a structure of the imaging device.

[0279] (Appendix 9) An imaging support device as described in Appendix 7 or Appendix 8, wherein the processor causes the imaging device to capture an image of the target category subject with the target category subject in focus when the target category subject is located within the designated imaging range.

[0280] (Appendix 10) An imaging support device as described in any one of Appendix 2 to Appendix 9, wherein the processor causes the imaging device to capture images of the reference subject and the target category subject using an exposure bracketing method when the difference in brightness between the reference subject and the target category subject is equal to or greater than a predetermined difference.

[0281] (Appendix 11) The imaging support device described in Appendix 10, wherein the processor causes the imaging device to capture the reference subject and the target category subject at an exposure determined based on the target category subject when the dissimilarity is less than the default dissimilarity.

[0282] (Appendix 12) The imaging support device according to any one of Supplementary Note 1 to Supplementary Note 11, wherein an image obtained by performing the imaging supported by performing the support process is used for learning.

Claims

1. A processor; A memory connected to or embedded in the processor, The processor acquires frequency information indicating a frequency of the features classified into categories based on features of a subject identified from a captured image obtained by capturing an image using an imaging device; A support process for supporting imaging by the imaging device is performed based on the frequency information. Imaging support device.

2. The categories are categorized into a plurality of categories including at least one target category; the target category is a category determined based on the frequency information, The imaging support device according to claim 1 , wherein the support process includes a process of supporting the imaging of a target category subject having the characteristic belonging to the target category.

3. The imaging support device according to claim 2 , wherein the support process includes a display process for displaying a recommendation to capture an image of the target category subject.

4. The imaging support device of claim 3, wherein the display processing is a processing for displaying a display image obtained by imaging by the imaging device on a display, and displaying a target category subject image showing the target category subject within the display image in a manner distinguishable from other image areas.

5. The processor, Detecting the target category subject based on an image capturing result by the imaging device; The imaging support device according to claim 2 , further comprising: an image capturing device configured to capture an image including an image corresponding to a target category subject on condition that the target category subject has been detected.

6. The processor outputs an image of a specified imaging range determined according to an external instruction.

6. The imaging support device according to claim 2, wherein the object representing the target category subject and the object representing the target category subject are displayed in different display modes.

7. The imaging support device according to any one of claims 2 to 6, wherein the processor performs a predetermined processing when the degree of difference between a first imaging condition given from outside and a second imaging condition given for the target category subject is equal to or greater than a predetermined degree of difference.

8. The imaging support device according to claim 2 , wherein the target category is a low-frequency category having a relatively low frequency among the plurality of categories.

9. An imaging support device as described in any one of claims 2 to 8, wherein when the target category subject is imaged by the imaging device, the target category is a category determined according to the state of the target category subject, and is a category into which the characteristics of the target category subject are classified.

10. The imaging support device according to claim 2 , wherein when a plurality of objects are imaged by the imaging device, the object category is an object object category that can identify each of the plurality of objects itself.

11. The imaging support device according to claim 1 , wherein the categories are created for at least one unit.

12. The imaging support device according to claim 11 , wherein one of the units is a period.

13. The imaging support device according to claim 11 or 12, wherein one of the units is a position.

14. The processor causes a classifier to classify the features; The imaging support device according to claim 1 , wherein the classifier classifies the feature when a scene to be imaged by the imaging device matches a specific scene.

15. The imaging support device according to claim 14 , wherein the specific scene is a scene that was imaged in the past.

16. The imaging support device according to claim 1 , wherein the support process includes a process of displaying the frequency information.

17. The imaging support device of claim 16, wherein the support processing includes, when the frequency information is displayed and the frequency information is specified by a reception device, a processing for supporting the imaging related to the category corresponding to the specified frequency information.

18. A processor; A memory connected to or embedded in the processor, The processor acquires frequency information indicating a frequency of captured images classified into categories based on features of a subject included in the captured images obtained by capturing images using an imaging device; A support process for supporting imaging by the imaging device is performed based on the frequency information. Imaging support device.

19. An imaging support device according to any one of claims 1 to 18; An image sensor, The processor performs the support processing to capture the image using the image sensor. Support Imaging device.

20. acquiring frequency information indicating a frequency of the features classified into categories based on the features of a subject identified from a captured image obtained by capturing an image with an imaging device; and and performing a support process for supporting imaging by the imaging device based on the frequency information. Imaging assistance method.

21. Acquiring frequency information indicating a frequency of captured images classified into categories based on characteristics of a subject identified from the captured images obtained by capturing images with an imaging device; and and performing a support process for supporting imaging by the imaging device based on the frequency information. Imaging assistance method.

22. On the computer, acquiring frequency information indicating a frequency of the features classified into categories based on the features of a subject identified from a captured image obtained by capturing an image with an imaging device; and A program for executing a process including performing an assistance process for assisting the imaging device in imaging based on the frequency information.

23. On the computer, Acquiring frequency information indicating a frequency of captured images classified into categories based on characteristics of a subject identified from the captured images obtained by capturing images with an imaging device; and A program for executing a process including performing an assistance process for assisting the imaging device in imaging based on the frequency information.

Images