Imaging support apparatus, imaging apparatus, imaging support method, and storage medium

Abstract

The present application relates to an image pickup support device, an image pickup device, an image pickup support method, and a storage medium. The image pickup support device includes a processor and a memory connected to or built in the processor, and the processor acquires frequency information indicating a frequency at which a feature included in a subject classified into a category based on a feature included in an image pickup image obtained by image pickup by the image pickup device, and performs support processing for supporting image pickup by the image pickup device based on the frequency information.

Description

Technical Field

[0001] The present invention relates to a camera support device, a camera device, a camera support method, and a storage medium. Background Technology

[0002] Japanese Patent Application Publication No. 2007-006033 discloses an object determination apparatus that selects a face as a processing object from a plurality of faces included in an image. The object determination apparatus described in Japanese Patent Application Publication No. 2007-006033 includes: a face detection member that detects faces from an image; a face information recording member that records the faces previously detected by the face detection member in association with the detection history related to that detection; and a face selection member that selects a face as a processing object from the faces included in the image based on the detection history.

[0003] Japanese Patent Application Publication No. 2009-252069 discloses an image processing apparatus characterized by comprising a face recognition dictionary, an image acquisition component, a face region detection component, a feature extraction component, a discrimination component, a face recognition dictionary correction component, and a classification component. In the face recognition dictionary, facial features for determining whether individuals belong to the same person are registered for each person. The image acquisition component acquires an image including individuals. The face region detection component detects face regions from the image acquired by the image acquisition component. The feature extraction component extracts facial features from the face regions detected by the face region detection component. The discrimination component determines whether facial features of the same person are registered in the face recognition dictionary based on the facial features extracted by the feature extraction component and the facial features registered in the face recognition dictionary. If facial features identified as belonging to the same person by the discrimination component are registered in the facial recognition dictionary, the facial recognition dictionary correction component corrects the registered facial features based on the extracted facial features. If facial features identified as belonging to the same person by the discrimination component are not registered in the facial recognition dictionary, the extracted facial features are registered as facial features of a new person. If facial features identified as belonging to the same person by the discrimination component are registered in the facial recognition dictionary, the classification component classifies the facial features of the person in the image acquired by the image acquisition component as known faces. If facial features identified as belonging to the same person by the discrimination component are not registered in the facial recognition dictionary, the facial features of the person in the image acquired by the image acquisition component are classified as unknown faces.

[0004] Japanese Patent Application Publication No. 2012-099943 discloses an image processing apparatus characterized by comprising: a storage member for storing face image data; a face detection member for detecting faces from an image signal; a face recognition member for determining whether the face detected by the face detection member is included in the face image data stored in the storage member; and an image processing member. When faces determined by the face recognition member to be included in the face image data stored in the storage member and faces determined not to be included in the face image data are detected from the image signal, the image processing member performs image processing on the areas of the faces determined to be included in the face image data and the areas of the faces determined not to be included in the face image data, applying higher image quality processing to these areas than to other areas.

[0005] Japanese Patent Application Publication No. 2009-003012 discloses a camera device characterized by comprising: a camera lens, capable of driving at least a portion of a plurality of lenses arranged along the optical axis and changing the focal position of the lens; an image acquisition member; a subject detection member; a focus evaluation value calculation member; a selection member; and a recording member. The image acquisition member continuously performs image capture and acquires multiple image data while changing the focal position of the camera lens. The subject detection member detects a main subject area in accordance with the movement state of the subject between the multiple image data obtained by the image acquisition member. The focus evaluation value calculation member calculates a focus evaluation value for the main subject area obtained by the subject detection member for each of the multiple image data. The selection member selects at least one image data from the multiple image data based on the focus evaluation value obtained by the focus evaluation value calculation member. The recording member records the image data selected by the selection member into a recording medium. Summary of the Invention

[0006] One embodiment of the present invention provides a camera support device, camera device, camera support method, and storage medium capable of supporting the shooting of a camera device based on the frequency at which the characteristics of the subject are classified into categories.

[0007] means for solving technical problems

[0008] The first aspect of the technology of the present invention is a camera support device, which includes: a processor; and a memory connected to or built into the processor, wherein the processor acquires frequency information and performs support processing for shooting by the camera device based on the frequency information, the frequency information representing the frequency of features of a subject determined from a camera image obtained by the camera device and classified into categories.

[0009] The second aspect of the present invention is the camera support device involved in the first aspect, wherein the category is classified into multiple categories including at least one object category, the object category is determined based on frequency information, and the support process includes a process of supporting the shooting of a subject of an object category that has characteristics belonging to the object category.

[0010] The third aspect of the present invention is the camera support device involved in the second aspect, wherein the support processing includes a display processing that displays a recommended subject category.

[0011] The fourth aspect of the present invention is the camera support device involved in the third aspect, wherein the display processing is as follows: displaying a display image obtained by the camera device on a display screen, and displaying an image of an object category subject representing an object category subject within the display image in a manner distinguishable from other image areas.

[0012] The fifth aspect of the present invention is a camera support device involved in any of the second to fourth aspects, wherein the processor detects an object category subject based on the imaging results of the camera device, and when an object category subject is detected, acquires an image including an image corresponding to the object category subject.

[0013] The sixth aspect of the technology of the present invention is a camera support device involved in any of the second to fifth aspects, wherein the processor displays, in different display modes, objects representing a specified camera range determined according to instructions given from the outside and objects representing the object category of the subject.

[0014] The seventh aspect of the technology of the present invention is a camera support device involved in any one of the second to sixth aspects, wherein, when the difference between the first camera condition imposed from the outside and the second camera condition imposed on the subject of the object category is greater than or equal to a predetermined difference, the processor performs predetermined processing.

[0015] The eighth aspect of the technology of the present invention is a camera support device involved in any of the second to seventh aspects, wherein the object category is a low-frequency category with relatively low frequency among multiple categories.

[0016] The ninth aspect of the technology of the present invention is a camera support device involved in any of the second to eighth aspects, wherein, when an object category subject is captured by the camera device, the object category is a category determined according to the state of the object category subject, and is a category classified with respect to the characteristics of the object category subject.

[0017] The tenth aspect of the present invention is a camera support device involved in any of the second to ninth aspects, wherein, when multiple objects are photographed by the camera device, the object category is an object category that can be determined for each of the multiple objects.

[0018] The eleventh aspect of the technology of the present invention is a camera support device involved in any of the first to tenth aspects, wherein the categories are created at least per unit.

[0019] The 12th aspect of the present invention is the camera support device according to the 11th aspect, wherein one of the units is period.

[0020] The 13th aspect of the present invention is the camera support device involved in the 11th or 12th aspects, wherein one of the units is position.

[0021] The 14th aspect of the technology of the present invention is a camera support device involved in any of the 1st to 13th aspects, wherein the processor causes the classifier to classify features, and the classifier classifies features when the scene of the camera object of the camera device is consistent with a specific scene.

[0022] The 15th aspect of the present invention is the camera support device involved in the 14th aspect, wherein the specific scene is a scene captured in the past.

[0023] The 16th aspect of the present invention is a camera support device involved in any one of the 1st to 15th aspects, wherein the support process is the following process: including the processing of display frequency information.

[0024] The 17th aspect of the present invention is the camera support device involved in the 16th aspect, wherein the support processing includes processing that supports shooting related to the category corresponding to the specified frequency information when frequency information is specified by the receiving device in a state where frequency information is displayed.

[0025] The 18th aspect of the present invention is a camera support device comprising: a processor; and a memory connected to or embedded therein, wherein the processor acquires frequency information and performs support processing for shooting by the camera device based on the frequency information, the frequency information representing the frequency of camera images classified into categories based on the features of the subject included in the camera images obtained by the camera device.

[0026] The 19th aspect of the present invention is a camera device comprising: a camera support device according to any one of the 1st to 18th aspects; and an image sensor, wherein a processor supports shooting using the image sensor by performing support processing.

[0027] The 20th aspect of the present invention is a camera support method, which includes the following processing: acquiring frequency information, the frequency information representing the frequency of features of a subject determined from a camera image captured by a camera device that are classified into categories; and performing support processing to support the camera device's shooting based on the frequency information.

[0028] The 21st aspect of the present invention is a camera support method, which includes the following processing: acquiring frequency information, the frequency information representing the frequency of camera images classified into categories based on features of a subject determined from camera images captured by a camera device; and performing support processing to support the camera device's shooting based on the frequency information.

[0029] The 22nd aspect of the present invention is a program for causing a computer to perform the following processing: acquiring frequency information, the frequency information representing the frequency of features of a subject classified into categories based on features determined from a photographic image obtained by a camera device; and performing support processing for the camera device to capture images based on the frequency information.

[0030] The 23rd aspect of the present invention is a program for causing a computer to perform the following processing: acquiring frequency information, the frequency information representing the frequency of video images classified into categories based on features of a subject determined from video images captured by a camera device; and performing support processing for capturing images by the camera device based on the frequency information. Attached Figure Description

[0031] Figure 1 This is a perspective view showing an example of the appearance of a camera device.

[0032] Figure 2 It means Figure 1 A rear view of an example of the rear side appearance of the camera device shown.

[0033] Figure 3 This is a schematic diagram illustrating an example of the configuration of pixels included in the photoelectric conversion element of a camera device.

[0034] Figure 4 It means relative to Figure 3 A conceptual diagram illustrating an example of the incident characteristics of subject light in the first phase difference pixel and the second phase difference pixel included in the photoelectric conversion element shown.

[0035] Figure 5 It means Figure 3 A schematic structural diagram of an example of the structure of a non-phase difference pixel included in the photoelectric conversion element shown.

[0036] Figure 6 This is a schematic diagram illustrating an example of the hardware structure of a camera device.

[0037] Figure 7 This is a block diagram illustrating an example of the structure of a controller included in a camera device.

[0038] Figure 8 This is a block diagram illustrating an example of the main functions of the CPU included in a camera device.

[0039] Figure 9 This is a conceptual diagram illustrating an example of the processing performed by the CPU as both the acquisition and control unit.

[0040] Figure 10 This is a conceptual diagram illustrating an example of the processing performed by the CPU as the control unit.

[0041] Figure 11 This is a conceptual diagram illustrating an example of the processing performed by the CPU as the subject recognition and control unit.

[0042] Figure 12 This is a conceptual diagram representing an example of the processing performed by the CPU as the feature extraction unit.

[0043] Figure 13 This is a conceptual diagram representing an example of the processing performed by the CPU as the feature extraction and classification unit.

[0044] Figure 14 This is a concept diagram illustrating how the features of a subject are categorized into multiple classes based on a classification component.

[0045] Figure 15 This is a concept diagram illustrating an example of how a subject recognition unit identifies a subject based on real-time preview image data.

[0046] Figure 16 This is a concept diagram illustrating an example of processing content when displaying camera-supported footage within a live preview image.

[0047] Figure 17 This is an example of a screen displaying a bubble chart or similar format within the camera's view.

[0048] Figure 18 This is an example of a screen image that shows a histogram or similar display within the camera's view.

[0049] Figure 19 This is an example of a screen image showing how the real-time preview image is displayed when camera support processing is performed.

[0050] Figure 20AThis is a flowchart illustrating an example of the camera support processing flow.

[0051] Figure 20B yes Figure 20A Continuing from the flowchart shown.

[0052] Figure 21 This is an example of a screen image showing the content displayed in a real-time preview image of a subject image representing a specified camera area and object category, displayed in different ways.

[0053] Figure 22 This is a flowchart representing the first variation of the camera support processing flow.

[0054] Figure 23 This is a flowchart representing the second variation of the camera support processing flow.

[0055] Figure 24 This is a flowchart representing the third variation of the camera support processing flow.

[0056] Figure 25 This is a flowchart representing the fourth variation of the camera support processing flow.

[0057] Figure 26 This is a flowchart representing the fifth variation of the camera support processing flow.

[0058] Figure 27 This is a conceptual diagram representing a calculation example of the difference between the first and second camera conditions.

[0059] Figure 28 This is a concept diagram illustrating an example of the processing involved in increasing depth of field.

[0060] Figure 29 This is a conceptual diagram representing a calculation example of the focus position of a subject within a specified camera range and the focus position of a subject of a certain object category.

[0061] Figure 30 This is a block diagram illustrating an example of the processing involved in a main exposure camera using a focusing bracket method.

[0062] Figure 31 This is a flowchart of the sixth variation of the camera support processing flow.

[0063] Figure 32 This is a block diagram illustrating an example of the structure of different subject category groups.

[0064] Figure 33 This is an example of a screen image that displays a bubble chart or similar format for different time periods within the camera's viewfinder.

[0065] Figure 34This is an example of a screen image that shows a method of displaying period category histograms, etc., within the camera support screen.

[0066] Figure 35 This is a timing diagram illustrating an example of performing camera support processing at specified time intervals and an example of performing camera support processing for each location checkpoint.

[0067] Figure 36 This is a flowchart of the 7th variation of the camera support processing flow.

[0068] Figure 37 This is a conceptual diagram illustrating an example of how the main exposure image is categorized into multiple categories by a classification component.

[0069] Figure 38 This is a concept diagram representing an example of a bubble chart for four-quadrant face categories.

[0070] Figure 39 This is a block diagram illustrating an example of a machine learning approach that uses training data, including master exposure image data obtained through master exposure photography with support processing, to learn a model.

[0071] Figure 40 This is a block diagram illustrating an example of how a camera device enables an external device to perform camera support processing.

[0072] Figure 41 This is a block diagram illustrating an example of how a camera support processing program is installed from a storage medium containing the camera support processing program into a controller within a camera device. Detailed Implementation

[0073] Hereinafter, with reference to the accompanying drawings, an example of an embodiment of the camera support device, camera device, camera support method, and storage medium related to the technology of the present invention will be described.

[0074] First, let's explain the terminology used in the following description.

[0075] CPU stands for Central Processing Unit. RAM stands for Random Access Memory. IC stands for Integrated Circuit. ASIC stands for Application Specific Integrated Circuit. PLD stands for Programmable Logic Device. FPGA stands for Field Programmable Gate Array. SoC stands for System-on-a-chip. SSD stands for Solid State Drive. USB stands for Universal Serial Bus. HDD stands for Hard Disk Drive. EEPROM stands for Electrically Erasable and Programmable Read Only Memory. EL stands for Electro-Luminescent. I / F stands for Interface. UI stands for User Interface. TOF stands for "Time of Flight". fps stands for "Frames per Second". MF stands for "Manual Focus". AF stands for "Auto Focus". CMOS stands for "Complementary Metal Oxide Semiconductor". CCD stands for "Charge Coupled Device". RTC stands for "Real Time Clock". GPS stands for "Global Positioning System". LAN stands for "Local Area Network". WAN stands for "Wide Area Network". GNSS stands for "Global Navigation Satellite System".For ease of explanation, a CPU will be shown as an example of the "processor" according to the present invention, but the "processor" according to the present invention may also be a combination of multiple processing devices such as a CPU and a GPU. As an example of the "processor" according to the present invention, in the case of a combination of a CPU and a GPU, the GPU operates under the control of the CPU and undertakes the execution of image processing.

[0076] In this specification, "perpendicular" means not only perfectly perpendicular, but also includes a degree of error that is generally permissible in the technical field to which this invention pertains and does not violate the spirit of this invention. Similarly, "consistent" means not only perfectly consistent, but also includes a degree of error that is generally permissible in the technical field to which this invention pertains and does not violate the spirit of this invention.

[0077] As an example, such as Figure 1 As shown, the imaging device 10 is a digital camera with an interchangeable lens and a mirror omitted. The imaging device 10 includes an imaging device body 12 and an interchangeable lens 14 that is interchangeably mounted on the imaging device body 12. Furthermore, while an interchangeable lens digital camera with a mirror omitted is presented here as an example of the imaging device 10, the technology of the present invention is not limited to this. It can also be a lens-fixed digital camera, a digital camera without a mirror, or a digital camera built into various electronic devices such as smart devices, wearable terminals, cell observation devices, ophthalmic observation devices, or surgical microscopes.

[0078] An image sensor 16 is provided in the main body 12 of the camera device. The image sensor 16 is a CMOS image sensor. The image sensor 16 captures an image area including a group of subjects. When the interchangeable lens 14 is mounted on the main body 12 of the camera device, the subject light representing the subject passes through the interchangeable lens 14 and is imaged onto the image sensor 16, and the image data representing the subject image is generated by the image sensor 16.

[0079] Furthermore, in this embodiment, a CMOS image sensor is exemplified as image sensor 16, but the technology of the present invention is not limited thereto. For example, the technology of the present invention also applies even if the image sensor 16 is another type of image sensor such as a CCD image sensor.

[0080] A release button 18 and a turntable 20 are provided on the upper surface of the main body 12 of the camera device. The turntable 20 is operated when setting the operation mode of the camera system and the operation mode of the playback system. By operating the turntable 20, the camera mode and the playback mode are selectively set as the operation mode in the camera device 10.

[0081] The release button 18 functions as both a camera preparation indicator and a camera indicator, detecting press operations in both the camera preparation indicator state and the camera indicator state. The camera preparation indicator state refers to, for example, being pressed from the standby position to the intermediate position (half-pressed position), while the camera indicator state refers to being pressed to the final pressed position (fully pressed position) beyond the intermediate position. Furthermore, hereinafter, the state of "being pressed from the standby position to the half-pressed position" will be referred to as the "half-pressed state," and the state of "being pressed from the standby position to the fully pressed position" will be referred to as the "fully pressed state." Depending on the structure of the camera device 10, the camera preparation indicator state can be the state where the user's finger touches the release button 18, and the camera indicator state can be the state where the user's finger changes from touching the release button 18 to moving away from it.

[0082] As an example, such as Figure 2 As shown, a touch panel / display 22 and indicator keys 24 are provided on the back of the camera device body 12.

[0083] Touch panel / display 22 includes display 26 and touch panel 28 (see also) Figure 3 As an example of display 26, an organic EL display can be cited. Display 26 may not be an organic EL display, but rather another type of display such as a liquid crystal display or an inorganic EL display.

[0084] The display 26 displays images and / or character information, etc. When the camera device 10 is in camera mode, the display 26 is used to display a real-time preview image obtained by taking real-time preview images, i.e., continuous shooting. The real-time preview image taking (hereinafter also referred to as "real-time preview image shooting") is performed, for example, at a frame rate of 60 fps. 60 fps is only one example; it can be a frame rate less than 60 fps or a frame rate greater than 60 fps.

[0085] Here, "live preview image" refers to a dynamic image for display based on image data obtained by the image sensor 16. Live preview images are also commonly referred to as live view images. Furthermore, live preview images are an example of "display images" covered by the technology of this invention.

[0086] When the camera device 10 is instructed to capture a still image via the release button 18, the display 26 can also be used to display the still image obtained by capturing a still image. In addition, the display 26 is also used to display playback images when the camera device 10 is in playback mode, as well as to display menu screens, etc.

[0087] Touch panel 28 is a transmissive touch panel that overlaps the surface of the display area of ​​display 26. Touch panel 28 receives instructions from the user by detecting contact with an indicator such as a finger or stylus. In addition, for ease of explanation, the "full press state" mentioned above also includes the state in which the user activates the camera start soft key via touch panel 28.

[0088] Furthermore, in this embodiment, as an example of the touch panel / display 22, an external touch panel / display 28 can be described as overlapping the display area surface of the display 26, but this is only one example. For example, as the touch panel / display 22, an embedded or in-wall touch panel / display can also be used.

[0089] Indicator key 24 accepts various instructions. Here, "various instructions" refers to, for example, instructions for displaying a menu screen that allows selection of various menus, instructions for selecting one or more menus, instructions for confirming selections, instructions for deleting selections, zooming in, zooming out, and frame transmission, etc. Furthermore, these instructions can be made via touch panel 28.

[0090] As an example, such as Figure 3 As shown, the image sensor 16 includes a photoelectric conversion element 30. The photoelectric conversion element 30 has a light-receiving surface 30A. The center of the light-receiving surface 30A is aligned with the optical axis OA (reference). Figure 1 The camera body 12 is configured in a consistent manner (reference). Figure 1 The photoelectric conversion element 30 has multiple photosensitive pixels arranged in a matrix, and the light-receiving surface 30A is formed by the multiple photosensitive pixels. Each photosensitive pixel is a pixel with a photodiode (PD), which performs photoelectric conversion on the received light and outputs an electrical signal corresponding to the amount of light received. The photosensitive pixels included in the photoelectric conversion element 30 are of two types: phase difference pixels (P), also known as image plane phase difference pixels, and non-phase difference pixels (N), which are different from phase difference pixels (P).

[0091] Color filters are configured in a photodiode (PD). The color filters include a G filter corresponding to the G (green) wavelength region, an R filter corresponding to the R (red) wavelength region, and a B filter corresponding to the B (blue) wavelength region, which are most helpful in obtaining the brightness signal.

[0092] Typically, non-phase difference pixels N are also referred to as normal pixels. The photoelectric conversion element 30 has three types of photosensitive pixels—R pixels, G pixels, and B pixels—as non-phase difference pixels N. The R pixels, G pixels, B pixels, and phase difference pixels P are arranged regularly in a predetermined periodic pattern in both the row direction (e.g., the horizontal direction with the bottom surface of the imaging device body 12 in contact with a horizontal plane) and the column direction (e.g., the vertical direction, perpendicular to the horizontal direction). R pixels are pixels corresponding to photodiodes PD with R filters, G pixels and phase difference pixels P are pixels corresponding to photodiodes PD with G filters, and B pixels are pixels corresponding to photodiodes PD with B filters.

[0093] Multiple phase-difference pixel lines 32A and multiple non-phase-difference pixel lines 32B are arranged on the light-receiving surface 30A. Phase-difference pixel lines 32A are horizontal lines that include phase-difference pixels P. Specifically, phase-difference pixel lines 32A are horizontal lines where phase-difference pixels P and non-phase-difference pixels N coexist. Non-phase-difference pixel lines 32B are horizontal lines that include only multiple non-phase-difference pixels N.

[0094] On the light-receiving surface 30A, phase difference pixel lines 32A and a predetermined number of non-phase difference pixel lines 32B are alternately arranged along the column direction. The term "determined number of rows" here refers to, for example, two rows. While two rows are shown here as a predetermined number of rows, the technology of this invention is not limited to this; the predetermined number of rows can be three or more rows, or even a dozen, several dozen, or several hundred rows, etc.

[0095] Phase difference pixel line 32A is arranged skipping two rows along the column direction from the first row to the last row. A portion of the pixels in phase difference pixel line 32A are phase difference pixels P. Specifically, phase difference pixel line 32A is a horizontal line in which phase difference pixels P and non-phase difference pixels N are arranged periodically. Phase difference pixels P are roughly divided into first phase difference pixels L and second phase difference pixels R. In phase difference pixel line 32A, as G pixels, the first phase difference pixels L and the second phase difference pixels R are arranged alternately in the row direction at several pixel intervals.

[0096] The first phase difference pixel L and the second phase difference pixel R are configured to alternate in the column direction. Figure 3 In the example, in the fourth column, the pixels are arranged in the order of first phase difference pixel L, second phase difference pixel R, first phase difference pixel L again, and second phase difference pixel R from the first row along the column direction. That is, the first phase difference pixel L and the second phase difference pixel R are arranged alternately from the first row along the column direction. Furthermore, in Figure 3In the example, in the 10th column, the second phase difference pixel R, the first phase difference pixel L, the second phase difference pixel R, and the first phase difference pixel L are arranged in the order of the first phase difference pixel R and the first phase difference pixel L, starting from the first row along the column direction. That is, the second phase difference pixel R and the first phase difference pixel L are arranged alternately from the first row along the column direction.

[0097] The photoelectric conversion element 30 is divided into two regions. Specifically, the photoelectric conversion element 30 has a non-phase difference pixel division region 30N and a phase difference pixel division region 30P. The phase difference pixel division region 30P is a group of phase difference pixels based on multiple phase difference pixels P, which receives light from the subject and generates phase difference image data as an electrical signal corresponding to the amount of light received. The phase difference image data is used, for example, for distance measurement. The non-phase difference pixel division region 30N is a group of non-phase difference pixels based on multiple non-phase difference pixels N, which receives light from the subject and generates non-phase difference image data as an electrical signal corresponding to the amount of light received. The non-phase difference image data is displayed as a visible light image on the display 26 (reference). Figure 2 ).

[0098] As an example, such as Figure 4 As shown, the first phase difference pixel L includes a light-shielding member 34A, a microlens 36, and a photodiode PD. In the first phase difference pixel L, a light-shielding member 34A is disposed between the microlens 36 and the light-receiving surface of the photodiode PD. The left half of the light-receiving surface of the photodiode PD in the row direction (the left side when facing the subject from the light-receiving surface (in other words, the right side when facing the subject from the light-receiving surface)) is blocked by the light-shielding member 34A.

[0099] The second phase difference pixel R includes a light-shielding member 34B, a microlens 36, and a photodiode PD. In the second phase difference pixel R, a light-shielding member 34B is disposed between the microlens 36 and the light-receiving surface of the photodiode PD. The right half of the light-receiving surface of the photodiode PD in the row direction (the right side when facing the subject from the light-receiving surface (in other words, the left side when facing the subject from the light-receiving surface)) is blocked by the light-shielding member 34B. Furthermore, for ease of explanation, the light-shielding members 34A and 34B will not be labeled as such and will be referred to as "light-shielding member" below.

[0100] The interchangeable lens 14 includes a camera lens 40. The light beam exiting through the pupil of the camera lens 40 is roughly divided into left region light 38L and right region light 38R. The left region light 38L refers to the left half of the light beam passing through the pupil of the camera lens 40 when it faces the subject from the side of the phase difference pixel P, and the right region light 38R refers to the right half of the light beam passing through the pupil of the camera lens 40 when it faces the subject from the side of the phase difference pixel P. The light beam exiting through the pupil of the camera lens 40 is divided left and right by a microlens 36, a light-blocking member 34A, and a light-blocking member 34B, which function as pupil division parts. The first phase difference pixel L receives the left region light 38L as subject light, and the second phase difference pixel R receives the right region light 38R as subject light. As a result, the photoelectric conversion element 30 generates first phase difference image data corresponding to the subject image corresponding to the light 38L passing through the left region, and second phase difference image data corresponding to the subject image corresponding to the light 38R passing through the right region.

[0101] In the imaging device 10, for example, in the same phase difference pixel line 32A, the distance to the subject is measured based on the deviation α (hereinafter also simply referred to as "deviation α") between a line of first phase difference image data and a line of second phase difference image data. Furthermore, since the method for deriving the subject distance from the deviation α is a known technique, a detailed description is omitted here.

[0102] As an example, such as Figure 5 As shown, the difference between phase difference pixel P and non-phase difference pixel N is that it does not have a light-blocking component. The photodiode PD of non-phase difference pixel N receives light 38L from the left region and light 38R from the right region as subject light.

[0103] As an example, such as Figure 6 As shown, the imaging lens 40 includes an objective lens 40A, a focusing lens 40B, and an aperture 40C. The objective lens 40A, the focusing lens 40B, and the aperture 40C are arranged in the order of objective lens 40A, focusing lens 40B, and aperture 40C from the subject side (object side) to the imaging device body 12 side (image side) along the optical axis OA.

[0104] Furthermore, the interchangeable lens 14 includes a sliding mechanism 42, a motor 44, and a motor 46. A focusing lens 40B is slidably mounted on the sliding mechanism 42 along the optical axis OA. The motor 44 is connected to the sliding mechanism 42, and the sliding mechanism 42 operates under the power of the motor 44, thereby moving the focusing lens 40B along the optical axis OA. The aperture 40C is a variable aperture. The motor 46 is connected to the aperture 40C, and the aperture 40C adjusts the exposure by operating under the power of the motor 46. Additionally, the structure and / or operation method of the interchangeable lens 14 can be changed as needed.

[0105] Motors 44 and 46 are connected to the camera device body 12 via bayonet mounts (not shown) and are controlled and driven according to commands from the camera device body 12. Furthermore, in this embodiment, stepper motors are used as an example of motors 44 and 46. Thus, motors 44 and 46 operate synchronously with pulse signals according to commands from the camera device body 12. And, in Figure 6 The example shown is of motors 44 and 46 being disposed in the interchangeable lens 14, but it is not limited to this. At least one of motors 44 and 46 may be disposed in the camera device body 12, or both motors 44 and 46 may be disposed in the camera device body 12.

[0106] In the imaging device 10, in imaging mode, the MF mode and AF mode are selectively set according to the instructions given to the imaging device body 12. MF mode is the manual focus mode. In MF mode, for example, by the user operating the focus ring of the interchangeable lens 14, the focusing lens 40B moves along the optical axis OA by a movement corresponding to the amount of operation of the focus ring, thereby adjusting the focus.

[0107] In AF mode, the camera body 12 calculates the focus position corresponding to the distance to the subject and moves the focusing lens 40B toward the calculated focus position, thereby adjusting the focus. Here, the focus position refers to the position of the focusing lens 40B on the optical axis OA in the focusing state.

[0108] Furthermore, for ease of explanation, the control that aligns the focusing lens 40B to the focus position will be referred to as "AF control" below. Also, for ease of explanation, the calculation of the focus position will be referred to as "AF calculation" below. In the imaging device 10, focus detection of multiple subjects is performed by the CPU 48A (described later). Then, the CPU 48A (described later) focuses on the subject based on the AF calculation result, i.e., the focus detection result.

[0109] The camera device main body 12 includes an image sensor 16, a controller 48, an image memory 50, a UI system device 52, an external I / F 54, a photoelectric conversion element driver 56, a motor driver 58, a motor driver 60, a mechanical shutter driver 62, and a mechanical shutter actuator 64. Furthermore, the camera device main body 12 includes a mechanical shutter 72. The image sensor 16 also includes a signal processing circuit 74.

[0110] The input / output interface 70 is connected to a controller 48, an image memory 50, a UI system device 52, an external I / F 54, a photoelectric conversion element driver 56, a motor driver 58, a motor driver 60, a mechanical shutter driver 62, and a signal processing circuit 74.

[0111] The controller 48 includes a CPU 48A, a memory 48B, and a RAM 48C. The CPU 48A is an example of a "processor" according to the technology of the present invention, the RAM 48C is an example of "memory" according to the technology of the present invention, and the controller 48 is an example of a "camera support device" and a "computer" according to the technology of the present invention.

[0112] CPU48A, memory 48B and memory 48C are connected via bus 76, which is connected to input / output interface 70.

[0113] In addition, Figure 6 In the example shown, for ease of illustration, a single bus 76 is depicted, but multiple buses are also possible. Bus 76 can be a serial bus or a parallel bus that includes a data bus, an address bus, and a control bus.

[0114] Memory 48B stores various parameters and programs. Memory 48B is a non-volatile storage device. Here, EEPROM is used as an example of memory 48B. EEPROM is only one example; HDDs and / or SSDs, etc., can also be used as memory 48B, either in place of EEPROM or together with EEPROM. Furthermore, memory 48C temporarily stores various information and is used as working memory. RAM can be used as an example of memory 48C, but it is not limited to this; other types of storage devices can also be used.

[0115] Various programs are stored in memory 48B. CPU 48A reads the required program from memory 48B and executes the read program on memory 48C. CPU 48A controls the entire camera device body 12 according to the program executed on memory 48C. Figure 6 In the example, the image memory 50, UI system device 52, external I / F 54, photoelectric conversion element driver 56, motor driver 58, motor driver 60, and mechanical shutter driver 62 are controlled by CPU 48A.

[0116] A photoelectric conversion element driver 56 is connected to the photoelectric conversion element 30. The photoelectric conversion element driver 56 supplies a capture timing signal, specifying the capture timing to be performed by the photoelectric conversion element 30, to the photoelectric conversion element 30 according to instructions from the CPU 48A. The photoelectric conversion element 30 performs reset, exposure, and outputs electrical signals according to the capture timing signal supplied from the photoelectric conversion element driver 56. Examples of capture timing signals include, for example, a vertical synchronization signal and a horizontal synchronization signal.

[0117] With the interchangeable lens 14 mounted on the camera unit body 12, the subject light incident on the imaging lens 40 is imaged onto the light-receiving surface 30A by the imaging lens 40. Under the control of the photoelectric conversion element driver 56, the photoelectric conversion element 30 performs photoelectric conversion on the subject light received by the light-receiving surface 30A, and outputs an electrical signal corresponding to the amount of subject light as analog image data representing the subject light to the signal processing circuit 74. Specifically, the signal processing circuit 74 reads analog image data from the photoelectric conversion element 30 in an exposure-sequence readout manner, frame by frame and for each horizontal line. The analog image data is roughly divided into analog phase difference image data generated by phase difference pixels P and analog non-phase difference image data generated by non-phase difference pixels N.

[0118] The signal processing circuit 74 generates digital image data by digitizing analog image data input from the photoelectric conversion element 30. The signal processing circuit 74 includes a non-phase difference image data processing circuit 74A and a phase difference image data processing circuit 74B. The non-phase difference image data processing circuit 74A generates digital non-phase difference image data by digitizing analog non-phase difference image data. The phase difference image data processing circuit 74B generates digital phase difference image data by digitizing analog phase difference image data.

[0119] For ease of explanation, without needing to distinguish between digital non-phase difference image data and digital phase difference image data, they will be referred to as "digital image data" below. For ease of explanation, without needing to distinguish between analog image data and digital image data, they will be referred to as "image data" below.

[0120] Mechanical shutter 72 is a focal plane shutter, positioned between aperture 40C and the light-receiving surface 30A. Mechanical shutter 72 has a front curtain (not shown) and a rear curtain (not shown). Both the front and rear curtains have multiple blades. The front curtain is positioned closer to the subject than the rear curtain.

[0121] The mechanical shutter actuator 64 is an actuator having a front curtain solenoid (not shown) and a rear curtain solenoid (not shown). The front curtain solenoid is the drive source for the front curtain and is mechanically connected to the front curtain. The rear curtain solenoid is the drive source for the rear curtain and is mechanically connected to the rear curtain. The mechanical shutter driver 62 controls the mechanical shutter actuator 64 according to instructions from the CPU 48A.

[0122] The front curtain is powered by a solenoid under the control of the mechanical shutter driver 62, and this power is applied to the front curtain, thereby selectively raising and lowering it. The rear curtain is also powered by a solenoid under the control of the mechanical shutter driver 62, and this power is applied to the rear curtain, thereby selectively raising and lowering it. In the imaging device 10, the opening and closing of the front and rear curtains are controlled by the CPU 48A, thereby controlling the exposure to the photoelectric conversion element 30.

[0123] In the imaging device 10, real-time preview image capture and recording image capture for recording still images and / or moving images are performed in an exposure-sequence readout mode (rolling shutter mode). The image sensor 16 has an electronic shutter function, and real-time preview image capture is achieved by activating the electronic shutter function without operating the mechanical shutter 72 in a fully open state.

[0124] In contrast, the main exposure shooting, i.e., still image shooting (hereinafter also referred to as "main exposure shooting"), is achieved by activating the electronic shutter function and making the mechanical shutter 72 work by transitioning it from the front curtain closed state to the rear curtain closed state. Furthermore, the image obtained by the main exposure shooting performed by the imaging device 10 (hereinafter also referred to as "main exposure image") is an example of the "image captured by the present invention."

[0125] Digital image data is stored in image memory 50. Specifically, non-phase difference image data processing circuit 74A causes image memory 50 to store non-phase difference image data, and phase difference image data processing circuit 74B causes image memory 50 to store phase difference image data. CPU 48A retrieves digital image data from image memory 50 and uses the retrieved digital image data to perform various processes.

[0126] The UI system device 52 includes a display 26, and the CPU 48A enables the display 26 to display various information. Furthermore, the UI system device 52 includes a receiver 80. The receiver 80 includes a touch panel 28 and a hard key unit 82. The hard key unit 82 includes indicator keys 24 (see reference). Figure 2The CPU 48A operates according to various instructions received via the touch panel 28. Furthermore, while the hard key unit 82 is included in the UI system device 52, the technology of the present invention is not limited thereto; for example, the hard key unit 82 may also be connected to an external I / F 54.

[0127] The external I / F54 manages the transmission and reception of various information between the camera device 10 and devices located outside the camera device 10 (hereinafter also referred to as "external devices"). An example of an external I / F54 is a USB interface. External devices such as smart devices, personal computers, servers, USB memory, memory cards, and / or printers (not shown) can be directly or indirectly connected to the USB interface.

[0128] Motor driver 58 is connected to motor 44 and controls motor 44 according to instructions from CPU 48A. By controlling motor 44, the position of focusing lens 40B on optical axis OA is controlled via sliding mechanism 42. Focusing lens 40B moves to avoid the main exposure period of image sensor 16 according to instructions from CPU 48A.

[0129] The motor driver 60 is connected to the motor 46 and controls the motor 46 according to instructions from the CPU 48A. The size of the aperture 40C is controlled by controlling the motor 46.

[0130] As an example, such as Figure 7 As shown, a camera support processing program 84 is stored in memory 48B. CPU 48A reads the camera support processing program 84 from memory 48B and executes the read camera support processing program 84 on memory 48C. CPU 48A performs camera support processing according to the camera support processing program 84 executed on memory 48C (see also...). Figure 20A and Figure 20B ).

[0131] By performing camera support processing, the CPU48A first acquires frequency information, which represents the frequency of subject features classified into categories based on the features of the subject (hereinafter also referred to as "subject features") determined from the main exposure image obtained by the camera device 10. Then, the CPU48A performs support processing (hereinafter also simply referred to as "support processing") to support the shooting of the camera device 10 based on the acquired frequency information. The content of the camera support processing will be described in more detail below.

[0132] As an example, such as Figure 8As shown, the CPU 48A executes the camera support processing program 84, operating as an acquisition unit 48A1, a subject recognition unit 48A2, a feature extraction unit 48A3, a classification unit 38A4, and a control unit 48A5. Furthermore, the classification unit 38A4 is an example of a "classifier" according to the technology of this invention.

[0133] As an example, such as Figure 9 As shown, the acquisition unit 48A1 acquires non-phase difference image data from the image memory 50 as real-time preview image data. The real-time preview image data is acquired by the acquisition unit 48A1 from the image memory 50 at a predetermined frame rate (e.g., 60 fps). The real-time preview image data is image data representing a real-time preview image. The real-time preview image data is obtained by capturing an image of the camera area using the image sensor 16. Figure 9 In the example shown, real-time preview image data is obtained by capturing a camera area including multiple people. Here, multiple people are an example of "multiple objects" involved in the technology of this invention.

[0134] Furthermore, in this embodiment, for ease of explanation, a person is shown as the subject of the imaging device 10, but the technology of the present invention is not limited to this, and the subject can be anything other than a person. Examples of subjects other than a person include small animals, insects, plants, buildings, landscapes, organs of organisms and / or cells of organisms. That is, the imaging area may not include a person, as long as it includes a subject that can be captured by the image sensor 16.

[0135] Whenever the acquisition unit 48A1 acquires one frame of real-time preview image data, the control unit 48A5 displays the real-time preview image represented by the real-time preview image data acquired by the acquisition unit 48A1 on the display 26. The real-time preview image includes multiple person images representing multiple people as multiple subject images representing multiple subjects.

[0136] As an example, such as Figure 10 As shown, with a live preview image displayed on the display 26, a designated camera range is determined according to an externally given instruction (e.g., an instruction received by the receiving device 80). For example, the designated camera range is determined by operating the touch panel 28 with the user's finger. One purpose of determining the designated camera range is, for example, to determine the camera range that becomes the focus object.

[0137] exist Figure 10 In the example, the rectangle formed by the user sliding their finger across the touch panel 28 while touching it is in the background. Figure 10In the example, the area within the double-dotted box is the designated camera range. The control unit 48A5 stores designated camera range information 86, which indicates the designated camera range determined according to instructions given from the outside, in the memory 48B. Whenever the designated camera range is redefined, the designated camera range information 86 stored in the memory 48B is updated.

[0138] As an example, such as Figure 11 As shown, the acquisition unit 48A1 acquires non-phase difference image data from the image memory 50 as the main exposure image data. The subject recognition unit 48A2 identifies the subject within the imaging area based on the main exposure image data acquired by the acquisition unit 48A1. Figure 11 In the example, a learned model 92 is stored in memory 48B, and the subject recognition unit 48A2 uses the learned model 92 to recognize the subject within the imaging area.

[0139] As an example of the learned model 92, a learned model that uses a cascaded classifier can be cited. A learned model using a cascaded classifier can be constructed, for example, as a learned model for image recognition through supervised machine learning of a neural network. Furthermore, the learned model 92 is not limited to a learned model using a cascaded classifier; it can also be a vocabulary for pattern matching. That is, if the learned model 92 is a learned model used in image analysis when recognizing a subject, it can be any learned model.

[0140] The subject recognition unit 48A2 identifies people included in the camera area as subjects by performing image analysis on the main exposure image data. Furthermore, the subject recognition unit 48A2 identifies features of people, such as their face (expression), posture, the opening and closing of their eyes, and whether there are people within the specified camera range, as subject features by performing image analysis on the main exposure image data.

[0141] The subject recognition unit 48A2 identifies a person's face as one of the subject features. A person's face can be, for example, any one of a smiling face, a crying face, an angry face, or a serious face. The subject recognition unit 48A2 identifies smiling faces, crying faces, angry faces, and serious faces as subject features belonging to the "person's face" category. Furthermore, here, a smiling face refers to a person's smiling expression, a crying face refers to a person's crying expression, an angry face refers to a person's angry expression, and a serious face refers to an expression that is not equivalent to any of the three: smiling, crying, or angry.

[0142] Furthermore, the subject recognition unit 48A2 identifies the person's posture as one of the subject features. The person's posture can be either frontal or non-frontal. The subject recognition unit 48A2 identifies frontal and non-frontal postures as subject features belonging to the "person's posture" category. Additionally, here, "frontal" refers to the person's posture relative to the lit surface 30A (see reference). Figure 6Facing forward. Furthermore, "not facing forward" refers to a person facing a direction other than facing forward.

[0143] Furthermore, the subject recognition unit 48A2 identifies a person's eyes as one of the subject features. A person's eyes can be either open or closed. The subject recognition unit 48A2 identifies both open and closed eyes as subject features belonging to the "person's eyes" category. Here, "open eyes" refers to a person with their eyes open, and "closed eyes" refers to a person with their eyes closed.

[0144] Furthermore, the subject recognition unit 48A2 identifies the presence or absence of a person within the designated camera range as one of the subject characteristics. The subject recognition unit 48A2 identifies both the area within and outside the designated camera range as subject characteristics belonging to the "designated camera range" category. Additionally, here, "within the designated camera range" refers to the state where a person is present within the designated camera range, and "outside the designated camera range" refers to the state where a person is outside the designated camera range.

[0145] Furthermore, the subject recognition unit 48A2 stores recognition result information 94, indicating the recognition of subjects (in this case, people) included in the camera area, in memory 48C. The recognition result information 94 is stored in memory 48C frame by frame. The recognition result information 94 includes subject name, subject features, and coordinates of the recognition area, and is stored in memory 48C with each subject included in the camera area forming a corresponding association between the subject name, features, and coordinates of the recognition area.

[0146] Here, the recognition region determination coordinates refer to the coordinates representing the position of the quadrilateral frame (hereinafter also referred to as the "subject frame") enclosing the feature region (e.g., the facial region representing a person's face) of the subject image recognized by the subject recognition unit 48A2 within the instant preview image. As an example of the recognition region determination coordinates, the coordinates of the two vertices on the diagonal of the subject frame within the instant preview image can be given (e.g., the coordinates of the upper left and lower right corners). Furthermore, if the subject frame is quadrilateral, the recognition region determination coordinates can be the coordinates of three vertices or four vertices. Moreover, the shape of the subject frame is not limited to quadrilaterals and can be other shapes. In this case, the coordinates used to determine the position of the subject image within the instant preview image are sufficient to be used as the recognition region determination coordinates.

[0147] As an example, such as Figure 12 As shown, the feature extraction unit 48A3 extracts different subject feature information from the recognition result information 94 stored in the memory 48C. The different subject feature information is information that establishes a one-to-one correspondence between the subject name and the subject feature on a subject unit basis.

[0148] As an example, such as Figure 13 As shown, a category database 96 is constructed in memory 48B. The category database 96 includes multiple different subject category groups 98. Each different subject category group 98 is assigned one for each subject. Figure 13 In the example, multiple category groups 98 are shown, including category group for person A, category group for person B, category group for person C, category group for person D, and category group for person E (a category group assigned to each person separately). Each of the multiple different subject category groups 98 is a category that can determine each person among the multiple people. In addition, the different subject category groups 98 are an example of "object categories" involved in the technology of the present invention.

[0149] The classification unit 48A4 determines different subject category groups 98 corresponding to the subject names from the subject names included in the different subject feature information, and classifies the subject features corresponding to the subject names into the determined different subject category groups 98.

[0150] The different subject category groups 98 include multiple categories. Each of the multiple categories included in the different subject category groups 98 is created on a per-unit basis, independent of each other. A unit refers to a unit of subject characteristic. Figure 13 The examples shown include, as one example among the multiple categories included in different subject category groups 98, the face category, pose category, eye category, and specified camera range category. The face category is determined by the subject features of the "person's face" unit. The pose category is determined by the subject features of the "person's pose" unit. The eye category is determined by the subject features of the "person's eyes" unit. The specified camera range category is determined by the subject features of the "specified camera range" unit applicable to the person.

[0151] As an example, such as Figure 14 As shown, when the face category, pose category, eye category, and specified camera range category are set as major categories, multiple subcategories belong to the major category. A subcategory is the category in which the subject features belonging to the major category are classified. Within the multiple subcategories, a corresponding association is established based on the number of classifications. Within a subcategory, the subject feature is classified by classification section 48A4. Each time a subject feature is classified and added to a subcategory, the number of times the subject feature has been classified is incremented by "1".

[0152] exist Figure 14In the example, among the multiple categories included in the face categories within the different subject category group 98 used for person A, the categories shown are smiling face, crying face, angry face, and serious face. Classification unit 48A4 classifies the subject feature of person A's "smiling face" into the smiling face category. Then, each time the subject feature of the "smiling face" is classified into the smiling face category, classification unit 48A4 increments the classification count for the smiling face category by "1". Similarly, the subject features of person A's "crying face," "angry face," and "serious face" are classified into their corresponding categories by classification unit 48A4, and "1" is incremented in the corresponding classification count.

[0153] The number of classifications is also correlated with the face category. The number of classifications for a face category is the sum of the number of classifications for smiling faces, crying faces, angry faces, and serious faces.

[0154] Furthermore, in Figure 14 In the example, among the multiple categories included in the pose categories within the different subject category group 98 used for person A, the frontal category and the non-frontal category are shown. The classification unit 48A4 classifies the "frontal" subject features of person A into the frontal category. Then, each time the classification unit 48A4 classifies a "frontal" subject feature into the frontal category, it adds "1" to the classification count for the frontal category. Similarly, the classification unit 48A4 classifies the "non-frontal" subject features of person A into the non-frontal category, adding "1" to the corresponding classification count, just as it does for "frontal" subject features.

[0155] The number of classifications is also correlated with the pose category. The number of classifications for a pose category is the sum of the number of classifications for the positive pose category and the number of classifications for the non-positive pose category.

[0156] Furthermore, in Figure 14 In the example, the eye category within the different subject category group 98 used for the corresponding person A includes multiple categories, showing the open-eye category and the closed-eye category. The classification unit 48A4 classifies the subject feature of person A with "open eyes" into the open-eye category. Then, each time the classification unit 48A4 classifies the subject feature of "open eyes" into the open-eye category, it adds "1" to the classification count for the open-eye category. Similarly, the subject feature of person A with "closed eyes" is also classified into the closed-eye category by the classification unit 48A4, and "1" is added to the corresponding classification count.

[0157] The number of classifications is also correlated with the number of eye categories. The number of classifications for each eye category is the sum of the number of classifications for the open-eye category and the number of classifications for the closed-eye category.

[0158] In addition, Figure 14 In the example, multiple categories are included in the specified camera range category within the different subject category group 98 used for person A, showing the categories within the specified camera range and the categories outside the specified camera range. The classification unit 48A4 classifies the subject features "within the specified camera range" into the specified camera range category. Then, each time the classification unit 48A4 classifies a subject feature "within the specified camera range" into the specified camera range category, it adds "1" to the classification number for the specified camera range category. Similarly, for subject features "outside the specified camera range", the classification unit 48A4 classifies them into the specified camera range category, and adds "1" to the corresponding classification number.

[0159] The number of classifications is also correlated with the categories of the specified camera range. The number of classifications for the specified camera range categories is the sum of the number of classifications for categories within the specified camera range and the number of classifications for categories outside the specified camera range.

[0160] Different subject category groups 98 are categories determined by the subject characteristics of the "subject name" unit. The subject characteristics of the subject name are classified into different subject category groups 98. The number of classifications is also correlated with different subject category groups 98. The number of classifications correlated with different subject category groups 98 is the sum of the number of classifications belonging to multiple major categories of different subject category groups 98.

[0161] As an example, such as Figure 15 As shown, the acquisition unit 48A1 acquires non-phase difference image data from the image memory 50 as real-time preview image data. The subject recognition unit 48A2 uses the learned model 92 to recognize the subject within the camera area based on the real-time preview image data acquired by the acquisition unit 48A1.

[0162] The subject recognition unit 48A2 identifies people included in the camera area as subjects by performing image analysis on the real-time preview image data. Furthermore, the subject recognition unit 48A2 identifies features of people, such as their faces, postures, the opening and closing of their eyes, and whether they are present within the specified camera range, as subject features by performing image analysis on the real-time preview image data.

[0163] Specifically, the subject recognition unit 48A2 performs image analysis on the real-time preview image data to identify smiling, crying, angry, and serious faces as subject features belonging to the face category. Furthermore, the subject recognition unit 48A2 performs image analysis on the real-time preview image data to identify frontal and non-frontal subjects as subject features belonging to the pose category. Additionally, the subject recognition unit 48A2 performs image analysis on the real-time preview image data to identify open and closed eyes as subject features belonging to the eye category. Moreover, the subject recognition unit 48A2 performs image analysis on the real-time preview image data to identify subjects within and outside a specified camera range as subject features belonging to the specified camera range category. The subject recognition unit 48A2 stores the recognition result information 94 in memory 48C, which represents the result of identifying people included in the camera area. The recognition result information 94 is stored in memory 48C in one-frame units.

[0164] As an example, such as Figure 16 As shown, the control unit 48A5 displays a real-time preview image, represented by the real-time preview image data acquired by the acquisition unit 48A1, on the display 26. Furthermore, the control unit 48A5 refers to the recognition result information 94 stored in the memory 48C and retrieves the number of classifications for each category of the person (hereinafter also referred to as "recognized person") identified as a subject by the subject recognition unit 48A2 based on the real-time preview image data from the category database 96. That is, the control unit 48A5 retrieves the number of classifications for each recognized person from each category of the different subject category groups 98 corresponding to the recognized person.

[0165] The control unit 48A5 generates a camera support screen 100 based on the number of classifications obtained from each category included in the category database 96, and overlays the generated camera support screen 100 onto the real-time preview image. Figure 16 In the example, camera support screen 100 is displayed in the upper left corner of the live preview image.

[0166] As an example, such as Figure 17 As shown, the camera support screen 100 displays a bubble chart 100A and a category selection screen 100B. The bubble chart 100A is a graph that draws bubbles representing the number of classifications on two axes: an axis representing multiple identified persons and an axis representing categories.

[0167] exist Figure 17 In the example provided, as an example of bubble chart 100A, a bubble chart related to face categories is shown (hereinafter also referred to as "face category bubble chart"). The face category bubble chart is a chart that plots bubbles for each of the multiple people including people A to E, showing the number of times each face belongs to the smiling face category, crying face category, angry face category, and serious face category.

[0168] On the camera support screen 100, in addition to the face category bubble chart, bubble charts related to pose category (hereinafter also referred to as "pose category bubble chart"), eye category (hereinafter also referred to as "eye category bubble chart"), and specified camera range category (hereinafter also referred to as "specified camera range category bubble chart") are selectively displayed according to externally given instructions. Here, externally given instructions can be, for example, instructions received by the receiving device 80. Figure 17 In the example, multiple soft keys displaying the names of each category are shown in a scrolling manner within the category selection screen 100B. If any soft key is activated via the touch panel using the user's finger, a bubble chart related to the category corresponding to the activated soft key is displayed as a new bubble chart 100A within the camera support screen 100.

[0169] Furthermore, if any person within bubble chart 100A is selected via the touch panel using a user's finger, the number of categories in bubble chart 100A displayed at the current moment for the selected person is used as the histogram 100C (see reference). Figure 18 (Displayed within 100 pixels of the camera's view.) Figure 17 In the example, the user selected person A within bubble chart 100A via a touch panel using their finger. In this case, as an example, ... Figure 18 As shown in the figure, histogram 100C displays a histogram representing the camera's tendency to capture faces of person A (hereinafter also referred to as the "face category histogram"). In the face category histogram, the horizontal axis represents the categories of smiling faces, crying faces, angry faces, and serious faces, and the vertical axis represents the number of classifications.

[0170] exist Figure 18 In the example shown, a face category histogram is displayed as histogram 100C. However, depending on the type of bubble chart 100A and the person selected by the user, different types of histograms are displayed as histogram 100C. Besides the face category histogram, histogram 100C can also include, for example, a histogram representing the shooting tendency of pose category (hereinafter also called "pose category histogram"), a histogram representing the shooting tendency of eye category (hereinafter also called "eye category histogram"), and a histogram representing the shooting tendency of a specified shooting range category (hereinafter also called "specified shooting range category histogram") for each selected person. Furthermore, the histogram is not limited to each person; it can also be a histogram with the horizontal axis set to persons A through E and the vertical axis set to the number of times a subject is classified into different subject category groups 98. Thus, the horizontal and vertical axes of the histogram can be axes of any element that can constitute the histogram.

[0171] exist Figure 18 In the example, a display switching indicator screen 100D is displayed within the camera support screen 100. The display switching indicator screen 100D is a screen that receives an instruction to switch from the display of the histogram 100C to the display of the bubble chart 100A. A message guiding the switch to the bubble chart 100A is displayed on the display switching indicator screen 100D (in... Figure 18 In the example, the message "Return to the bubble chart screen?" and the toggle soft key (in) Figure 18 In the example, the soft key is displayed as "Yes". If the toggle soft key is activated via the touch panel by the user's finger, the display will switch from histogram 100C to bubble chart 100A.

[0172] Here, through receiving device 80 (in Figure 18 In the example, touch panel 28) is used to specify the number of classifications for any category within histogram 100C. Figure 18 In the example, the user selects the smiley face category via touch panel 28 using their finger, thereby specifying the number of times the smiley face category is classified. Thus, when any number of classifications for any category is specified, the control unit 48A5 performs support processing for the category (hereinafter also referred to as "object category") corresponding to the specified number of classifications.

[0173] Furthermore, in Figure 18 In the example, the "smiling face" category is specified as the object category from among multiple categories (smiling face, crying face, angry face, and serious face) within histogram 100C for person A. The object category is determined based on the number of classifications (in... Figure 18 In the example, the category is specified based on the number of classifications, according to the instructions received by the touch panel 28. In this case, as a supporting process, the following processing is performed: Support the use of image sensor 16 to capture a person A (hereinafter also referred to as "object category subject S") with subject features belonging to the smile category, i.e., a "smiley face" (reference). Figure 19 To take photos.

[0174] When a subject of a certain category is captured by the camera device 10, the subject category is determined based on the state of the subject S (e.g., facial expression, posture, eye opening and closing state, and / or the positional relationship between the person and the specified camera range, etc.), and is a category classified based on the subject characteristics of the subject S. Figure 18 In the example, when person A is photographed by camera device 10 as subject S, the subject feature of the least photographed expression among person A's smiling face, crying face, angry face, and serious face is the smiling face, which is designated by the user as the subject category.

[0175] Compared to the crying face, angry face, and serious face categories, the smiling face category, which is specified by the user as the object category in histogram 100C, has the lowest classification frequency. This means that fewer main exposure images of person A with a smiling face are taken than fewer main exposure images with other expressions. Furthermore, in histogram 100C, the smiling face category is an example of a "low-frequency category" covered by the technology of this invention.

[0176] Thus, if the object category is specified from histogram 100C, for example, Figure 19 As shown, the control unit 48A5 displays a real-time preview image obtained by the camera device 10 capturing an area including the subject S of the object category on the display 26, and performs display processing as a support process. This display processing determines the subject S of the smiling object category based on the recognition result information 94, and recommends shooting the determined subject S of the object category. The display processing includes the following steps: on the real-time preview image, displaying an arrow indicating the subject image S1 of the object category S representing the smiling object category S, and displaying information recommending shooting the subject S of the object category S represented by the subject image S1 indicated by the arrow (hereinafter also referred to as "shooting recommendation information"). Figure 19 In the example, as a camera recommendation message, the message "It is recommended to photograph this subject." is shown.

[0177] Furthermore, the display processing includes the following steps: displaying the object category subject image S1 in a manner distinguishable from other image areas within the instant preview image. In this case, for example, the control unit 48A5 has recognition result information 94 stored in memory 48C to detect the face region of the object category subject image (the image region representing the face of the object category subject S), and displays the detection box 102 surrounding the detected face region in the instant preview image, thereby displaying the object category subject image S1 in a manner distinguishable from other image areas within the instant preview image.

[0178] Furthermore, the method of displaying the object category subject image S1 in the instant preview image in a way that distinguishes it from other image areas is not limited to this. For example, the object category subject image S1 can also be displayed only in a peak mode in the instant preview image.

[0179] Next, refer to Figure 20A and Figure 20B The function of the camera device 10 will be explained.

[0180] exist Figure 20A and Figure 20BThis diagram illustrates an example of the camera support processing flow executed by the CPU 48A when a camera mode is set for the camera device 10. This camera support processing flow is an example of the "camera support method" according to the technology of this invention. Furthermore, for ease of explanation, the following description assumes that the camera device 10 captures a real-time preview image at a specified frame rate. Also, for ease of explanation, the following description assumes that a specified camera range information 86 is stored in the memory 48B, and that a category database 96 has been constructed.

[0181] exist Figure 20A In the camera support processing shown, firstly, in step ST100, the acquisition unit 48A1 acquires real-time preview image data from the image memory 50.

[0182] In the next step ST102, the control unit 48A5 causes the real-time preview image represented by the real-time preview image data acquired in step ST100 to be displayed on the display 26.

[0183] In the next step ST104, the subject recognition unit 48A2 uses the learned model 92 to identify the subject within the camera area based on the real-time preview image data obtained in step ST100.

[0184] In the next step ST106, the control unit 48A5 obtains the number of classifications for each category of the subject identified in step ST104 from the category database 96.

[0185] In the next step ST108, the control unit 48A5 creates a camera support screen 100 based on the number of classifications obtained in step ST106, and displays the created camera support screen 100 in a portion of the instant preview image.

[0186] In the next step ST110, the control unit 48A5 determines whether any number of classifications has been specified from the histogram 100C within the camera support screen 100. If, in step ST110, no number of classifications has been specified from the histogram 100C within the camera support screen 100, the determination is rejected, and the camera support process is transferred to the next step. Figure 20B The step ST116 is shown. In step ST110, if any number of classifications has been specified from the histogram 100C within the camera support screen 100, the determination is affirmative, and the camera support process proceeds to step ST112.

[0187] In step ST112, the control unit 48A5 displays the detection box 102 in the instant preview image in a manner that surrounds the face region of the object category subject image S1, which represents the object category subject S having subject features that belong to the object category specified based on the specified number of classifications.

[0188] In the next step ST114, the control unit 48A5 displays camera recommendation information within the real-time preview image. After the processing in step ST114 is completed, the camera support processing is transferred to... Figure 20B Step ST116 is shown.

[0189] In step ST116, the control unit 48A5 determines whether the conditions for starting the main exposure (hereinafter referred to as the "main exposure start condition") are met. An example of a main exposure start condition is a condition where the device becomes fully pressed as indicated by an instruction received by the receiving device 80. If the main exposure start condition is not met in step ST116, the determination is negative, and the camera support process proceeds to step ST130. If the main exposure start condition is met in step ST116, the determination is positive, and the camera support process proceeds to step ST118.

[0190] In step ST118, regarding the imaging area including the subject S (object category), the control unit 48A5 causes the image sensor 16 to perform a main exposure image. By performing the main exposure image, main exposure image data representing the main exposure image of the imaging area including the subject S (object category) is stored in the image memory 50.

[0191] In the next step ST120, the acquisition unit 48A1 acquires the main exposure image data from the image memory 50.

[0192] In the next step ST122, the subject recognition unit 48A2 uses the learned model 92 to recognize the subject within the camera area based on the main exposure image data obtained in step ST120, and stores the recognition result information 94 in the memory 48C.

[0193] In the next step ST124, the feature extraction unit 48A3 extracts different subject feature information from the recognition result information 94 stored in the memory 48C for each subject.

[0194] In the next step ST126, the classification unit 48A4 determines the different subject category groups 98 corresponding to the subject names from the different subject feature information extracted in step ST124, and classifies the subject features into the corresponding categories within the determined different subject category groups 98.

[0195] In the next step ST128, the classification unit 48A4 updates the classification number by adding "1" to the classification number of the categories that have classified the subject features.

[0196] In the next step ST130, the control unit 48A5 removes the live preview image and other information (e.g., live preview image, camera support screen 100, detection frame 102, and camera recommendation information) from the display 26.

[0197] In the next step ST132, the control unit 48A5 determines whether the conditions for ending the camera support processing (hereinafter also referred to as "camera support processing end conditions") are met. Examples of camera support processing end conditions include conditions such as the cancellation of the camera mode set on the camera device 10, or the acceptance of an instruction to end the camera support processing via the receiving device 80. In step ST120, if the camera support processing end conditions are not met, the determination is negative, and the camera support processing proceeds to step ST100. In step ST132, if the camera support processing end conditions are met, the determination is positive, and the camera support processing ends.

[0198] As explained above, in the imaging device 10, the control unit 48A5 obtains the number of times subject features are classified into categories based on subject features determined from the main exposure image. Then, the control unit 48A5 performs support processing to support the imaging device 10's shooting based on the number of classifications. Therefore, according to this structure, the imaging device 10 can support shooting based on the number of times subject features are classified into categories.

[0199] Furthermore, in the camera device 10, categories are divided into multiple categories, including object categories. The object categories are determined based on the number of classifications. Figure 18 In the example, the object category is specified by the user selecting the number of classifications. Then, the control unit 48A5 performs support shooting processing on the object category subject S, which has subject characteristics belonging to the object category. Figure 19 In the example, the smiling object category subject S is processed to support shooting.

[0200] In addition, Figure 19 In the example, the smiley face category is specified by the number of times the smiley face category is selected by the user, thus supporting the shooting of the subject S of the smiling face category. However, if a subcategory other than the smiley face category is specified among multiple subcategories within the face category by the number of times other subject features (e.g., crying face, angry face, and / or serious face) are selected by the user, then the processing of shooting the subject S of the subject category that has subject features belonging to the specified subject category is supported.

[0201] It is not limited to the subcategory belonging to the face category. When the subcategory is specified by the number of times the subject features are classified into the subcategory belonging to other major categories by the user, the subject S with subject features belonging to the subcategory specified as the object category is supported for shooting.

[0202] Furthermore, when different subject category groups 98 are specified by the number of times the subject name is classified as a subject feature in different subject category groups 98 by the user, the subject category subject S with subject features belonging to different subject category groups 98 that are specified as subject categories, that is, the subject category subject S with subject name corresponding to different subject category groups 98 that are specified as subject categories, is supported for shooting.

[0203] Therefore, based on this structure, it is possible to effectively photograph subjects of the type of object requested by the user, compared to the case where the photographer judges whether the subject S is the object category requested by the user based solely on the photographer's senses.

[0204] Furthermore, in this embodiment, as a recommended subject category, the display of the subject S is used to provide video recommendation information (in... Figure 19 The example shows the display of the message "Recommended to photograph this subject". Therefore, according to this structure, compared to judging whether a subject S possesses the characteristics requested by the user based solely on the photographer's perception, it is possible to effectively photograph subject categories with expressions requested by the user.

[0205] Furthermore, in this embodiment, the control unit 48A5 displays a real-time preview image on the display 26, and within the real-time preview image, the object category subject image S1 is displayed in a manner distinguishable from other image areas. Thus, according to this structure, the user can visually identify the object category subject.

[0206] Furthermore, in this embodiment, the control unit 48A5 displays a real-time preview image on the display 26, and within the real-time preview image, a detection box 102 is displayed for the face region of the object category subject image S1. Therefore, according to this structure, the user can identify that the subject corresponding to the display area where the detection box 102 is displayed is an object category subject.

[0207] Furthermore, in this embodiment, the user specifies the category with a relatively low number of classifications among multiple categories (in...) as the object category. Figure 18(In the example, the smiley face category). Thus, according to this structure, it is possible to effectively photograph subjects in categories with relatively low classification frequencies among multiple categories, compared to judging whether a subject belongs to a category with a relatively low classification frequency among multiple categories solely based on the photographer's perception.

[0208] Furthermore, in this embodiment, the object category is determined based on the state of the subject S (e.g., a person's facial expression). Figure 19 In the example, a smile category is specified as the object category, determined based on the state of the subject S, which is a "smiley face". Then, the control unit 48A5 processes the data to support the shooting of subjects having the characteristics of a subject classified as the specified category, i.e., the object category subject S. Therefore, according to this structure, it is possible to increase the number of shots of subjects in the same state, or conversely, to decrease the number of shots of subjects in the same state.

[0209] Furthermore, in this embodiment, multiple different subject category groups 98 are included in the category database 96, which can determine the category of each of the multiple people. Therefore, the control unit 48A5 performs the following processing: supports shooting subjects corresponding to the different subject category groups 98 specified by the user, i.e., object category subjects S. Thus, according to this structure, it is possible to increase the number of times the same subject is shot, or conversely, to decrease the number of times the same subject is shot.

[0210] Furthermore, in this embodiment, the categories for classifying subject features are created in multiple units. Examples of these multiple units include, for instance, a unit for "subject name (e.g., a person's name or an identifier that identifies a person)," a unit for "the person's face (expression)," a unit for "the person's posture," a unit for "the person's eyes," and a unit for "specifying the camera range." Thus, according to this structure, the camera device 10 can support shooting based on the number of times the subject features are classified into the specified unit's category.

[0211] Furthermore, in this embodiment, as support processing for the recording by the camera device 10, the control unit 48A5 performs processing including displaying the number of classifications on the display 26. Figure 17 In the example, bubble chart 100A is used to represent the number of classifications. Figure 18 In the example, a histogram 100C is used to represent the number of classifications. Therefore, based on this structure, users can determine the number of times each feature of various subjects was photographed.

[0212] Furthermore, in this embodiment, when the histogram 100C is displayed on the display 26 and the receiving device 80 specifies the number of classifications within the histogram 100C, the control unit 48A5 performs processing to support shooting related to the category corresponding to the specified number of classifications. Therefore, according to this structure, it is possible to support shooting subjects with subject characteristics desired by the user.

[0213] Furthermore, in the above embodiments, examples have been given where the subject S is located within the designated camera range, but the technology of the present invention is not limited thereto. For example, when the subject S is located outside the designated camera range, such as... Figure 21 As shown, within the live preview image, the object category subject image S1 is displayed at a position offset from the object Ob representing the specified camera range. In this case, the control unit 48A5 displays the object Ob representing the specified camera range and the object category subject image S1 (an example of "object representing object category subject" according to the technology of the present invention) in a different display mode within the live preview image. Figure 21 In the example, the object category subject image S1 is displayed in a peaked manner, and the object Ob is colored in a semi-transparent color (e.g., semi-transparent gray). This allows for the visual identification of the specified camera range and object category subject S that the user intends to capture.

[0214] Furthermore, in the above embodiment, by having the user specify the number of classifications shown in the histogram 100C via the touch panel 28, the system supports categories corresponding to the specified number of classifications (in... Figure 18 In the example shown, the shooting is related to the smiley face category, but the technology of the present invention is not limited to this. For example, the control unit 48A5 can perform the following processing: among multiple categories, determine the low-frequency category with a relatively low number of classifications as the object category, and support shooting related to the determined low-frequency category. For example, the low-frequency category refers to the category with the fewest classifications (e.g., the smiley face category) among multiple subcategories (smiley face category, crying face category, angry face category, and serious face category) belonging to a large category (e.g., face category) specified by the user via the receiving device 80. The control unit 48A5 performs the following processing: in the real-time preview image, determine the subject of interest (e.g., person A) specified by the user via the receiving device 80 that has subject characteristics classified as a low-frequency category, and support shooting of the determined subject of interest. In this case, the selection of the histogram 100C (user operation) is not required, and the camera support screen 100 may not be displayed.

[0215] Thus, in the case where the control unit 48A5 supports the shooting of subjects with features classified as low-frequency categories, for example, Figure 22 As shown, camera support processing is performed by the CPU48A. (And...) Figure 20A Compared to the flowchart shown, Figure 22 The difference in the flowchart shown is that steps ST200 and ST202 are used instead of steps ST106 to ST112.

[0216] exist Figure 22 In the camera support processing step ST200 shown, regarding the subject of interest specified by the user, the control unit 48A5 determines a low-frequency category among multiple subcategories within a broad category specified by the user. For example, according to Figure 22 In the bubble chart 100A shown, when the user specifies the face category as the major category and the user specifies person A as the subject of interest, the control unit 48A5 determines the smile category as the low-frequency category.

[0217] In the next step ST202, the control unit 48A5, in the real-time preview image, determines the subject image (hereinafter also referred to as the "subject image of interest") that represents a subject with subject features classified as a low-frequency category among the subjects of interest specified by the user, and displays a detection box 102 (see reference) for the face region of the subject image of interest. Figure 19 and Figure 21 For example, if the user designates person A as the subject of interest, and in step ST200, if a smiling face category is automatically determined as a low-frequency category, firstly, the control unit 48A5 determines the image of person A representing a smiling face within the real-time preview image based on the recognition result information 94 as the subject of interest image. Then, the control unit 48A5 displays a detection box 102 surrounding the face region of the image of person A representing a smiling face within the real-time preview image. Furthermore, in this embodiment, "automatic" means that the action is performed primarily by the controller 48, rather than triggered by human operation (e.g., user operation).

[0218] Therefore, according to Figure 22 As an example, because it supports shooting subjects with subject characteristics that are classified into the least frequently classified category, it is possible to effectively shoot low-frequency category subjects compared to shooting subjects based solely on the photographer's perception of whether they are low-frequency category subjects.

[0219] In the above embodiments, examples have been given of main exposure imaging performed when the conditions for receiving instructions from the user by the receiving device 80 are met as the main exposure start conditions in the imaging support processing; however, the technology of the present invention is not limited thereto. For example, the control unit 48A5 detects the object category subject based on the imaging results of the imaging device 10, and can automatically acquire an image including an image corresponding to the object category subject when the object category subject is detected.

[0220] In this case, for example, with Figure 22 Similarly, in the example, firstly, the control unit 48A5 determines the low-frequency category as the object category. Then, based on the recognition result information 94, if the control unit 48A5 detects the existence of a subject image of interest corresponding to the low-frequency category (i.e., a low-frequency subject image of interest) in the instant preview image, it automatically starts the main exposure recording. Here, the low-frequency subject image of interest corresponding to the low-frequency category refers to, for example, a subject image of interest representing a subject with a "smiling face" characteristic, where the control unit 48A5 determines that the "smile" category is the subject of interest when person A is designated as the subject of interest. Furthermore, by performing the main exposure recording, one frame of main exposure image data can be stored in a predetermined storage area (e.g., image memory 50), and in the one frame of main exposure image data, only data related to the low-frequency subject image of interest can be stored in the predetermined storage area.

[0221] Thus, when a subject of a certain object category is detected, and the control unit 48A5 automatically acquires an image including an image corresponding to the subject of that object category, for example, Figure 23 As shown, camera support processing is performed by the CPU48A. (And...) Figure 20A and Figure 20B Compared to the flowchart shown, Figure 23 The difference in the flowchart shown is that steps ST300 and ST302 are used instead of steps ST106 to ST116.

[0222] exist Figure 23 In the camera support processing step ST300 shown, the control unit 48A5 performs the operation with... Figure 22 The process shown in step ST200 is the same.

[0223] In the next step ST302, the control unit 48A5 determines whether a low-frequency subject image exists in the live preview image based on the recognition result information 94. If, in step ST302, no low-frequency subject image exists in the live preview image, the determination is rejected, and the camera support processing proceeds to step ST130. If, in step ST302, a low-frequency subject image exists in the live preview image, the determination is affirmed, and the camera support processing proceeds to step ST118.

[0224] Thus, when the control unit 48A5 detects the presence of a low-frequency subject image in the instant preview image, the main exposure recording begins. Therefore, compared to the case where the subject is observed visually and the recording begins upon receiving instructions from the user via the receiving device 80, the time required to record the subject is reduced.

[0225] exist Figure 23 In the example given, in step ST302 of the camera support processing, it is determined whether a low-frequency subject image of interest exists in the live preview image. An example of starting the main exposure camera when it is determined that a low-frequency subject image of interest exists in the live preview image has been described. However, the technology of the present invention is not limited to this. For example, it can also be performed by the control unit 48A5. Figure 24 The process shown in step ST352 is used instead. Figure 23 The process shown is ST302.

[0226] exist Figure 24 In step ST352 of the camera support processing shown, the control unit 48A5 determines whether the camera range condition is met. Here, the camera range condition refers to the fact that the object category of the subject is included in the specified camera range information 86 (reference). Figure 10 and Figure 11 The condition within the specified camera range is indicated by ). The determination of whether the object category subject is included within the specified camera range is performed, for example, by determining whether the subject of interest represented by the low-frequency subject of interest image is included within the specified camera range based on the specified camera range information 86 and the recognition result information 94. In step ST352, if the camera range condition is not met, the determination is rejected, and the camera support process proceeds to step ST130. In step ST352, if the camera range condition is met, the determination is rejected, and the camera support process proceeds to step ST118.

[0227] Thus, by performing the main exposure photography under the condition that the subject is included in the specified camera range, the time required to photograph the subject can be reduced compared to the case where the subject is visually observed to be included in the specified camera range and photography begins under the condition that the receiving device 80 receives instructions from the user.

[0228] exist Figure 24 In the example, the main exposure is performed in step ST118 of the camera support processing, but it can also be performed while focusing on a subject of an object category within a specified camera range. Here, "shooting" can refer to shooting for real-time preview of an image or shooting for recording an image (e.g., a still image or a moving image).

[0229] In this case, Figure 25 The camera support processing shown is executed by CPU48A. (And...) Figure 24 Compared to the flowchart shown, Figure 25 The difference in the flowchart shown is that step ST400 is used instead of step ST118.

[0230] exist Figure 25 In step ST400 of the image support processing shown, regarding the imaging area including the subject S of the object category, after the control unit 48A5 focuses on the subject of the object category (e.g., the subject of interest represented by the low-frequency subject of interest image in the instant preview image) based on the AF calculation result, it causes the image sensor 16 to perform a main exposure image. After performing the processing in step ST400, the image support processing proceeds to step ST120.

[0231] Thus, by focusing on the subject within the designated camera range and then taking the main exposure, the time required to focus on the subject after placing it within the designated camera range can be reduced.

[0232] exist Figure 25 The examples given illustrate a form of main exposure imaging when the imaging range conditions are met, but the technology of the present invention is not limited thereto. For example, the control unit 48A5 may perform a predetermined process when the difference between the first imaging condition imposed from the outside (e.g., imaging condition determined according to an instruction received by the receiving device 80) and the second imaging condition imposed on the subject of the object category is a predetermined difference or greater.

[0233] In this case, for example, Figure 26 The camera support processing shown is executed by CPU48A. (And...) Figure 25 Compared to the flowchart shown, Figure 26 The difference in the flowchart shown is that steps ST450 to ST456 replace steps ST352, ST400, and ST120.

[0234] exist Figure 26In step ST450 of the camera support processing shown, the control unit 48A5 acquires the first camera condition and the second camera condition. Examples of the first camera condition and the second camera condition will be described later.

[0235] In the next step ST452, the control unit 48A5 calculates the degree of difference between the first camera condition and the second camera condition obtained in step ST450 (for example, the degree to which the first camera condition and the second camera condition deviate).

[0236] In the next step ST454, the control unit 48A5 determines whether the difference calculated in step ST452 is greater than or equal to a predetermined difference. If, in step ST454, the difference calculated in step ST452 is less than the predetermined difference, the determination is rejected, and the control unit 48A5 performs processing equivalent to step ST400 and steps ST120 to ST132 (see reference). Figure 25 After processing, the process moves to step ST100. In step ST454, if the difference calculated in step ST452 is above the specified difference, the determination is affirmed, and the camera support process moves to step ST456.

[0237] In step ST456, the control unit 48A5 performs prescribed processing. Details will be described later; this prescribed processing includes post-exposure imaging processing (deep-field adjustment after adjusting the depth of field) and / or focus bracket imaging processing (focus bracket method for main exposure). After performing the processing in step ST456, the camera support processing proceeds to step ST122.

[0238] Thus, according to Figure 26 As an example, when the difference between the first camera condition imposed from the outside and the second camera condition imposed on the subject of the object category is greater than a specified difference, specified processing is performed, which can help the user shoot under the desired camera conditions.

[0239] exist Figure 27 and Figure 28 The example illustrates the camera processing after depth-of-field adjustment. As an example, such as... Figure 27 As shown, if the first imaging condition is the location of a specified imaging range, and the second imaging condition is the location of a subject within a specified object category, and the location of the subject within the specified object category is not within the specified imaging range, the control unit 48A5 determines that the difference is greater than or equal to a predetermined difference. If the difference is greater than or equal to the predetermined difference, the control unit 48A5 controls the imaging device 10 to ensure that the subject within the specified imaging range and the subject within the specified object category are included in the depth of field. While the subject within the specified imaging range and the subject within the specified object category are included in the depth of field, the image sensor 16 performs a main exposure imaging.

[0240] As an example, when the degree of difference is above the specified degree of difference, such as... Figure 28 As shown, firstly, the acquisition unit 48A1 calculates the focus position for each of the subjects within the specified camera range and each of the subject categories based on the AF operation results for each of the multiple subjects, namely at least one subject within the specified camera range (hereinafter also referred to as "subject within the specified camera range") and each of the subject categories. For example, the acquisition unit 48A1 calculates the focus position for each of the subjects within the specified camera range and each of the subject categories based on the phase difference image data corresponding to each position of the subject image within the specified camera range (representing the image of the subject within the specified camera range) and the subject image S1 of the subject category in the live preview image. Furthermore, the method for calculating the focus position is only one example; the focus position can also be calculated using TOF or contrast AF methods.

[0241] The acquisition unit 48A1 calculates the depth of field, which includes the subjects and subject categories within the specified camera range, based on multiple focus positions calculated for each subject and subject category within the specified camera range. The depth of field is calculated using a first formula. The first formula is, for example, a formula that sets multiple focus positions as independent variables and depth of field as a dependent variable. Alternatively, a first table that establishes a correspondence between multiple focus positions and depth of field can be used instead of the first formula.

[0242] The acquisition unit 48A1 calculates the F-value of the depth of field obtained from the calculation. The acquisition unit 48A1 calculates the F-value using a second formula. The second formula used here is, for example, a formula that sets the depth of field as an independent variable and the F-value as a dependent variable. Alternatively, a second table that establishes a correspondence between the depth of field value and the F-value can be used instead of the second formula.

[0243] The control unit 48A5 controls the motor 46 via the motor driver 60 according to the F value calculated by the acquisition unit 48A1, thereby making the aperture 40C work.

[0244] Thus, according to Figure 27 and Figure 28 As exemplified, when the subject S is not located within the specified camera range, the aperture is set to f / 40 so that both the subject and the subject S are contained within the depth of field within the specified camera range. The main exposure is then performed while both the subject and the subject S are contained within the depth of field. Therefore, compared to the case where the subject and the subject S are not contained within the depth of field, a higher contrast image can be obtained without multiple shots, using both the subject image and the subject image within the specified camera range.

[0245] However, it is also possible to consider a situation where, due to the structure of the camera device 10, neither the subject nor the object category subject S within the specified camera range is included in the depth of field. In such a case, the control unit 48A5 causes the camera device 10 to capture images of the subject and object category subject within the specified camera range using a focusing bracket.

[0246] In this case, as an example, such as Figure 29 As shown, firstly, the acquisition unit 48A1 calculates a first focus position and a second focus position based on phase difference image data corresponding to each position of the subject image and the object category subject image S1 within a specified shooting range in the live preview image. The first focus position is the focus position of the center of the subject within the specified shooting range, and the second focus position is the focus position of the object category subject S. The first focus position and the second focus position are the focus positions used when performing main exposure shooting in the focus bracket mode.

[0247] If the first focus position and the second focus position are calculated by the acquisition unit 48A1, then as an example, ... Figure 30 As shown, the control unit 48A5 moves the focusing lens 40B to the first focus position, and instructs the image sensor to start the main exposure recording based on the time when the focusing lens 40B reaches the first focus position. Accordingly, the image sensor 16 performs the main exposure recording. Thus, after performing the main exposure recording with the focusing lens 40B aligned to the first focus position, the control unit 48A5 moves the focusing lens 40B to the second focus position, and instructs the image sensor to start the main exposure recording based on the time when the focusing lens 40B reaches the second focus position. Accordingly, the image sensor 16 performs the main exposure recording. Furthermore, here, the first and second focus positions are shown as examples of focus positions used in the focus bracket method of main exposure recording, but this is only one example; more than three focus positions may be used as focus positions in the focus bracket method of main exposure recording.

[0248] Thus, according to Figure 29 and Figure 30 As exemplified, when both the subject and the object category subject S within the specified camera range are not included in the depth of field due to the structure of the camera device 10, a main exposure is performed on the subject and the object category subject S within the specified camera range using a focusing bracket. Therefore, compared to performing only one frame of video recording when the subject and the object category subject S are not included in the depth of field due to the structure of the camera device 10, even when both the subject and the object category subject S are not included in the depth of field, a high-contrast image can be obtained as the subject image and the object category subject image within the specified camera range.

[0249] exist Figure 27In the example shown, the position of the specified imaging range is illustrated as the first imaging condition, and the position of the object category subject S is illustrated as the second imaging condition; however, the technology of the present invention is not limited to this. For example, the first imaging condition may be set to the brightness of the reference subject (e.g., the subject within the specified imaging range), and the second imaging condition may be set to the brightness of the object category subject S. In this case, if the difference between the brightness of the reference subject and the brightness of the object category subject S (hereinafter also referred to as "brightness difference") is a predetermined brightness difference or higher, the control unit 48A5 performs the exposure bracket mode imaging process as specified above. The exposure bracket mode imaging process is the process of having the imaging device 10 photograph the reference subject and the object category subject S in an exposure bracket mode.

[0250] Furthermore, when the brightness difference is less than the specified brightness difference, the control unit 48A5 causes the camera device 10 to capture the reference subject and the object category subject S with an exposure determined by the reference subject.

[0251] As an example, the exposure bracket method of image processing is executed by the CPU48A. Figure 31 The camera support processing shown is used to achieve this. (Compared to...) Figure 26 Compared to the flowchart shown, Figure 31 The difference in the flowchart shown is that steps ST500 to ST508 are used instead of steps ST450 to ST456.

[0252] exist Figure 31 In the camera support processing step ST500 shown, the acquisition unit 48A1 acquires the metering value for each of the subjects and object category subjects S within a specified camera range. The metering value can be calculated based on real-time preview image data or detected by a metering sensor (not shown).

[0253] In the next step ST502, the control unit 48A5 uses two metering values ​​acquired in step ST500 for each of the subject and object category S within the specified imaging range to calculate the brightness difference. The brightness difference is, for example, the absolute value of the difference between the two metering values.

[0254] In the next step ST504, the control unit 48A5 determines whether the brightness difference calculated in step ST502 is greater than or equal to a predetermined brightness difference. The predetermined brightness difference can be a fixed value or a variable value that changes according to given instructions and / or given conditions. In step ST504, if the brightness difference is less than the predetermined brightness difference, the determination is rejected, and the camera support processing proceeds to step ST508. In step ST504, if the brightness difference is greater than or equal to the predetermined brightness difference, the determination is affirmed, and the camera support processing proceeds to step ST506.

[0255] In step ST506, the control unit 48A5 controls the camera device 10 to perform main exposure imaging in the exposure bracket mode for each subject and object category S within the specified imaging range. After performing the processing in step ST506, the imaging support processing proceeds to step ST122. Furthermore, the main exposure image data of each frame obtained by performing main exposure imaging in the exposure bracket mode can be stored individually in a designated storage area, or it can be stored as composite image data of a single frame obtained through compositing in a designated storage area.

[0256] In step ST508, the control unit 48A5 controls the imaging device 10, thereby performing main exposure imaging on the subject within the specified imaging range and the subject category S within the specified imaging range by determining the exposure based on the subject within the specified imaging range. After performing the processing of step ST508, the control unit 48A5 performs steps equivalent to ST120 to ST132 (see reference). Figure 25 After processing, proceed to step ST100.

[0257] Thus, according to Figure 31 As exemplified, when the brightness difference exceeds a specified level, a main exposure is performed on the subject and object category subject S within a designated shooting range using an exposure bracket method. Therefore, compared to shooting only one frame when there is brightness unevenness between the subject and object category subject S within the designated shooting range, an image with less brightness unevenness can be obtained as the subject image and object category subject image S1 within the designated shooting range.

[0258] Furthermore, according to Figure 31 As exemplified, when the brightness difference is less than a specified brightness difference, the main exposure is performed on the subject and the object category subject S within the specified shooting range using an exposure determined based on a reference subject. This reduces the time required for main exposure shooting of the subject and the object category subject S within the specified shooting range after eliminating brightness unevenness between them.

[0259] In the above embodiments, examples of major categories included in the different subject category groups 98 include face category, pose category, eye category, and designated camera range category, but the technology of the present invention is not limited to these. For example, as Figure 32 As shown, within the different subject category groups 98, the broad categories may include a category determined by subject characteristics in "duration" units, namely the period category, and / or a category determined by subject characteristics in "location" units, namely the location category. Here, period refers to the period during which the subject is captured. And location refers to the position where the subject is captured.

[0260] Within the period category, there are multiple year-month-day categories that differ from each other, which are further subcategories. The number of classifications is associated with each of these year-month-day categories. Furthermore, the number of classifications is also associated with the period category. The number of classifications for a period category is the sum of the number of classifications for each of the multiple year-month-day categories.

[0261] Within the location category, there are multiple sub-location categories, each with distinct locations. The number of classifications is associated with each of these sub-location categories. Furthermore, the number of classifications is also associated with the location category itself. The number of classifications for a location category is the sum of the number of classifications for each of its sub-location categories.

[0262] exist Figure 32 In the example, map data 104 is stored in memory 48B. Map data 104 is data that establishes a correspondence between latitude, longitude, and altitude location coordinates and addresses on the map. Map data 104 is referenced by classification unit 48A4. Furthermore, the sub-location category is not limited to addresses on the map; it can also be location coordinates.

[0263] exist Figure 32 In the example, a GPS receiver 108, which serves as both an RTC 106 and a GNSS receiver, is connected to the classification section 48A4. The RTC 106 acquires the current time. The RTC 106 receives power from a power system disconnected from the power system of the controller 48, and continues to count the current time (year, month, day, hour, minute, second) even when the controller 48 is powered off.

[0264] GPS receiver 108 receives radio waves from multiple GPS satellites (not shown), which are examples of multiple GNSS satellites, and calculates the position coordinates that can determine the current position of camera device 10 based on the reception results.

[0265] Each time a main exposure of one frame is performed, the classification unit 48A4 obtains the current time from the RTC 106 and classifies it into the corresponding year-month-day category among multiple year-month-day categories included in the period category, using the obtained current time as the shooting time. Each time the classification unit 48A4 classifies the shooting time into the year-month-day category, it adds "1" to the classification count of the previous year-month-day category after the shooting time classification. Alternatively, while the classification unit 48A4 obtains the current time from the RTC 106, it can also obtain the current time via a communication network such as the Internet.

[0266] Each time a main exposure of one frame is taken, the classification unit 48A4 obtains the location coordinates from the GPS receiver 108 as the location where the shot was taken (hereinafter also referred to as the "camera location"). The classification unit 48A4 determines the address corresponding to the acquired camera location from the map data 104. Then, the classification unit 48A4 classifies the camera location into a sub-location category corresponding to the determined address. Each time the classification unit 48A4 classifies the camera location into a sub-location category, it adds "1" to the classification number of the previous sub-location category after classifying the camera location.

[0267] exist Figure 33 In the example, within the camera support screen 100, a bubble chart related to the period category (hereinafter also referred to as the "period category bubble chart") is displayed as a bubble chart 100A. (See also: Face category bubble chart) Figure 17 Similarly, the category bubble chart during the period has bubbles representing the number of classifications on two axes: one representing multiple identified individuals and the other representing the year, month, and day categories.

[0268] Furthermore, if a location category is selected from the category selection screen 100B of the receiving device 80 (e.g., touch panel 28), a bubble chart related to the location category (hereinafter, location category bubble chart) is displayed as a bubble chart 100A in the camera support screen 100. A bubble chart related to the face category (see reference) Figure 17 Similarly, the location category bubble chart also plots bubbles representing the number of classifications on two axes: one representing multiple identified individuals and the other representing smaller location categories.

[0269] exist Figure 34 In the example, within the camera support screen 100, a histogram related to the period category (hereinafter also referred to as the "period category histogram") is displayed as histogram 100C. The horizontal axis of the period category histogram represents the year, month, and day category, and the vertical axis represents the number of categories. Furthermore, within the camera support screen 100, as histogram 100C, a histogram related to the location category (hereinafter also referred to as the "location category histogram") is also displayed in a way that allows switching between histograms of other categories (illustration omitted). The horizontal axis of the location category histogram represents the sub-location category, and the vertical axis represents the number of categories.

[0270] Thus, within different subject category groups 98, a period category is included as a broad category, determined by the subject characteristics in "period" units. Furthermore, the period category includes multiple year-month-day categories that differ from each other. Then, whenever a one-frame main exposure is captured, the capture time is categorized into a year-month-day category, and a period category bubble chart and period category histogram corresponding to the number of categorizations are displayed on the display 26 along with the live preview image. The period category bubble chart and period category histogram are used in the same way as the face category bubble chart and face category histogram described in the above embodiment. Therefore, according to this structure, the capturing of the camera device 10 can be supported based on the number of categorizations counted by classifying the capture time into year-month-day categories. Additionally, a year-month-day category divided by year, month, and day is shown here, but this is only one example; it could also be a period category divided by year, month, day, hour, minute, or second.

[0271] Furthermore, within the different subject category groups 98, a position category, determined by subject features in "position" units, is included as a major category. Each position category includes multiple minor position categories whose positions differ from each other. Then, whenever a one-frame main exposure is performed, the camera position is classified into a minor position category, and a position category bubble chart and a position category histogram corresponding to the number of classifications are displayed on the display 26 along with the live preview image. The position category bubble chart and position category histogram are used in the same way as the face category bubble chart and face category histogram described in the above embodiment. Thus, according to this structure, the camera device 10 can support shooting based on the number of classifications counted by classifying the camera position into minor position categories.

[0272] In the above embodiments, examples are given assuming that camera support processing is continuously performed during the period when a camera mode is set. However, the technology of the present invention is not limited to this, and camera support processing can also be continuously performed according to time and / or location. For example, such as Figure 35 As shown, in time checkpoints divided at specified time intervals (e.g., 1 hour), camera support processing can be limited to the specified time (e.g., 10 minutes). The specified time interval for determining the time checkpoints can be fixed or varied based on given instructions and / or given conditions (e.g., camera conditions). And, as an example, Figure 35 As shown, among multiple location checkpoints divided by each location, camera support processing can be performed only within a specified time. For example, location checkpoints can be determined using map data 104 and GPS receiver 108.

[0273] Furthermore, in the above embodiments, examples have been given illustrating how the subject features are categorized by the classification unit 48A4, regardless of the scene being filmed by the camera device 10; however, the technology of the present invention is not limited to this. For example, when the scene of the filmed object of the camera device 10 coincides with a specific scene (e.g., a sports meet scene, a beach scene, and a concert scene, etc.), the classification unit 48A4 can categorize the subject features. The specific scene may be a scene filmed in the past.

[0274] In this case, as an example, Figure 36 The camera support processing shown is executed by CPU48A. (And...) Figure 23 Compared to the flowchart shown, Figure 36 The difference in the flowchart shown is that there is a step ST550 between step ST118 and step ST120.

[0275] exist Figure 36 In step ST550 of the camera support processing shown, the subject recognition unit 48A2 identifies the subject within the camera area based on the latest main exposure image data obtained in step ST118 by performing a main exposure shot, thereby determining the current camera scene. Furthermore, the subject recognition unit 48A2 determines past camera scenes based on past main exposure image data (e.g., main exposure image data obtained within a period specified by the user). Then, the subject recognition unit 48A2 determines whether the current camera scene is consistent with a past camera scene. In step ST550, if the current camera scene is inconsistent with a past camera scene, the determination is negative, and the camera support processing proceeds to step ST130. In step ST550, if the current camera scene is consistent with a past camera scene, the determination is positive, and the camera support processing proceeds to step ST120. Therefore, in step ST126, the classification unit 48A4 classifies the subject features into categories according to each subject.

[0276] Thus, only when the current shooting scene is consistent with a specific scene, the classification unit 48A4 classifies the features of each subject into categories, so that the features of the subjects determined from the main exposure image data obtained by shooting a main exposure of a scene that is not desired by the user are not classified into categories.

[0277] Furthermore, only when the current shooting scene is consistent with the past shooting scene, the classification unit 48A4 classifies the subject features into categories according to each subject. Therefore, it is possible to determine the subject features into categories from the main exposure image data obtained by taking a main exposure shot of the current shooting scene consistent with the past shooting scene.

[0278] Furthermore, in the above embodiments, examples of classifying subject features into multiple categories have been given, but the technology of the present invention is not limited thereto. For example, such as Figure 37 As shown, the main exposure image obtained by performing a main exposure photograph can be classified into each of multiple categories by the classification unit 48A4. In this case, whenever the main exposure image is classified into a category, "1" is added to the classification number of the category after the main exposure image is classified. The different subject category groups 98 constructed in this way are also used in the same way as the different subject category groups 98 described in the above embodiment. Thus, according to this structure, the imaging device 10 can support the shooting based on the number of classifications of the main exposure image into categories.

[0279] Furthermore, in the above embodiment, a face category histogram related to person A is shown (see reference). Figure 18 However, the technology of this invention is not limited thereto. For example, such as Figure 38 As shown, a four-quadrant face category bubble chart can also be used instead of a face category histogram. In the four-quadrant face category bubble chart, the smiling face category is assigned to the first quadrant, the crying face category to the second quadrant, the angry face category to the third quadrant, and the serious face category to the fourth quadrant. Then, the number of classifications corresponding to each category is represented by the size of the bubble. Additionally, as... Figure 38 As shown, when displayed using four quadrant categories, facial expressions can be categorized more finely, and a scatter plot can be used instead of a bubble chart. In this case, by adjusting the drawing position of points based on the categorized facial expressions, the desired facial expression can be represented. For example, even within the same smile category, the closer an image is to a smiling face that resembles a crying face, the closer it is to the left side of the first quadrant's graph. Furthermore, the closer an image is to a smiling face that resembles a serious face, the closer it is to the lower part of the first quadrant's graph. Based on this scatter plot, users can gain a more detailed understanding of what kind of face of the person was captured.

[0280] Furthermore, in the above embodiments, bubble chart 100A and histogram 100C are shown as examples, but the technology of the present invention is not limited to these. Other tables may be used, or numerical values ​​representing the number of classifications may be displayed in a form that divides each subject and each category.

[0281] Furthermore, in the above embodiments, examples of shooting forms that support categories related to the number of times the user selects a category from the histogram 100C have been described, but the technology of the present invention is not limited to this, and the categories that can be shot can also be directly selected by the user from the histogram 100C, etc., via the receiving device 80 (e.g., touch panel 28).

[0282] Furthermore, while the above embodiment exemplifies storing the main exposure image data in image memory 50, data including the main exposure image data obtained by performing a main exposure photograph can also be used as training data in the machine learning of the learned model 92, wherein the main exposure photograph is supported by the aforementioned support processing. Thus, a learned model 92 based on the main exposure image data obtained by performing a main exposure photograph, wherein the main exposure photograph is supported by the support processing, can be created.

[0283] Furthermore, in the above embodiments, examples of camera support processing being performed by the controller 48 within the camera device 10 have been described, but the technology of the present invention is not limited thereto. For example, such as Figure 40 As shown, camera support processing can be performed by a computer 114 within an external device 112 that is communicatively connected to the camera device 10 via a network 110 such as a LAN or WAN. Figure 40 In the example, computer 114 includes CPU 116, memory 118, and RAM 120. A category database 96 is constructed in memory 118, and a camera support processing program 84 is stored therein.

[0284] The camera device 10 requests the external device 112 to perform camera support processing via network 110. Accordingly, the CPU 116 of the external device 112 reads the camera support processing program 84 from memory 118 and executes the camera support processing program 84 on memory 120. The CPU 116 performs camera support processing according to the camera support processing program 84 executed on memory. Then, the CPU 116 provides the processing result obtained by performing the camera support processing to the camera device 10 via network 110.

[0285] Furthermore, the camera support processing can be performed separately by the camera device 10 and the external device 112, or it can be performed separately by multiple devices including the camera device 10 and the external device 112. In the case of distributed processing, for example, the CPU 48A of the camera device 10 can operate as the acquisition unit 48A1 and the control unit 48A5, while the CPU of a device other than the camera device 10 (e.g., the external device 112) can operate as the subject recognition unit 48A2, the feature extraction unit 48A3, and the classification unit 48A4. That is, by having an external device with higher computing power than the camera device 10 handle the relatively large processing load, the processing load applied to the camera device 10 can be reduced.

[0286] Furthermore, while a still image is exemplified as the main exposure image in the above embodiments, the technology of the present invention is not limited to this, and a moving image may also be used as the main exposure image. The moving image may be a recording moving image or a display moving image, that is, a real-time preview image or a later viewing image.

[0287] Furthermore, in the above embodiments, camera recommendation information is displayed within the live preview image, but it may not necessarily be displayed within the live preview image. For example, when performing a main exposure shot of a moving image, similar to the camera recommendation information, a display indicating the subject category (at least one of arrows, face frames, and messages) can be shown. Thus, when performing a main exposure shot of a moving image, the display indicating the subject category allows the user to identify the subject category included in the moving image using the camera device 10. Furthermore, the user can use the camera device 10 to cut out frames including the subject category after the main exposure shot of the moving image, thereby acquiring a still image including the subject category. In this case, whenever the camera device 10 determines the subject category through the main exposure shot of the moving image, it performs the same category classification as described above, and can update and display histograms and / or bubble charts, etc. Therefore, the user can understand what kind of subject is included simply by using the camera device 10 to capture a moving image. Furthermore, in this case, values ​​based on still images and values ​​based on moving images can be displayed in different ways in histograms and / or bubble charts. For example, in a histogram, the histogram based on still images and the histogram based on moving images can be displayed in different colors according to a stacked bar chart. Thus, the user can understand which of the still images or moving images each classification is based on. Moreover, the aforementioned histograms and / or bubble charts can be created solely based on the classification of the subject represented by the subject area included in a moving image. Thus, the user can understand which category the subject area included in that moving image belongs to. Additionally, such histograms and / or bubble charts can be displayed on the display 26 based on user operations, etc., after the still image or moving image has been captured, or in a playback mode where a real-time preview image is not displayed.

[0288] Furthermore, while examples of continuously increasing classification counts have been described in the above embodiments, the technology of the present invention is not limited thereto. For example, the classification counts corresponding to at least one category included in different subject category groups 98 can be reset periodically or at specified times. For example, they can be reset according to time and / or location. Specifically, they can be reset once a day, once an hour, or every 100 meters when the location changes.

[0289] Furthermore, in the above embodiments, the smiley face category is exemplified as the object category, but the technology of the present invention is not limited to this; other categories may also be used as object categories, and multiple categories may be used as object categories. In this case, for example, in Figure 18 In the face category histogram shown, users can select the number of categories or multiple categories associated with each category (e.g., smiling face category and crying face category).

[0290] Furthermore, in the above embodiment, person A is exemplified as the subject of the object category, but the technology of the present invention is not limited to this, and there can be multiple subjects of the object category. In this case, for example, in Figure 17 In the face category bubble chart shown, the user can select multiple people (e.g., person A, person B, and person C). Therefore, for example, if the object category is a smiley face, the various support processes described above will be performed if at least one of the multiple people selected by the user is a smiley face.

[0291] Furthermore, in the above embodiments, the example illustrates the simple number of times the subject features are classified into categories, i.e., the number of classifications, but the technology of the present invention is not limited to this. For example, it could be the number of classifications per unit time.

[0292] Furthermore, in the above embodiment, a detection frame 102 (see reference) is shown as an example. Figure 19 and Figure 21 However, the technology of the present invention is not limited to this, and information related to the subject surrounded by the detection frame 102 (e.g., name and / or category name, etc.) can also be displayed together with the detection frame 102.

[0293] Furthermore, while a physical camera (hereinafter also referred to as a "physical camera") is exemplified as the imaging device 10 in the above embodiments, the technology of the present invention is not limited thereto. A virtual camera may also be used instead of a physical camera. This virtual camera virtually captures the subject from a virtual viewpoint based on image data obtained by multiple physical cameras positioned at different locations, thereby generating virtual viewpoint image data. In this case, the image represented by the virtual viewpoint image data, i.e., the virtual viewpoint image, is an example of the "image captured by the present invention."

[0294] In the above embodiments, examples of dividing the region 30N with non-phase difference pixels and dividing the region 30P with phase difference pixels have been described, but the technology of the present invention is not limited thereto. For example, instead of dividing the region 30N with non-phase difference pixels and dividing the region 30P with phase difference pixels, a region sensor can be configured to selectively generate and read phase difference image data and non-phase difference image data. In this case, a plurality of photosensitive pixels are arranged in a two-dimensional manner in the region sensor. As the photosensitive pixels included in the region sensor, for example, a separate pair of photodiodes without light-shielding components are used. When generating and reading non-phase difference image data, photoelectric conversion is performed through the entire area of ​​the photosensitive pixels (the pair of photodiodes), and when generating and reading phase difference image data (for example, in the case of passive ranging), photoelectric conversion is performed through one of the photodiodes in the pair of photodiodes. Here, one of the photodiodes in the pair of photodiodes is the photodiode corresponding to the first phase difference pixel L described in the above embodiments, and one of the photodiodes in the pair of photodiodes is the photodiode corresponding to the second phase difference pixel R described in the above embodiments. Alternatively, phase difference image data and non-phase difference image data can be selectively generated and read from all photosensitive pixels included in the area sensor, but this is not a limitation; phase difference image data and non-phase difference image data can also be selectively generated and read from a subset of photosensitive pixels included in the area sensor.

[0295] In the above embodiments, an image plane phase difference pixel is exemplified as a phase difference pixel P, but the technology of the present invention is not limited thereto. For example, a non-phase difference pixel N may be configured instead of the phase difference pixel P included in the photoelectric conversion element 30, or a phase difference AF plate including multiple phase difference pixels P may be separately disposed from the photoelectric conversion element 30 in the main body 12 of the imaging device.

[0296] In the above embodiments, an AF method utilizing ranging results based on phase difference image data, namely a phase difference AF method, is illustrated, but the technology of the present invention is not limited to this. For example, a contrast AF method can be used instead of a phase difference AF method. Furthermore, an AF method based on ranging results using the parallax of a pair of images obtained from a stereo camera, or an AF method utilizing ranging results based on a time-of-flight (TOF) method such as a laser beam, can be used.

[0297] In the above embodiments, a focal plane shutter has been described as an example of a mechanical shutter 72, but the technology of the present invention is not limited thereto. Even if other types of mechanical shutters such as lens shutters are used instead of focal plane shutters, the technology of the present invention is still valid.

[0298] In the above embodiments, examples of storing a camera support processing program 84 in memory 48B have been described, but the technology of the present invention is not limited thereto. For example, such as Figure 41 As shown, the camera support processing program 84 can be stored in the storage medium 200. The storage medium 200 is a non-temporary storage medium. As an example of the storage medium 200, any portable storage medium such as an SSD or USB memory can be cited.

[0299] The camera support processing program 84 stored in the storage medium 200 is installed in the controller 48. The CPU 48A executes camera support processing according to the camera support processing program 84.

[0300] Furthermore, the camera support processing program 84 may be stored in the storage unit of other computers or server devices connected to the controller 48 via a communication network (not shown), and the camera support processing program 84 may be downloaded and installed in the controller 48 upon request from the camera device 10.

[0301] In addition, it is not necessary to store all the camera support processing programs 84 in the storage unit or memory 48B of other computers or server devices connected to the controller 48, but only a part of the camera support processing programs 84 can be stored.

[0302] exist Figure 41 The example shown illustrates a form in which the controller 48 is built into the camera device 10, but the technology of the present invention is not limited thereto. For example, the controller 48 may also be located outside the camera device 10.

[0303] exist Figure 41 In the example, CPU48A is a single CPU, but it can also be multiple CPUs. Furthermore, a GPU can be used instead of CPU48A.

[0304] exist Figure 41 The example illustrates a controller 48, but the technology of the present invention is not limited thereto, and devices including ASICs, FPGAs, and / or PLDs can be used instead of controller 48. Furthermore, a combination of hardware and software structures can be used instead of controller 48.

[0305] As the hardware resource for performing the camera support processing described in the above embodiments, various processors as shown below can be used. For example, a general-purpose processor, such as a CPU, can be used, which functions as a hardware resource for performing camera support processing by executing software, i.e., a program. Furthermore, processors with circuit structures specifically designed for performing particular processing, such as FPGAs, PLDs, or ASICs, can also be used as processors, i.e., dedicated circuits. Memory is also built into or connected to any processor, and any processor performs camera support processing using memory.

[0306] The hardware resources for performing camera support processing can consist of one of these various processors, or a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for performing camera support processing can be a single processor.

[0307] As an example of a single processor, firstly, there is a method where a processor is constructed by combining one or more CPUs and software, with the processor functioning as a hardware resource for performing camera support processing. Secondly, there is a method, such as SoC (System-on-a-Chip), where a processor, implemented by a single IC chip, performs the functions of an entire system including multiple hardware resources for performing camera support processing. In this way, camera support processing is implemented using one or more of the aforementioned processors as hardware resources.

[0308] Furthermore, the hardware architecture of these various processors, more specifically, can utilize circuits composed of circuit elements such as semiconductor components. And the aforementioned camera support processing is merely one example. Therefore, without departing from the main point, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order.

[0309] The above descriptions and illustrations are detailed explanations of the parts involved in the technology of this invention, and are merely one example of the technology of this invention. For example, the descriptions related to the above structure, function, effect, and effect are examples of the structure, function, effect, and effect of the parts involved in the technology of this invention. Therefore, without departing from the spirit of the technology of this invention, unnecessary parts may be deleted, new elements may be added, or substitutions may be made to the above descriptions and illustrations. Furthermore, in order to avoid complicating the technology involved in this invention and to facilitate understanding of the parts involved in the technology of this invention, descriptions related to common technical knowledge that do not require special explanation in aspects where the technology of this invention can be implemented have been omitted from the above descriptions and illustrations.

[0310] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that it can be just A, just B, or a combination of A and B. Furthermore, in this specification, when "and / or" is added to represent more than three items, the same concept as "A and / or B" may also be applied.

[0311] All documents, patent applications and technical standards described in this specification are referenced in this specification to the same extent that each document, patent application and technical standard is specifically and separately described and referenced by reference.

[0312] The following notes are also disclosed regarding the above implementation methods.

[0313] (Postscript 1)

[0314] A camera support device comprising:

[0315] Processor; and

[0316] Memory, connected to or built into the aforementioned processor,

[0317] The processor acquires frequency information, which represents the frequency of features that classify a subject into categories based on features determined from a photographic image captured by a camera device.

[0318] Based on the aforementioned frequency information, support processing is performed to enable the aforementioned camera device to capture images.

[0319] (Postscript 2)

[0320] According to the camera support device described in Appendix 1, wherein,

[0321] The above categories are classified into multiple categories that include at least one object category.

[0322] The above object categories are determined based on the frequency information mentioned above.

[0323] The aforementioned support processing includes the processing of supporting the above-mentioned shooting of subjects that have the above-mentioned characteristics belonging to the above-mentioned object category.

[0324] (Note 3)

[0325] According to the camera support device described in Appendix 2, wherein,

[0326] The above support processing includes the following: display processing for recommending the shooting of subjects of the above object categories.

[0327] (Note 4)

[0328] According to the camera support device described in Appendix 3, wherein...

[0329] The above display processing is as follows: displaying a display image on a monitor, and displaying a frame that surrounds at least a portion of the object category subject image within the display image.

[0330] (Note 5)

[0331] The camera support device according to any one of Appendices 2 to 4, wherein,

[0332] The processor described above performs the following processing:

[0333] The above-mentioned object category is detected based on the imaging results of the aforementioned camera device.

[0334] Under the condition that the above-mentioned object category of the subject is detected, an image including the image corresponding to the above-mentioned object category of the subject is acquired.

[0335] (Note 6)

[0336] According to the camera support device described in Appendix 5, wherein...

[0337] The processor, upon detecting that the object category is included within a specified imaging range, causes the imaging device to perform the aforementioned shooting accompanied by a main exposure.

[0338] (Note 7)

[0339] The camera support device according to any one of Appendices 2 to 6, wherein,

[0340] When the subject of the aforementioned object category is located outside the designated camera range determined according to instructions given from the outside, the processor controls the camera device to include the designated camera range and the subject of the aforementioned object category in the depth of field.

[0341] (Note 8)

[0342] According to the camera support device described in Appendix 7, wherein...

[0343] When the specified camera range and the subject of the specified object category are not included in the depth of field due to the structure of the camera device, the processor causes the camera device to capture the specified camera range and the subject of the specified object category in a focusing bracket manner.

[0344] (Note 9)

[0345] According to the camera support device described in Appendix 7 or 8, wherein,

[0346] When the object category subject is located within the specified camera range, the processor causes the camera device to capture the object category subject while it is focused on the object category subject.

[0347] (Postscript 10)

[0348] The camera support device according to any one of Appendices 2 to 9, wherein,

[0349] When the difference between the brightness of the reference subject and the brightness of the object category subject is greater than or equal to a predetermined difference, the processor causes the camera device to capture images of the reference subject and the object category subject in an exposure bracket manner.

[0350] (Postscript 11)

[0351] According to the camera support device described in Appendix 10, wherein,

[0352] When the difference is less than the specified difference, the processor causes the camera device to capture the reference subject and the object category subject by means of an exposure determined based on the object category subject.

[0353] (Postscript 12)

[0354] The camera support device according to any one of Appendices 1 to 11, wherein,

[0355] The images obtained by taking the above-described images are used for learning, and the above-described images are supported by the above-described support processing.

Claims

1. A camera support device, comprising: Processor; and Memory, connected to or built into the processor, The processor acquires frequency information, which represents the frequency of features that classify a subject into categories based on features determined from photographic images captured by a camera device. Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

2. The camera support device according to claim 1, wherein, Each of the plurality of subcategories is associated with a frequency information relating to the feature classified as the corresponding subcategory.

3. The camera support device according to claim 1 or 2, wherein, The major category is associated with the frequency information of the feature that is classified into the major category.

4. The camera support device according to claim 1 or 2, wherein, Each of the plurality of subcategories is associated with the frequency information of the feature classified into the corresponding subcategory. The major category is associated with the frequency information of the feature classified into the major category. The frequency information associated with the major category is the sum of the frequency information associated with each of the multiple minor categories.

5. The camera support device according to claim 1, wherein, The support processing includes the following: display processing for recommending the display of the subject of the object category.

6. The camera support device according to claim 5, wherein, The display processing is as follows: displaying the display image obtained by the camera device on the display screen, and displaying the object category subject image representing the object category subject within the display image in a manner distinguishable from other image areas.

7. The camera support device according to any one of claims 1, 5, or 6, wherein, The processor performs the following processing: The object category of the photographed subject is detected based on the imaging results of the camera device. Under the condition that the object category subject is detected, an image including the image corresponding to the object category subject is acquired.

8. The camera support device according to any one of claims 1, 5, or 6, wherein, The processor displays objects representing a specified camera range determined according to externally given instructions and objects representing the subject category of the object in different display modes.

9. The camera support device according to any one of claims 1, 5, or 6, wherein, If the difference between the first camera condition imposed from the outside and the second camera condition imposed on the subject of the object category is greater than a specified difference, the processor performs specified processing.

10. The camera support device according to any one of claims 1, 5, or 6, wherein, The object category is a low-frequency category with a relatively low frequency among the multiple subcategories.

11. The camera support device according to any one of claims 1, 5, or 6, wherein, When the object category is captured by the camera device, the object category is a category determined based on the state of the object category subject, and is a category classified with respect to the characteristics of the object category subject.

12. The camera support device according to any one of claims 1, 5, or 6, wherein, When multiple objects are captured by the camera device, the object category is the object category that can determine each of the multiple objects.

13. The camera support device according to any one of claims 1, 5, or 6, wherein, The categories are created at least per unit.

14. The camera support device according to claim 13, wherein, One of the units mentioned is the period.

15. The camera support device according to claim 13, wherein, One of the units mentioned is location.

16. The camera support device according to any one of claims 1, 5, or 6, wherein, The processor enables the classifier to classify the features. When the scene of the object being filmed by the camera device is consistent with a specific scene, the classifier classifies the features.

17. The camera support device according to claim 16, wherein, The specific scene mentioned is one that was filmed in the past.

18. The camera support device according to any one of claims 1, 5, or 6, wherein, The support processing includes the following: processing for displaying the frequency information.

19. The camera support device according to claim 18, wherein, The support processing includes, when the frequency information is displayed and the receiving device specifies the frequency information, supporting the shooting process related to the category corresponding to the specified frequency information.

20. A camera support device, comprising: Processor; and Memory, connected to or built into the processor, The processor acquires frequency information, which represents the frequency of the camera images classified into categories based on features of the subjects included in the camera images captured by the camera device. Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

21. A camera device comprising: The camera support device according to any one of claims 1 to 20; and Image sensor, The processor supports the capture using the image sensor by performing the support processing.

22. A camera support method, comprising the following processing: Acquire frequency information, which represents the frequency of features of a subject classified into categories based on features determined from video images captured by a camera device; and Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

23. A camera support method, comprising the following processing: Acquire frequency information, the frequency information representing the frequency of the camera images classified into categories based on features of the subject determined from camera images captured by a camera device; and Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

24. A storage medium storing a program for causing a computer to perform the following processes: Acquire frequency information, which represents the frequency of features of a subject classified into categories based on features determined from video images captured by a camera device; and Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

25. A storage medium storing a program for causing a computer to perform the following processes: Acquire frequency information, the frequency information representing the frequency of the camera images classified into categories based on features of the subject determined from camera images captured by a camera device; and Based on the frequency information, support processing is performed to enable the camera device to capture images. The categories consist of a major category and multiple subcategories belonging to the major category. The features belonging to the feature classified into the major category are classified into the multiple minor categories. The major category and the multiple minor categories include at least one object category. The object category is determined based on the frequency information. The support processing includes processing that supports the shooting of subjects with the features belonging to the object category.

Images