Image processing apparatus, image processing method, and program

Abstract

To allow a user to freely set a focus detection area to an area in a detectable subject.SOLUTION: An image processing apparatus has: detection means that detects a plurality of types of vehicles; acquisition means that acquires information on at least one of a photographing direction relative to the vehicle and an inclination direction of the vehicle; and setting means that sets an area to be focused. The setting means changes the area to be focused according to the types of the vehicles detected by the detection means and the information acquired by the acquisition means.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] It is necessary for a user (photographer) to set an area in which focus detection is desired within a subject detected by an imaging device and track the subject. As a related technique, the technique of Patent Document 1 has been proposed. In the technique of Patent Document 1, when the subject is a human face, the eyes in the face are detected, the size of the detected eyes is determined, and the focus detection area is set to the eyes or face. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-123301 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1 mentioned above, when the subject is a person, the focus detection area can be set to the person's eyes or face. However, depending on the shooting scene, there is a risk that the focus detection area will be set to an area other than that intended by the user for different subjects.

[0005] An object of the present invention is to set a focus detection area to an area that a user desires to set for a detectable subject. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the image processing device of the present invention has a detection means for detecting multiple types of vehicles, an acquisition means for acquiring information regarding at least one of the shooting direction relative to the vehicle or the inclination direction of the vehicle, and a setting means for setting a region to be focused, and is characterized in that the setting means changes the region to be focused depending on the type of vehicle detected by the detection means and the information acquired by the acquisition means. Effect of the Invention

[0007] According to the present invention, for a detectable subject, the focus detection area can be set to an area that the user desires to set. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment of the present invention. [Diagram 2] 1 is a diagram showing a pixel array of an imaging element in an imaging device according to a first embodiment. [Diagram 3] FIG. 3A is a plan view of an imaging pixel in the imaging element of the first embodiment, and FIG. 3B is a cross-sectional view of the imaging pixel in the imaging element of the first embodiment. [Figure 4] 2 is a diagram for explaining the structure of an imaging pixel in the image sensor of the first embodiment. FIG. [Diagram 5] FIG. 2 is a diagram for explaining pupil division by the image sensor of the first embodiment. [Figure 6] 5A and 5B are diagrams for explaining the relationship between the defocus amount and the image shift amount in the first embodiment. [Figure 7] FIG. 2 is a diagram for explaining a focus detection area in the first embodiment. [Figure 8] 5 is a flowchart showing the flow of live view shooting in the imaging apparatus according to the first embodiment. [Figure 9] 4 is a flowchart showing the flow of imaging processing in the first embodiment. [Figure 10] 5 is a flowchart showing the flow of subject tracking AF processing in the first embodiment. [Figure 11] 5 is a flowchart showing the flow of subject detection processing and tracking processing in the first embodiment. [Figure 12] 5 is a flowchart showing the flow of focus area detection processing in the first embodiment. [Figure 13] 5 is a flowchart showing the flow of focus detection area setting processing in the first embodiment. [Figure 14] 14A, 14B, 14C, and 14D are diagrams for explaining the focus region detected in the focus region detection process of the first embodiment. [Figure 15] 4A to 4C are diagrams showing examples of scenes in which detection as a focus area in the first embodiment may be effective. [Figure 16] 5 is a flowchart showing the flow of predictive AF processing according to the first embodiment. [Figure 17] 5A and 5B are diagrams for explaining the image plane movement amount of a subject and a predicted curve in the first embodiment. [Figure 18] 18(A), 18(B), 18(C), 18(D), 18(E) and 18(F) are diagrams for explaining the focus movable range in the first embodiment. [Figure 19] FIG. 2 is a diagram for explaining a focus movable range in the first embodiment. [Figure 20] FIG. 4 is a diagram for explaining predictive calculation and items to be changed during focus control in the first embodiment. [Figure 21] 10 is a flowchart showing the flow of focus detection area setting processing according to a second embodiment of the present invention. [Figure 22] 22A and 22B are conceptual diagrams for explaining local regions and focus detection candidate regions in the second embodiment. [Figure 23] FIGS. 23A, 23B, 23C, and 23D are conceptual diagrams for explaining the setting of focus detection candidate areas in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0010] First Embodiment Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the configuration of an imaging device (camera 100) according to the first embodiment of the present invention.

[0011] 1, the camera 100 has an imaging optical system, a zoom actuator 111, an aperture actuator 112, a focus actuator 114, an electronic flash 115, an AF assist light emitting unit 116, an image sensor 107, and a shutter 108. The camera 100 also has a CPU 121, an electronic flash control circuit 122, an assist light drive circuit 123, an image sensor drive circuit 124, an image processing circuit 125, a focus drive circuit 126, an aperture drive circuit 128, and a zoom drive circuit 129. The camera 100 further has a display 131, a group of operation switches 132, a flash memory 133, a subject detection unit 140, a dictionary data storage unit 141, and a focus area detection unit 142.

[0012] The first lens group 101, the aperture 102, the second lens group 103, the third lens group 105, and the optical low-pass filter 106 constitute an imaging optical system. The first lens group 101 is disposed closest to the subject (front side) in the imaging optical system as an image forming optical system, and is held so as to be movable in the optical axis direction. The aperture 102 adjusts the amount of light by adjusting its aperture diameter. The second lens group 103 moves in the optical axis direction together with the aperture 102, and performs magnification change (zooming) together with the first lens group 101 moving in the optical axis direction. The third lens group (focus lens) 105 moves in the optical axis direction to perform focus adjustment. The optical low-pass filter 106 is an optical element for reducing false colors and moire in a captured image.

[0013] The zoom actuator 111 rotates a cam barrel (not shown) around the optical axis, and moves the first lens group 101 and the second lens group 103 in the optical axis direction by a cam provided on the cam barrel, thereby varying the magnification. The aperture actuator 112 drives a plurality of light-shielding blades (not shown) in opening and closing directions to adjust the amount of light of the aperture 102. The focus actuator 114 moves the third lens group 105 in the optical axis direction to adjust the focus.

[0014] The focus driving circuit 126 drives the focus actuator 114 in response to a focus driving command from the CPU 121, and moves the third lens group 105 in the optical axis direction. The aperture driving circuit 128 drives the aperture actuator 112 in response to an aperture driving command from the CPU 121. The zoom driving circuit 129 drives the zoom actuator 111 in response to a zoom operation by the user.

[0015] In the first embodiment, a case will be described in which the imaging optical system, zoom actuator 111, aperture actuator 112, focus actuator 114, focus drive circuit 126, aperture drive circuit 128, and zoom drive circuit 129 are provided integrally with the camera body. The camera body also includes an image sensor 107. However, an interchangeable lens having the imaging optical system, zoom actuator 111, aperture actuator 112, focus actuator 114, focus drive circuit 126, aperture drive circuit 128, and zoom drive circuit 129 may be detachable from the camera body.

[0016] The electronic flash 115 has a light-emitting element such as a xenon tube or an LED, and emits light to illuminate the subject. The AF auxiliary light emission unit 116 has a light-emitting element such as an LED, and improves focus detection performance for dark or low-contrast subjects by projecting an image of a mask having a predetermined opening pattern onto the subject via a projection lens. The electronic flash control circuit 122 also controls the electronic flash 115 to light up in synchronization with the imaging operation. The auxiliary light drive circuit 123 controls the AF auxiliary light emission unit 116 to light up in synchronization with the focus detection operation.

[0017] The CPU 121 is responsible for various controls in the camera 100. The CPU 121 has a calculation unit, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, etc. The CPU 121 executes a computer program stored in the ROM to drive various circuits in the camera 100 and control a series of processes (operations) such as AF processing, imaging processing, image processing, and recording. The CPU 121 functions as an image processing device. The image processing device may include a subject detection unit 140, a dictionary data storage unit 141, a focus area detection unit 142, etc., in addition to the CPU 121.

[0018] The image sensor 107 is composed of a two-dimensional CMOS photosensor including a plurality of pixels and its peripheral circuits, and is disposed on the imaging plane of the imaging optical system. The image sensor 107 photoelectrically converts the subject image formed by the imaging optical system. The image sensor drive circuit 124 controls the operation of the image sensor 107, and transmits to the CPU 121 a digital signal obtained by A / D converting an analog signal generated by the photoelectric conversion.

[0019] The shutter 108 has a focal plane shutter configuration, and drives the focal plane shutter in response to a command from a shutter drive circuit built into the shutter 108 based on an instruction from the CPU 121. The image sensor 107 is shielded from light while a signal from the image sensor 107 is being read out. Furthermore, when exposure is being performed, the focal plane shutter is opened and a photographing light beam is guided to the image sensor 107.

[0020] The image processing circuit (image processing unit) 125 applies predetermined image processing to image data stored in the RAM in the CPU 121. The image processing applied by the image processing circuit 125 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation processing (demosaic processing), and gamma correction processing, as well as signal format conversion processing and scaling processing.

[0021] Furthermore, image processing circuit 125 determines the main subject based on posture information of the subject and position information of an object specific to the scene (hereinafter referred to as a "specific object"). The result of the determination process performed by image processing circuit 125 may be used for other image processing (for example, white balance adjustment processing). Image processing circuit 125 stores the processed image data, the joint positions of each subject, the position and size information of the specific object, the center of gravity of the subject determined to be the main subject, position information of the face and eyes, etc. in the RAM in CPU 121.

[0022] The display (display means) 131 has a display element such as an LCD (Liquid Crystal Display) and displays information about the imaging mode of the camera 100, a preview image before imaging, a confirmation image after imaging, an index of the focus detection area, and a focused image. The operation switch group 132 includes a main switch (power switch), a release switch (photography trigger switch), a zoom operation switch, a photography mode selection switch, and the like, and is operated by the user. The flash memory 133 records captured images. The flash memory 133 is detachable from the camera 100.

[0023] The subject detection unit 140 as a subject detection means performs subject detection processing based on dictionary data for subject detection generated by machine learning. In the first embodiment, the subject detection unit 140 uses dictionary data for subject detection for each subject in order to detect a plurality of types of subjects. Each dictionary data for subject detection is, for example, data in which the features of the corresponding subject are registered. The subject detection unit 140 performs subject detection while sequentially switching the dictionary data for subject detection for each subject. In the first embodiment, the dictionary data for subject detection for each subject is stored in the dictionary data storage unit 141. Therefore, a plurality of dictionary data for subject detection are stored in the dictionary data storage unit 141. The CPU 121 determines which of the plurality of dictionary data for subject detection to use for subject detection based on the subject priority set in advance and the settings of the camera 100 (imaging device).

[0024] The focus area detection unit 142 as a focus area detection means detects an area to be focused (an area to be in focus) in an area of ​​a subject based on dictionary data for focus area detection generated by machine learning. In the first embodiment, the focus area detection unit 142 receives at least an image signal of an area (hereinafter referred to as a "subject detection area") of a subject (hereinafter referred to as a "detected subject") detected by the subject detection unit 140 as an input, and obtains a focus area as an output. The focus area is an area in a detected subject to be focused. In the first embodiment, the dictionary data for focus area detection for each subject is stored in the dictionary data storage unit 141. Therefore, a plurality of dictionary data for focus area detection are stored in the dictionary data storage unit 141. The dictionary data for focus area detection used by the focus area detection unit 142 associated with the dictionary data for subject detection used by the subject detection unit 140 is selected and used. Details will be described later.

[0025] Dictionary data storage unit 141, which serves as a storage means, stores dictionary data for subject detection and dictionary data for focus region detection for each subject. Subject detection unit 140 estimates the position of the subject in the image based on captured image data and dictionary data for subject detection. Subject detection unit 140 may also estimate information such as the position, size, and reliability of the subject, and output the estimated information. Furthermore, subject detection unit 140 may output other information. Similarly, focus region detection unit 142, as described above, uses image data of the subject detection region as an input image, and outputs a region to be focused on (focus region) in the input image based on dictionary data for focus region detection.

[0026] The subject detection dictionary data used by the subject detection unit 140 includes, for example, person dictionary data for detecting a "person" as a subject, animal dictionary data for detecting an "animal", vehicle dictionary data for detecting a "vehicle", etc. Furthermore, dictionary data for detecting the "whole person" and dictionary data for detecting the "person's face" may be stored separately in the dictionary data storage unit 141.

[0027] The dictionary data for focus area detection is, for example, dictionary data that outputs the area of ​​the head of the driver of the vehicle or the area of ​​the side of the housing of the vehicle depending on the size of the subject and the shooting settings when a "vehicle" is detected as the subject and is used as the input image. The focus area detection unit 142 uses the dictionary data for focus area detection. In this way, in the present invention, by outputting the area to be focused on (focus area) separately from the subject detection area, an image focused on an appropriate area depending on the shooting scene can be obtained. Details will be described later.

[0028] In the first embodiment, the subject detection unit 140 is configured by a CNN (convolutional neural network) that has been machine-learned (deep learning), and estimates the position of a subject included in captured image data, etc. In addition, the focus region detection unit 142 is configured by a CNN (hereinafter referred to as a "trained CNN") that has been machine-learned (deep learning), and estimates the position to be focused on within a region of a detected subject, etc. In the first embodiment, the subject detection unit 140 and the focus region detection unit 142 are each configured by a CNN that has been machine-learned using a different method. The subject detection unit 140 and the focus region detection unit 142 may be realized by a GPU (graphics processing unit) or a circuit specialized for estimation processing using CNN.

[0029] In the present invention, the machine learning of CNN may be performed by any method. For example, a predetermined computer such as a server may perform machine learning of CNN to generate a trained CNN (i.e., a trained model), and the camera 100 may acquire the trained CNN from the predetermined computer. For example, the predetermined computer may perform machine learning of CNN of the subject detection unit 140 by performing supervised learning using image data for learning as input and a position of a subject corresponding to the image data for learning as teacher data. In addition, the predetermined computer may perform machine learning of CNN of the focus region detection unit 142 by performing supervised learning using image data for learning as input and a position to focus corresponding to the subject of the image data for learning as teacher data. In this way, the trained CNN (trained model) of the subject detection unit 140 and the focus region detection unit 142 is generated.

[0030] Here, as described above, the subject detection unit 140 detects a subject using dictionary data for subject detection. Also, the subject detection unit 140 detects a subject using dictionary data for subject detection for different types of subjects (people, animals, vehicles, etc.). In the first embodiment, each dictionary data for subject detection used by the subject detection unit 140 is generated by applying a trained CNN constituting the subject detection unit 140. Also, the focus region detection unit 142 detects a focus region using dictionary data for focus region detection. Each dictionary data for focus region detection used by the focus region detection unit 142 is also generated by applying a trained CNN constituting the focus region detection unit 142.

[0031] Furthermore, the machine learning of CNN may be performed by the camera 100 (imaging device) or the CPU 121 (image processing device).

[0032] As described above, in the first embodiment, the subject detection unit 140 and the focus region detection unit 142 are each configured by a different machine-learned CNN. However, the present invention is not limited to this, and the subject detection unit 140 and the focus region detection unit 142 may each be configured by a different machine-learned neural network. In addition, the subject detection unit 140 and the focus region detection unit 142 may be configured by a trained model other than the trained CNN. For example, the subject detection unit 140 and the focus region detection unit 142 may be configured by a trained model that is trained by any machine learning algorithm such as a support vector machine or a logistic regression.

[0033] Next, the pixel array of the image sensor 107 of the imaging device (camera 100) according to the first embodiment will be described with reference to Fig. 2. Fig. 2 shows a pixel array of the image sensor 107 in a range of 4 pixel columns x 4 pixel rows as viewed from the optical axis direction (hereinafter referred to as the "z direction").

[0034] As shown in FIG. 2, one pixel unit 200 includes four imaging pixels arranged in two rows and two columns. By arranging a large number of pixel units 200 on the image sensor 107, photoelectric conversion of a two-dimensional subject image can be performed. In one pixel unit 200, an imaging pixel (hereinafter referred to as an "R pixel") 200R having a spectral sensitivity of R (red) is arranged at the upper left, and imaging pixels (hereinafter referred to as "G pixels") 200G having a spectral sensitivity of G (green) are arranged at the upper right and lower left, respectively. Furthermore, an imaging pixel (hereinafter referred to as a "B pixel") 200B having a spectral sensitivity of B (blue) is arranged at the lower right. Each imaging pixel includes a first focus detection pixel 201 and a second focus detection pixel 202 divided in the horizontal direction (hereinafter referred to as the "x direction").

[0035] In the image sensor 107 of the camera 100 according to the first embodiment, the pixel pitch P of the imaging pixels is 4 μm, and the number of imaging pixels N is approximately 20.75 million pixels (5575 columns in the x direction × 3725 rows in the vertical direction (hereinafter referred to as the "y direction"). The pixel pitch PAF of the focus detection pixels is 2 μm, and the number of focus detection pixels NAF is approximately 41.5 million pixels (11150 columns in the x direction × 3725 rows in the y direction).

[0036] In the first embodiment, a case will be described in which each imaging pixel is divided into two in the horizontal direction, but each imaging pixel may be divided in the vertical direction. Also, the image sensor 107 in the first embodiment has a plurality of imaging pixels each including a first focus detection pixel 201 and a second focus detection pixel 202, but the imaging pixel and the first and second focus detection pixels may be provided as separate pixels. For example, the first and second focus detection pixels may be discretely arranged among the plurality of imaging pixels.

[0037] Fig. 3(A) shows one imaging pixel (G pixel 200G in the figure) as viewed from the light receiving surface side (+z direction) of the image sensor 107 of the first embodiment. Fig. 3(B) shows the aa cross section of the imaging pixel in Fig. 3(A) as viewed from the -y direction.

[0038] 3B, one imaging pixel is provided with one microlens 305 for collecting incident light. Furthermore, the imaging pixel is provided with a photoelectric conversion unit 301 and a photoelectric conversion unit 302 that are divided into N parts in the x direction (divided into two parts in the first embodiment). The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively. The centers of gravity of the photoelectric conversion unit 301 and the photoelectric conversion unit 302 are decentered in the -x direction and the +x direction, respectively, with respect to the optical axis of the microlens 305.

[0039] An R, G or B color filter 306 is provided between the microlens 305 and the photoelectric conversion unit 301 and the photoelectric conversion unit 302 in each imaging pixel. Note that the spectral transmittance of the color filter may be changed for each photoelectric conversion unit, or the color filter may be omitted.

[0040] Light incident on the imaging pixel from the imaging optical system is collected by the microlens 305, dispersed by the color filter 306, received by the photoelectric conversion unit 301 and the photoelectric conversion unit 302, and then photoelectrically converted.

[0041] Next, the relationship between the structure of the imaging pixel shown in Figures 3(A) and 3(B) and pupil division will be described with reference to Figure 4. Figure 4 shows the aa cross section of the imaging pixel shown in Figure 3(A) as viewed from the +y direction, and also shows the exit pupil of the imaging optical system. In Figure 4, the x and y directions of the imaging pixel are reversed with respect to Figure 3(B) in order to correspond to the coordinate axes of the exit pupil.

[0042] As shown in Fig. 4, the first pupil region 501 of the exit pupil, whose center of gravity is decentered toward the +X direction, is an area that is substantially conjugate with the light receiving surface of the photoelectric conversion unit 301 of the imaging pixel on the -x direction side by the microlens 305. The light flux that passes through the first pupil region 501 is received by the photoelectric conversion unit 301, i.e., the first focus detection pixel 201. Moreover, the second pupil region 502 of the exit pupil, whose center of gravity is decentered toward the -X direction, is an area that is substantially conjugate with the light receiving surface of the photoelectric conversion unit 302 of the imaging pixel on the +x direction side by the microlens 305. The light flux that passes through the second pupil region 502 is received by the photoelectric conversion unit 302, i.e., the second focus detection pixel 202. A pupil region 500 indicates a pupil region that can receive light from the entire imaging pixels combining all of the photoelectric conversion units 301 and 302 (the first focus detection pixel 201 and the second focus detection pixel 202).

[0043] Next, pupil division by the image sensor will be described with reference to FIG. 5. FIG. 5 shows pupil division by the image sensor 107. As shown in FIG. 5, a pair of light beams passing through the first pupil region 501 and the second pupil region 502, respectively, are incident on each imaging pixel of the image sensor 107 at different angles, and are received by the first focus detection pixel 201 and the second focus detection pixel 202, which are divided into two. In the first embodiment, output signals from the first focus detection pixel 201 of the multiple imaging pixels of the image sensor 107 are collected to generate a first focus detection signal, and output signals from the second focus detection pixel 202 of the multiple imaging pixels of the image sensor 107 are collected to generate a second focus detection signal. In addition, the output signals from the first focus detection pixel 201 and the output signals from the second focus detection pixel 202 of the multiple imaging pixels are added to generate an imaging pixel signal. Then, the imaging pixel signals from the multiple imaging pixels are combined to generate an imaging signal for generating an image with a resolution equivalent to the number of effective pixels N (number of imaging pixels N).

[0044] Next, the relationship between the defocus amount of the imaging optical system and the phase difference between the first focus detection signal and the second focus detection signal acquired from the image sensor 107 (hereinafter referred to as "image shift amount") will be described with reference to Fig. 6. In Fig. 6, the image sensor 107 is disposed on an imaging plane 600, and the exit pupil of the imaging optical system is divided into a first pupil region 501 and a second pupil region 502 as described with reference to Figs.

[0045] As shown in FIG. 6, the defocus amount d is defined so that the distance (size) from the imaging position C of the light beam from the subject (801, 802) to the imaging plane 600 is |d|, and a front-focus state where the imaging position C is on the subject side of the imaging plane 600 is represented by a negative sign (d<0). The defocus amount d is defined so that a back-focus state where the imaging position C is on the opposite side of the subject from the imaging plane 600 is represented by a positive sign (d>0). In a focused state where the imaging position C is on the imaging plane 600, d=0. The imaging optical system is in a focused state (d=0) with respect to the subject 801, and in a front-focus state (d<0) with respect to the subject 802. The front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as a defocus state (|d|>0).

[0046] In the front focus state (d<0), the light beam from the subject 802 that passes through the first pupil region 501 (second pupil region 502) is once collected and then spreads to a width Γ1 (Γ2) centered on the center of gravity position G1 (G2) of the light beam, forming a blurred image on the imaging surface 600. This blurred image is received by each first focus detection pixel 201 (each second focus detection pixel 202) on the image sensor 107, and a first focus detection signal (second focus detection signal) is generated. In other words, the first focus detection signal (second focus detection signal) is a signal that represents an image of the subject 802 at the center of gravity position G1 (G2) of the light beam on the imaging surface 600, blurred by the blur width Γ1 (Γ2).

[0047] The blur width Γ1 (Γ2) of the subject image increases roughly in proportion to an increase in the magnitude |d| of the defocus amount d between the first focus detection signal and the second focus detection signal. Similarly, the magnitude |p| of the image shift amount p (= the difference G1-G2 in the center of gravity position of the light beam) of the subject image between the first focus detection signal and the second focus detection signal also increases roughly in proportion to an increase in the magnitude |d| of the defocus amount d. In the back focus state (d>0), the direction of the image shift of the subject image between the first focus detection signal and the second focus detection signal is opposite to that in the front focus state, but the same is true.

[0048] In this way, as the defocus amount increases, the image shift amount of the subject image between the first focus detection signal and the second focus detection signal increases. In the first embodiment, "focus detection using an image sensor phase difference detection method" is performed to calculate the defocus amount from the image shift amount of the subject image between the first focus detection signal and the second focus detection signal obtained using the image sensor 107.

[0049] Next, the focus detection area of ​​the image sensor 107 from which the first focus detection signal and the second focus detection signal are obtained will be described with reference to Fig. 7. In Fig. 7, A(n,m) indicates the n-th focus detection area in the x direction and the m-th focus detection area in the y direction among a plurality of focus detection areas (three in the x direction and three in the y direction, totaling nine) set in the effective pixel area 1000 of the image sensor 107. The first focus detection signal and the second focus detection signal are generated from output signals from a plurality of first focus detection pixels 201 and second focus detection pixels 202 included in the focus detection area A(n,m). I(n,m) indicates an index that indicates the position of the focus detection area A(n,m) on the display 131.

[0050] It should be noted that the nine focus detection areas shown in FIG. 7 are merely an example, and in the present invention, the number, positions, and sizes of the focus detection areas are not limited to the example in FIG. 7. For example, one or more areas may be set as focus detection areas in a predetermined range centered on a position specified by the user or the position of the subject detected by the subject detection unit 140 (hereinafter also referred to as the "subject position"). In the first embodiment, the focus detection areas are arranged so that a higher resolution focus detection result can be obtained when acquiring a defocus map, which will be described later. For example, a total of 9600 focus detection areas are arranged on the image sensor 107, divided into 120 horizontally and 80 vertically.

[0051] Next, the flow of live view shooting of the imaging device (camera 100) according to the first embodiment will be described. Fig. 8 is a flowchart showing the flow of live view shooting of the camera 100 according to the first embodiment. Specifically, Fig. 8 shows processing for causing the camera 100 to perform operations from before imaging to display a live view image on the display 131 to capturing a still image. The CPU 121 executes the processing of Fig. 8 according to a computer program. In the following description, S means step.

[0052] First, in S1, the CPU 121 causes the image sensor drive circuit 124 to drive the image sensor 107, and acquires image data from the image sensor 107. After that, the CPU 121 acquires first and second focus detection signals from a plurality of first and second focus detection pixels included in each of the focus detection areas shown in FIG. 7, from the acquired image data. The CPU 121 also adds the first and second focus detection signals of all effective pixels of the image sensor 107 to generate an image signal, and has the image processing circuit 125 perform image processing on the image signal (image data) to acquire image data. Note that when the image sensor and the first and second focus detection pixels are provided separately, the CPU 121 acquires image data by performing a complementation process on the focus detection pixels.

[0053] Next, in S2, the CPU 121 causes the image processing circuit 125 to generate a live view image from the image data obtained in S1, and causes the generated live view image to be displayed on the display 131. Note that the live view image is a reduced image matched to the resolution of the display 131, and the user can adjust the image capture composition, exposure conditions, and the like while viewing the live view image. For this reason, the CPU 121 performs exposure adjustment based on the photometric value obtained from the image data, and displays it on the display 131. The exposure adjustment is realized by appropriately adjusting the exposure time, opening and closing the aperture of the shooting lens, and adjusting the gain for the image sensor output.

[0054] Next, in S3, the CPU 121 determines whether or not a switch Sw1 (hereinafter simply referred to as "Sw1") that instructs the start of an image capture preparation operation has been turned on by half-pressing a release switch included in the operation switch group 132. When the CPU 121 determines that Sw1 is not turned on in S3, it repeats the determination made in S3 to monitor the timing at which Sw1 is turned on. On the other hand, when the CPU 121 determines that Sw1 is turned on in S3, it advances the process to S400 and performs subject tracking AF processing (subject tracking autofocus processing). In the subject tracking AF processing, it performs detection of a subject area from the obtained imaging signal and focus detection signal, detection of a focus area, setting of a focus detection area, predictive AF processing for suppressing the influence of a time lag from focus detection timing to image exposure timing, and the like. Details of the "subject tracking AF processing" that causes the camera 100 to perform a subject tracking AF operation will be described later.

[0055] After performing subject tracking AF processing, CPU 121 advances the process to S5 and determines whether or not switch Sw2 (hereinafter simply referred to as "Sw2"), which instructs the start of imaging operation, has been turned on by full pressing of the release switch. If CPU 121 determines in S5 that Sw2 is not turned on, CPU 121 returns the process to S3. On the other hand, if CPU 121 determines in S5 that Sw2 is turned on, CPU 121 advances the process to S300 and executes imaging processing. Details of the "imaging processing" that causes camera 100 to perform imaging operation will be described later. When imaging processing ends, CPU 121 advances the process to S7.

[0056] In S7, the CPU 121 determines whether or not a main switch included in the operation switch group 132 has been turned off. If the CPU 121 determines in S7 that the main switch has been turned off, the CPU 121 ends the live view shooting. On the other hand, if the CPU 121 determines in S7 that the main switch has not been turned on, the process returns to S3.

[0057] In the first embodiment, the subject tracking AF process is performed after it is detected that Sw1 is turned on in S3 (after it is determined that Sw1 is turned on), but the timing of the subject tracking AF process is not limited to this. By performing the subject tracking AF process in S400 before Sw1 is turned on, it is possible to eliminate the need for the photographer to perform a preparatory operation before shooting.

[0058] Next, a description will be given of the flow of the imaging process executed by the CPU 121 in S300 of Fig. 8. Fig. 9 is a flowchart showing the flow of the imaging process executed by the CPU 121 in S300 of Fig. 8.

[0059] In S301, the CPU 121 performs exposure control processing to determine imaging conditions (shutter speed, aperture value, imaging sensitivity, etc.). This exposure control processing can be performed using luminance information acquired from image data of a live view image. Then, in S301, the CPU 121 transmits the determined aperture value to the aperture drive circuit 128 to drive the aperture 102. Also, in S301, the CPU 121 transmits the determined shutter speed to the shutter 108 to perform an operation of opening the focal plane shutter. Furthermore, in S301, the CPU 121 causes the image sensor 107 to accumulate charge during the exposure period via the image sensor drive circuit 124.

[0060] After performing the exposure control process, in S302, the CPU 121 causes the image sensor drive circuit 124 to read out all pixels of the image sensor 107 image signal for capturing a still image. The CPU 121 also causes the image sensor drive circuit 124 to read out one of the first focus detection signal and the second focus detection signal from the focus detection area (focus target area) in the image sensor 107. The first focus detection signal or the second focus detection signal read out at this time is used to detect the focus state of the image during image playback, which will be described later. By subtracting one of the first focus detection signal and the second focus detection signal from the image signal, the other focus detection signal can be obtained.

[0061] Next, in S303, the CPU 121 causes the image processing circuit 125 to perform defective pixel correction processing on the imaging data that was read out and A / D converted in S302.

[0062] Furthermore, in S304, the CPU 121 causes the image processing circuit 125 to perform image processing and encoding such as demosaic processing, white balance adjustment processing, gamma correction processing (tone correction processing), color conversion processing, and edge enhancement processing on the imaging data after the defective pixel correction processing.

[0063] Then, in S305, the CPU 121 records the still image data obtained as a result of the image processing and encoding processing in S304 and one of the focus detection signals read out in S302 in the flash memory 133 as an image data file.

[0064] Next, in S306, CPU 121 records camera characteristic information (imaging device characteristic information) as characteristic information of camera 100 (imaging device) in association with the still image data recorded in S305 in flash memory 133 and memory (RAM) within CPU 121. The camera characteristic information includes, for example, the following information: Imaging conditions (aperture value, shutter speed, imaging sensitivity, etc.) Information about image processing performed by the image processing circuit 125 Information regarding the light receiving sensitivity distribution of the imaging pixels and focus detection pixels of the image sensor 107 Information regarding vignetting of the imaging light flux within the camera 100 Information on the distance from the mounting surface of the imaging optical system of the camera 100 to the imaging element 107. Information on manufacturing errors of the camera 100.

[0065] Information on the light receiving sensitivity distribution of the imaging pixels and focus detection pixels of the image sensor 107 (hereinafter simply referred to as "light receiving sensitivity distribution information") is information on the sensitivity of the image sensor 107 according to the distance (position) on the optical axis from the image sensor 107. Since this light receiving sensitivity distribution information depends on the microlens 305 and the photoelectric conversion units 301 and 302, it may be information on these. Furthermore, the light receiving sensitivity distribution information may be information on the change in sensitivity with respect to the incident angle of light.

[0066] Next, in S307, the CPU 121 records lens characteristic information (photographing lens characteristic information) as characteristic information of the imaging optical system in association with the still image data recorded in S305 in the flash memory 133 and the memory (RAM) in the CPU 121. The lens characteristic information includes, for example, the following information: Exit Pupil Information Information about the frame of the lens barrel that blocks the light beam - Information on focal length and F-number at the time of shooting -Information about aberrations in imaging optical systems -Information about manufacturing errors in imaging optical systems Information on the position of the focus lens 105 during imaging (subject distance)

[0067] Next, in S308, the CPU 121 records image-related information as information related to the still image data in the flash memory 133 and the memory (RAM) in the CPU 121. The image-related information includes, for example, information related to the focus detection operation before imaging, information related to the movement of the subject, and information related to the focus detection accuracy.

[0068] Next, in S309, the CPU 121 causes the display 131 to display a preview of the captured image. This allows the user to easily check the captured image. When the process in S309 ends, the CPU 121 ends the image capture process and proceeds to S7 in FIG. 8.

[0069] Next, a description will be given of the flow of the subject tracking AF process executed by the CPU 121 in S400 of Fig. 8. Fig. 10 is a flowchart showing the flow of the subject tracking AF process executed by the CPU 121 in S400 of Fig. 8.

[0070] In S401, the CPU 121 calculates the amount of image shift of the subject image between the first focus detection signal and the second focus detection signal obtained in each of the multiple focus detection areas obtained in S2, and calculates the defocus amount for each focus detection area from the calculated image shift amount. In this way, the CPU 121 obtains a defocus map by calculating the defocus amount for each focus detection area. As described above, in the first embodiment, a group of focus detection results obtained from focus detection areas arranged on the image sensor 107 in a total of 9600 points divided into 120 horizontally and 80 vertically is called a defocus map.

[0071] Next, in S402, the CPU 121 performs subject detection processing and tracking processing. The above-mentioned subject detection unit 140 performs subject detection processing to detect the subject area. In the subject detection processing, depending on the state of the obtained image, it may be impossible to detect the subject area. In such a case, the CPU 121 performs tracking processing using other means such as template matching to estimate the position of the subject. The details of the subject detection processing and tracking processing will be described later.

[0072] Next, in S403, CPU 121 causes focus area detection unit 142 to perform focus area detection processing to detect a focus area. Details of the focus area detection processing will be described later. In the present invention, subject detection unit 140 (first detection means) performs subject area detection processing (subject detection processing) in S402, and focus area detection unit 142 (second detection means) performs focus area detection processing (focus area detection processing) in S403.

[0073] The difference between the subject area detection process and the focus area detection process will be described below. In the subject area detection process, when the subject is a person, the face area and the eye area of ​​the person are detected as the subject area. In addition, when the subject is a vehicle such as a motorcycle, the subject area detection process detects the entire body area of ​​the motorcycle and the helmet area of ​​the driver driving the motorcycle as the subject area. In other words, in the subject area detection process, when the subject is a living thing, the entire body and organs of the living thing are detected, and when the subject is a non-living thing such as a vehicle, parts having a certain function of the non-living thing (for example, the tires of the vehicle, the handlebars of the vehicle, etc.) are detected.

[0074] On the other hand, in the focus area detection process, an area on which the photographer wants to focus (hereinafter referred to as "area to be focused") is detected as a focus area according to the shooting scene (information on the shooting scene of the subject). For example, when the subject is a person, in a shooting scene in which the face is photographed relatively large and facing diagonally forward under a shallow depth of field condition, the focus area detection process detects the eyelash area (eyelash area) of the front eye as the focus area. Also, when the subject is a person, in a shooting scene in which the face is photographed relatively large and facing diagonally forward under a shallow depth of field condition, the focus area detection process detects the open pupil area as the focus area. In either shooting scene, the subject area detection process (subject detection process) performed in S402 detects the pupil area as the subject area.

[0075] In the first embodiment, the eyelash area is detected as a focus area in an area different from the pupil area, but there are cases where the gap between the eyelashes is large and focus detection cannot be performed properly. In such a case, the display is performed in the eyelash area, but the focus adjustment may be performed by adding a previously registered offset amount to the result of the focus adjustment performed in the pupil area.

[0076] Similarly, when photographing a motorcycle road race, the subject is often the motorcycle and its driver. When the motorcycle in the race is cornering in a direction approaching the photographer, the body of the motorcycle falls toward the photographer (nearby), and when the motorcycle is cornering in a direction away from the photographer, the body of the motorcycle falls toward the photographer (farther). In such a photographing environment, the area that the photographer wants to focus on may be the area of ​​the driver's organs, which are living things, or the area of ​​the motorcycle's parts, which are non-living things, depending on the photographing scene, and is not uniquely determined. For example, in a photographing scene where the motorcycle body is leaning toward the near side, the area that the photographer wants to focus on is the area of ​​the helmet, and in a photographing scene where the motorcycle body is leaning toward the far side, the area that the photographer wants to focus on is the area of ​​the body near the engine or gasoline tank. This is because, when photographing an image with a shallow depth of field, if the area in focus of the subject is too far back, it will look unnatural.

[0077] In S402, a specific area of ​​the subject is fixedly detected, and the orientation of the subject is also detected. In S403, a region that the photographer wants to focus on is statistically detected based on the orientation of the subject (for example, the tilt direction of the motorcycle body), the shooting environment (for example, the subject size, shallow depth), and the background environment. The region detected in S402 is a first local region (a region corresponding to at least a part of the subject region), and the region detected in S403 is a second local region (a region corresponding to at least a part of the subject region). In addition, in S402, the subject detection unit 140 detects an area indicating the subject characteristics, such as the whole body or organs of the person when the subject is a person, or parts of the vehicle when the subject is a vehicle, as the subject region (first local region). In S403, the focus region detection unit 142 detects an area indicating the shooting scene characteristics, such as the pattern of the subject, the subject size, the depth, and the tilt direction of the subject, as the focus region (second local region). The area indicating the shooting scene characteristics is also an area according to the characteristics of the focus target.

[0078] Next, in S404, CPU 121 performs a focus detection area setting process to set a focus detection area (area to be focused) using the information on the subject detection area obtained in S402 and the information on the focus area as the area to be focused obtained in S403. In S404, CPU 121 functions as a local area selection unit. The focus detection area setting process will be described in detail later.

[0079] Next, in S405, the CPU 121 acquires the focus detection result (defocus amount) of the focus detection area set in the focus detection area setting process in S404. The focus detection result acquired in S405 may be a focus detection result close to the desired area selected from the focus detection results calculated in S401 (the defocus map acquired in S401). In addition, the focus detection result acquired in S405 may be used to newly calculate the defocus amount using a focus detection signal corresponding to the set focus detection area. In addition, the focus detection area for calculating the defocus amount is not limited to one, and multiple focus detection areas may be arranged around the focus detection area and used for calculation.

[0080] Next, in S406, the CPU 121 performs predictive AF processing using the defocus amount obtained in S405 and the defocus amount obtained in the past. The predictive AF processing is a process required when there is a time lag between the timing of focus detection and the timing of image exposure, and is a process of predicting the position of the subject a predetermined time after the timing of focus detection and performing AF control. The details of the predictive AF processing will be described later.

[0081] When the predictive AF process performed in S406 ends, the CPU 121 ends the subject tracking AF process and advances the process to S5 in FIG.

[0082] Next, a description will be given of the subject detection process and tracking process executed by the CPU 121 in S402 of Fig. 10. Fig. 11 is a flowchart showing the flow of the subject detection process and tracking process executed by the CPU 121 in S402 of Fig. 10.

[0083] In S2000, the CPU 121 sets dictionary data according to the type of the object to be detected from the data detected from the image data acquired in S2. Specifically, in S2000, based on the priority of the object set in advance and the setting of the camera 100 (imaging device), the dictionary data to be used in the object detection process and the tracking process is selected (set) from a plurality of dictionary data stored in the dictionary data storage unit 141. For example, the objects are classified and stored as a plurality of dictionary data, such as "people", "vehicles", and "animals". In the first embodiment, the number of dictionary data to be selected may be one or more. When one dictionary data is selected, it is possible to repeatedly detect the object that can be detected by one dictionary data at a high frequency. On the other hand, when a plurality of dictionary data is selected, the dictionary data is set sequentially according to the priority as the detected object, so that the object can be detected sequentially.

[0084] Next, in S2001, subject detection unit 140 performs subject detection using the image data read in S2 as an input image and the dictionary data set in S2000. At this time, subject detection unit 140 outputs information such as the position, size, and reliability of the detected subject as subject detection area information. At this time, CPU 121 may cause display device 131 to display the subject detection area information output by subject detection unit 140. Also, in S2001, subject detection unit 140 hierarchically detects multiple areas of the subject as subject detection areas from the image data. For example, when "person" or "animal" is set as dictionary data in S2000, subject detection unit 140 hierarchically detects multiple areas such as a "whole body" area, a "face" area, and an "eye" area as subject detection areas. The detected "whole body" area is an overall area showing the whole body of the subject, and the detected "face" area and "eye" area are local areas showing the organs of the subject. Although local regions such as the "face" region or the "eye" region of a person or animal are regions to be focused on as a subject, they may not be detectable due to surrounding obstacles or the direction of the face. In the present invention, even in such a case, the subject detection unit 140 is configured to detect the subject hierarchically in order to continue to robustly detect the subject by detecting the entire body. Similarly, when "vehicle" is set as dictionary data in S2000, the subject detection unit 140 hierarchically detects the entire region including the driver of the vehicle and the vehicle body and the region of the driver's helmet (driver's head) as a local region as a subject detection region. In the present invention, when "vehicle" is set as dictionary data, the subject detection unit 140 is configured to detect the subject hierarchically by detecting the entire vehicle including the driver of the vehicle and the vehicle body.

[0085] Next, in S2002, the CPU 121 performs a known template matching process using the subject detection area obtained in S2001 as a template. Using the multiple images obtained in S2, a similar area is searched for in the image obtained immediately before, using the subject detection area obtained in the past image as a template. As is well known, any information may be used for template matching, such as luminance information, color histogram information, feature point information such as corners and edges, etc. There are various possible matching methods and template update methods, and any method may be used. The tracking process performed in S2002 is performed to realize stable subject detection processing and tracking processing by detecting an area similar to the past subject detection data from the image data obtained immediately before when a subject is not detected in S2001.

[0086] When the tracking process performed in S2002 ends, the CPU 121 ends the subject detection process and the tracking process, and advances the process to S403 in FIG.

[0087] Next, a description will be given of the focus area detection process executed by the CPU 121 in S403 of Fig. 10. Fig. 12 is a flowchart showing the flow of the focus area detection process executed by the CPU 121 in S403 of Fig. 10.

[0088] In S3000, the CPU 121 determines whether or not to perform focus area detection processing. As described above, the focus area is an area in the subject that should be focused on, and the focus area detection processing is processing for detecting an area (focus area) different from the subject detection area detected by the subject detection processing described in FIG. 11. Therefore, when it is inappropriate or impossible to detect the area to be focused on in the subject, the focus area detection processing is skipped. The focus area detection processing is skipped when the size of the subject area detected in S402 is smaller than a predetermined size, or when the depth difference in the subject in the shooting settings or live view settings is smaller than a predetermined value. In these cases, the difference in focus state (the difference between the area in focus and the area out of focus) in the subject area is difficult to visually recognize, so the focus area detection processing is skipped.

[0089] When the size of the subject region is smaller than a predetermined size, it becomes difficult to visually recognize the difference in focus state within the subject region. Therefore, in the first embodiment, in S3000, the CPU 121 determines to skip the focus region detection process when the size of the subject region is smaller than a predetermined size.

[0090] As is well known, the depth difference within a subject is determined by the distance to the subject and the aperture diameter of the photographing optical system. The farther the subject distance is and the smaller the aperture diameter is, the deeper the depth becomes, and the wider the region in the subject region that is in an allowable blur (in focus). That is, the wider the region in the subject region that is in depth becomes. This makes it difficult to visually recognize the difference in focus state within the subject region. Therefore, in the first embodiment, in S3000, CPU 121 determines to skip the focus region detection process when the depth difference in the subject region is smaller than a predetermined value.

[0091] As described above, in S3000, if the CPU 121 determines not to perform the focus area detection process (i.e., determines to skip the focus area detection process), the CPU 121 ends the focus area detection process and proceeds to S404 in FIG.

[0092] On the other hand, in S3000, when CPU 121 determines to perform focus area detection processing, CPU 121 advances processing to S3001 and performs signal acquisition of the subject area. That is, in S3001, CPU 121 acquires image data of all subject detection areas including the whole area and local area hierarchically detected by subject detection unit 140. As described above, the whole area of ​​the subject is the area of ​​the whole body of the subject if the subject is a human or animal, and is the area including the vehicle and the driver of the vehicle if the subject is a vehicle such as a motorcycle. Subject detection unit 140 as a third detection means outputs the detection result of the whole area detected as the subject detection area (signal of the whole area) as a result of performing the subject detection processing and tracking processing described in S402 based on the image data obtained in S2. If there are multiple subject areas (subject detection areas) detected by subject detection unit 140, the focus area detection processing performed in S3002 is performed multiple times.

[0093] Next, in S3002, CPU 121 causes focus area detection unit 142 to perform focus area detection processing for detecting a focus area. As described above, focus area detection unit 142 detects an area to be focused as a focus area based on the state of the subject in the subject area (subject detection area) detected by subject detection unit 140, based on an instruction from CPU 121. In the focus area detection processing, only one area may be detected as the focus area, or multiple areas may be detected. When multiple areas are detected as the focus area, the imaging device (camera 100) automatically selects the multiple detected areas, or the photographer selects the multiple detected areas, thereby appropriately setting the area to be focused on. At this time, CPU 121 may cause display 131 to display the information on the focus area output by focus area detection unit 142.

[0094] When the focus region detection process performed in S3002 (if there are multiple subject detection regions, the focus region detection process is performed multiple times), the CPU 121 ends the focus region detection process and advances the process to S404 in FIG.

[0095] Next, a description will be given of the focus detection area setting process executed by the CPU 121 in S404 of Fig. 10. Fig. 13 is a flowchart showing the flow of the focus detection area setting process executed by the CPU 121 in S404 of Fig. 10.

[0096] In S4000, CPU 121 acquires information such as the position, size, and reliability of the subject as information on the subject detection area obtained as the output of the subject detection process and tracking process performed in S402. Next, in S4001, CPU 121 acquires information such as the position, size, and reliability of the focus area as information on the focus area obtained as the output of the focus area detection process performed in S403.

[0097] Next, in S4002, the CPU 121 sets the focus detection area using the subject detection area information obtained in S4000 and the focus area information obtained in S4001. The focus detection area can be set by selecting a focus detection result that is reliable and indicates a subject that is relatively close from the results of the subject detection area and the focus area set as the focus area. Alternatively, the focus detection area can be set by relocating the focus detection area within the obtained subject detection area and the area set as the focus area, acquiring image data and a focus detection signal again, and selecting the focus detection result in the same manner.

[0098] The following method can be used to select the area to be used for setting the focus detection area from the subject detection area and the focus area. When only one of the subject detection area and the focus area is detected, the detected area is set as the focus detection area. When neither the subject detection area nor the focus area is detected, the focus detection area is set at the same position as the previous focus detection area. When both the subject detection area and the focus area are detected, the focus area is set as the focus detection area with priority over the subject detection area. When both the subject detection area and the focus area are detected, the subject detection area may be set as the focus detection area or the focus area may be set as the focus detection area according to information on the photographing scene of the subject. The camera 100 (imaging device) may be configured to display the set focus detection area on the display 131. The camera 100 (imaging device) may be configured to display the subject detection area, the focus area, and the focus detection area in a differentiated manner or selectively.

[0099] When the setting of the focus detection area performed in S4002 is completed, the CPU 121 ends the focus detection area setting process and advances the process to S405 in FIG.

[0100] Next, the focus area detected by the focus area detection process performed in S403 will be described with reference to Figures 14(A) to 14(D) and 15. Figures 14(A) to 14(D) show examples of scenes that a photographer may want to capture when the subject is a motorcycle and its driver.

[0101] FIG. 14(A) shows a motorcycle and its driver traveling with the direction of approaching the camera 100 (imaging device) as the direction of travel. FIG. 14(B) shows a motorcycle and its driver approaching the front and attempting to corner to the left as seen by the driver. FIG. 14(C) shows a side view of the motorcycle as it attempts to corner to the left as seen by the driver. FIG. 14(D) shows a side view of the motorcycle as it attempts to corner to the right as seen by the driver. In FIG. 14(A) to FIG. 14(D), a whole region 900 and a local region 901 are shown as subject regions (subject detection regions) detected in S402. Similarly, in FIG. 14(A) to FIG. 14(D), local regions 902 and 903 are shown as focus regions detected in S403.

[0102] 14(A) and 14(B) show how the driver's head is detected as a local region 901 for subject detection, and local regions 902 and 903 are detected as focus regions. A plurality of regions (local region 902 and local region 903) are detected as focus regions because an image focused on either region may be desired depending on the photographer's preference or intention. In S4002, when setting the focus detection region, the CPU 121 determines the priority of the focus region in consideration of the settings of the imaging device such as driver priority or close distance priority, the position of the detection region in the image in the shooting range, and continuity with the previous focus detection region setting. For example, when the CPU 121 determines that close distance priority is selected, it sets the focus detection region 903 as the focus detection region.

[0103] FIG. 14C shows a state in which the driver's head is detected as the local region 901 of the subject detection, and local region 902 is detected as the focus region. In the shooting scene of FIG. 14C, the body of the motorcycle is tilted away from the camera 100 (imaging device). Therefore, a depth difference occurs between the local region 902, which is the body of the motorcycle, and the subject detection region 901. In such a situation, close distances are prioritized, and an image focused on the local region 902 is often preferred. Therefore, in the case of the shooting scene of FIG. 14C, the focus region detection process of the present invention does not detect the local region 901 of the head as the focus region, but detects the local region 902 of the body.

[0104] In the shooting scene in Fig. 14(D), the body of the motorcycle is tilted toward the front, so it is close to the head region, which is an important organ of the subject, and the region to be focused on coincides with it. Therefore, the local region 901 detected by subject detection and the local region 902 detected by focus region detection overlap as shown in Fig. 14(D).

[0105] Note that the subject detection area and the focus area have been described using Figures 14(A) to 14(D) in which the subject detection area encompasses the focus area, but in the present invention, the size relationship between the areas is not limited to this.

[0106] In this way, in the present invention, not only are the head and pupils and the like detected as important organs during photography, but the focus area is also detected as an area different from the subject detection area depending on the photography scene, making it possible to perform focus adjustment that is more in line with the photographer's intentions.

[0107] There are various possible shooting scenes in which focus region detection is effective. Figure 15 shows main examples of scenes in which focus region detection is effective.

[0108] As shown in FIG. 15, when taking a portrait of a subject, the right side of the face is closer to the camera 100 (imaging device), the focus is generally set on the right eye of the subject. Therefore, the right eye is detected as an important organ in subject detection, and the right eye is also detected in focus area detection. By detecting the focus area of ​​the present invention (focus area detection process performed in S403 in FIG. 10), the eyelash area of ​​the right eye is detected as the focus area when the person's face is large and the shooting scene is shallow. This makes it possible to obtain an image with high contrast of the eyelashes, which makes it easier to see the focus state.

[0109] Also, when the subject is a "motorcycle," as explained in Fig. 14(A) to Fig. 14(D), when shooting from the front in the direction of travel, both subject detection and focus area detection detect the helmet, and focus adjustment is performed on the helmet. The same is true when the body of the vehicle is tilted forward. On the other hand, when the body of the vehicle is tilted backward, subject detection detects the helmet (head) as a vital organ, but detects the body area near the engine as the focus area.

[0110] Also, when the subject is a "car (for example, a race car such as an F1 car)" and the subject is photographed from the front in the direction of travel and slightly above, the subject detection detects the helmet (head) as a vital organ. However, in the above-mentioned photographed scene, the focus area is detected as a position in front of the driver's seat in order to include the entire car body within the depth. This is to deal with the fact that when photographing a race car such as an F1 car from above the front, the car body has depth, and if the helmet (head) is focused on, the front of the car body is out of the depth and becomes blurred. When machine learning is performed as the CNN constituting the focus area detection unit 142 (hereinafter referred to as "CNN for focus area detection"), the focus area is set for each shooting direction of the image of the race car and learned, thereby realizing the above-mentioned detection. In such a photographed scene, it is considered to output the necessary depth information as the output of the focus area detection. It is sufficient to output the aperture value information of the photographing optical system for including the entire car body within the depth. By setting the focus detection area based on the detection of the focus area and setting the aperture value as necessary, it is possible to obtain an image in which the entire car spreading in the depth direction is included within the depth.

[0111] When the subject is a car (e.g., a race car such as an F1 car), in a shooting scene shot from the side in the direction of travel, subject detection detects the helmet (head) as a vital organ, but detects the side of the car body as the focus area. This is because the helmet (head) is located further back than the car body, for the same reason as when the car body is tilted behind a motorcycle.

[0112] Next, the difference between the machine learning of the focus region detection CNN for realizing focus region detection and the machine learning of the CNN constituting the subject detection unit 140 for realizing subject detection (hereinafter referred to as "subject detection CNN") will be described.

[0113] The collection of a group of images to which training data for focus area detection is added will be described. First, for images to which training data is added during machine learning, subject detection is performed on the collected group of images, and images in which a desired subject is detected are extracted. Images having a depth difference within the detected subject area are extracted using the contrast distribution within the subject area and the corresponding defocus map information. Training data is added to the extracted images. When there are areas with high and shallow contrast within the subject area, or when there are areas with small and large defocus amounts from the defocus map information, it is determined that there is a depth difference. On the other hand, when there is no depth difference within the subject area (when there is no contrast difference or when the distribution of defocus amounts is within a predetermined value), the image data is learned as a negative sample. By carrying out machine learning in this manner, it is possible to realize a CNN for focus area detection that performs focus area detection when there is a depth difference within the subject detection area, and does not perform focus area detection when there is no depth difference.

[0114] The teacher data may be added to the extracted images while checking each image, but if the area with high contrast or small defocus amount is known within the subject area, the teacher data can be added automatically. After the teacher data is automatically added, the teacher data may be manually fine-tuned.

[0115] In order to efficiently collect learning data, data augmentation can be performed. Well-known methods include translation, enlargement / reduction, rotation, noise addition, and blurring. In the present invention, as a method of data augmentation effective for the focus detection region, a method of blurring a region other than the focus region is used, rather than a well-known method of blurring the entire image or the entire subject region. This makes it possible to obtain image data corresponding to different depth differences in the subject region from an image to which one piece of teacher data is added. In addition, by making the degree of blurring different for each image, the blur state of the region other than the focus region that becomes the teacher data can be made different, and robust learning can be performed even when photographed with a different aperture diameter of the photographing optical system. As a method of blurring, a method of increasing the blurring to be added depending on the distance from the focus detection region, a method of making the presence or absence of blurring different between the focus detection region and the other regions, and a method of appropriately processing the boundary region can be used. In addition, the degree of blurring to be added to the region other than the focus region may be set depending on the change in the aperture diameter. This makes it possible to generate learning data that is close to images that are actually captured.

[0116] In the first embodiment, image data is input to the CNN for focus region detection, but the input data of the CNN for focus region detection is not limited to image data. By inputting information from which depth can be inferred, such as a contrast map or a defocus map, in addition to image data, to the CNN for focus region detection, as input data of the CNN for focus region detection, the focus region can be detected more appropriately. In that case, when the CNN for focus region detection is made to undergo machine learning, a contrast map or a defocus map may be prepared in addition to image data, and learning may be performed.

[0117] Next, a description will be given of the predictive AF process executed by the CPU 121 in S406 of Fig. 10. Fig. 16 is a flowchart showing the flow of the predictive AF process executed by the CPU 121 in S406 of Fig. 10.

[0118] In S6000, CPU 121 determines whether the subject is a moving object moving in the optical axis direction. Specifically, CPU 121 determines whether the subject is moving in the optical axis direction by referring to time series data of past defocus detection results and determining whether adjacent differences between a plurality of time series data have the same sign. In S6000, if CPU 121 determines that the subject is a moving object moving in the optical axis direction, it advances the process to S6001. On the other hand, in S6000, if CPU 121 determines that the subject is not a moving object moving in the optical axis direction, it advances the process to S6012.

[0119] In S6001, the CPU 121 calculates the traveling direction of the subject detected in the latest image data. The subject's orientation has already been detected by the subject detection process and tracking process performed in S402. There are two methods for calculating the traveling direction of the subject: a method that uses local detection within the subject (e.g., the face or eyes) and a method that uses the posture detection result of the subject.

[0120] First, a method of using local detection within a subject (for example, a face or an eye) will be described. In local detection within a subject, when the subject is a person, the local detection area includes the eye, head, and torso of the person, and when the subject is a vehicle such as a motorcycle, the local detection area includes the head (helmet) of the driver of the vehicle. A known method will be described when the subject is only a person. When the local detection area is the eye, and both eyes are detected, the traveling direction of the subject is set to the optical axis direction, when only the right eye is detected, the traveling direction of the subject is set to the right direction, and when only the left eye is detected, the traveling direction of the subject is set to the left direction. When the subject is not only a person (for example, when a motorcycle is also included), the position of the eye may not be known, or the detection of the eye may differ from the traveling direction of the subject. In this invention, in such a case, the traveling direction of the detected subject is estimated from the size of a rectangular frame indicating the range of the entire detected subject and the positional relationship of the local detection area within the detected subject relative to the range of the entire detected subject. Here, a case where the subject is a motorcycle and a driver will be described as an example. When the aspect ratio of the detection range of the entire subject is long and short and the position of the local detection area (the driver's helmet in this example) is at the top of the entire range of the detected subject, the subject is considered to be facing forward, and the traveling direction of the detected subject is the optical axis direction. When the aspect ratio of the detection range of the entire subject is short and long and the position of the local detection area is at the top right of the entire range of the detected subject, the traveling direction of the detected subject is considered to be the rightward direction. As described above, the traveling direction of the detected subject can be calculated from the aspect ratio of the entire range of the detected subject and the positional relationship between the range of the detected subject and the local detection area.

[0121] Even with the above-mentioned method, if the detected subject's traveling direction suddenly changes (for example, if the detected subject suddenly moves upward from a state approaching the optical axis direction, such as a jumping motion), there is a possibility that the traveling direction of the detected subject cannot be calculated. In such a case, it is necessary to estimate the traveling direction of the detected subject before the detected subject changes its traveling direction.

[0122] A method of using the posture detection result of the subject as an estimation method for estimating the moving direction of the detected subject will be described. There are various methods for detecting the posture of the subject, but in the first embodiment, the joint parts of the subject are first estimated from an image using a neural network that has undergone deep learning. The posture information of the subject is detected by connecting the estimated joints. The moving direction of each subject may be learned in advance, or the moving direction may be estimated from the movement amount of each joint between frames. Also, a characteristic previous movement before the moving direction changes (for example, a movement before a jump) may be learned in advance. Furthermore, the moving direction may be estimated in combination with local detection within the subject (for example, a face, eyes, etc.). When the subject is only a person, the moving direction is estimated from posture information obtained by detecting the joint parts of the arms and legs before the jump. For example, it is estimated that the moving direction changes (for example, the posture before the jump) due to the arms being lowered or both legs being bent. Even when the subject is not only a person (for example, when a motorcycle is also included), the moving direction of the detected subject is estimated from the positional relationship of the joint parts of the person's arms and legs. In the case where a motorcycle is also included, as shown in Fig. 14(D), if the right leg is detected, the joint is bent, and the joints of the waist and spine are detected, it can be estimated that the detected subject is moving in the direction of the optical axis and to the right. In addition, in the case where a motorcycle is also included, the moving direction of the detected subject can also be estimated by detecting the inclination of the tires and handlebars.

[0123] The estimation of whether the light is traveling in the optical axis direction may be detected by estimating the subject position from the defocus amount.

[0124] In S6002, the CPU 121 predicts the future travel direction of the subject (predicts the future travel direction of the subject). Specifically, the CPU 121 predicts the travel direction of the subject from a time series change in the calculation result of the travel direction of the subject in past frames. The future travel direction of the subject may be estimated from a time series change amount between frames in the aspect ratio of the detection range of the entire subject (hereinafter also simply referred to as the "aspect ratio") and the positional relationship between the detection range of the entire subject and the local detection area (hereinafter also simply referred to as the "positional relationship of the subject area"). Here, an example will be described in which the subject is a motorcycle and a driver. When the aspect ratio changes from a state in which the length is long to a state in which the width is long, and the positional relationship of the local detection area changes to the upper right with respect to the detection range of the entire subject, the travel direction changes from approaching in the direction of the optical axis to the right, so it can be estimated that the travel direction has changed to the right. In this example, the local detection area is the driver's helmet.

[0125] In addition, when predicting the direction of travel of a subject by estimating the subject's posture, for example, the movement before a jump can be estimated from posture information based on the joints of the subject's arms and legs and time-series changes in the joints, and it can be predicted that the direction of travel of the detected subject will change upward.

[0126] As described above, in S6002, the CPU 121 predicts the future traveling direction of the detected subject from the time-series change in the calculation result of the traveling direction of the detected subject obtained in a plurality of past frames. The prediction of the future traveling direction of the detected subject will be described with reference to Figs. 18(A) to 18(F). Fig. 18(A) is a diagram showing the traveling direction of the detected subject with an arrow. The downward direction is the optical axis direction, and the right side is the rightward direction. When the traveling direction of the detected subject gradually changes as in Fig. 18(A), the traveling direction is calculated from the aspect ratio of the entire range of the detected subject and the positional relationship with the local region, and the next traveling direction of the detected subject is predicted from the time-series change in the traveling direction. When the traveling direction of the detected subject suddenly changes as in Fig. 18(C), the posture of the subject before the sudden change in the traveling direction is detected by the posture estimation of the subject described above, and the next traveling direction of the detected subject is predicted.

[0127] In S6003, CPU 121 determines whether the image plane velocity of the subject is large. The image plane velocity of the subject is calculated from a time series change in the image plane position of the subject. In S6003, if CPU 121 determines that the image plane velocity of the subject is large, it advances the process to S6004. On the other hand, in S6003, if CPU 121 determines that the image plane velocity of the subject is not large, it advances the process to S6012. In S6004, CPU 121 determines whether there is a change in the traveling direction of the subject, and if it determines that there is a change in the traveling direction, it advances the process to S6005, and if it determines that there is no change in the traveling direction, it advances the process to S6008.

[0128] In S6005, the CPU 121 changes the number of historical data used in the prediction calculation. Specifically, the CPU 121 changes the number of historical data used for the data of the subject position calculated from the defocus amount and focus position of the past frame used when predicting the subject position. FIG. 17 illustrates an example of the time series change of the image plane position of the subject. In FIG. 17, the horizontal axis indicates time, the vertical axis indicates the image plane movement amount of the subject, the black circle indicates the image plane position history data of the subject based on the result of focus detection, and the dotted line indicates the prediction curve by the prediction process. The historical data is the position of the subject on the image plane (the image plane position of the subject) and information on the time acquired in the past. The explanation will be given using the conceptual diagrams of FIG. 18(A) to FIG. 18(F). FIG. 18(A) is a diagram showing the moving direction of the subject with an arrow. FIG. 18(B) corresponds to FIG. 18(A). In FIG. 18B, the horizontal axis indicates time, the vertical axis indicates the image plane position of the subject, the solid line indicates the trajectory of the subject, the black circle indicates the image plane position of the subject at the timing of focus detection, and the dotted line indicates the focus movable range. FIG. 18A assumes that the camera is shooting from below, the up-down direction corresponds to the optical axis direction, and shows the subject's travel direction estimated by the subject travel direction calculation performed in S6001 and the subject travel direction prediction performed in S6002. FIG. 18A is a diagram showing an example in which the subject travel direction approaches the optical axis direction and changes to the right halfway. In FIG. 18B, the time range in which the travel direction is in the optical axis direction is 18-b1, and the time range in which the travel direction includes the right direction is 18-b2. In the time range of 18-b1, the travel direction does not change and the image plane speed change is not large, so the number of history data used in the prediction calculation is not changed. On the other hand, in the time range of 18-b2, the direction of travel changes and the image plane velocity also changes, so by reducing the number of pieces of history data in which the direction of travel is in the optical axis direction, the number of pieces of history data used in the predictive calculation is reduced, thereby reducing errors in the predictive calculation described below.

[0129] 18(C) and 18(D) show examples in which the subject's travel direction is different. FIG. 18(C) is a diagram showing an example in which the subject's travel direction suddenly changes from the optical axis direction to the right. FIG. 18(D) corresponds to FIG. 18(C). In FIG. 18(D), the horizontal axis indicates time, the vertical axis indicates the image plane position of the subject, the time range in which the subject's travel direction is in the optical axis direction is 18-d1, and the time range in which the subject's travel direction is in the right direction is 18-d2. In the examples of FIG. 18(C) and FIG. 18(D), a sudden change in the subject's travel direction is estimated by subject travel direction prediction, and when the subject travel direction changes only from the optical axis direction to the right direction, the number of history data used in the prediction calculation is reset immediately before the direction change, so that it is not used. This makes it possible to suppress erroneous prediction that the subject's travel direction is in the optical axis direction even if the subject travel direction suddenly changes to the right.

[0130] Furthermore, Fig. 18(E) and Fig. 18(F) show examples in which the subject's travel direction changes differently. Fig. 18(E) is a diagram showing an example in which the subject's travel direction alternates between the optical axis direction and the right direction, and between the optical axis direction and the left direction. Fig. 18(F) corresponds to Fig. 18(E). In Fig. 18(F), the horizontal axis indicates time, and the vertical axis indicates the image plane position of the subject. In the examples of Fig. 18(E) and Fig. 18(F), the subject's travel direction changes, but the image plane speed of the subject does not change, so the number of history data used in the prediction calculation is not changed. As described above, by changing the number of history data used in the prediction calculation according to the subject's travel direction, the prediction error of the image plane position of the subject can be reduced.

[0131] In S6006, the CPU 121 sets the focus movable range. The focus movable range will be described with reference to FIG. 19. In FIG. 19, the horizontal axis indicates time, the vertical axis indicates the image plane position of the subject, the solid line indicates the focus position, and the dotted line indicates the focus movable range. In the case where the subject is a moving object, if the photographer mistakenly moves the AF frame away from the subject and the subject becomes the background, it will take time to return the focus to the subject if the focus is moved. In the present invention, the range in which the subject moves is estimated from the image plane movement speed of the subject and the subject distance, and the focus movable range is set based on the estimated range in which the subject moves, so that the focus is not moved when the subject is outside the focus movable range.

[0132] This makes it possible to prevent the subject from suddenly becoming blurred even if the inside of the AF frame mistakenly becomes the background due to framing, etc. In the first embodiment, the focus movable range has been described, but the focus may not move when the subject is outside the focus movable range, that is, the focus stop time may be changed.

[0133] The setting of the subject's travel direction and the focus movable range will be described with reference to Figs. 18(A) to 18(F). The dotted lines in Figs. 18(B), 18(D) and 18(F) indicate the focus movable range. In Fig. 18(B), when the subject's travel direction changes from the optical axis direction to the right, the subject does not move in the optical axis direction, so the focus movable range is set smaller than when moving in the optical axis direction. In Fig. 18(D), when the subject's travel direction changes from the optical axis direction to the right or immediately before the change, the focus movable range is set smaller than when moving in the optical axis direction.

[0134] In S6007, CPU 121 changes the focus detection area. Specifically, CPU 121 changes the focus detection area by predicting the subject travel direction performed in S6002 described above so as to widen the focus detection area or move the center of gravity of the focus detection area in the subject travel direction other than the optical axis direction. This makes it possible to prevent the subject from going outside the focus detection area even if the subject travel direction changes.

[0135] In S6008, CPU 121 calculates the predicted image plane position of the subject. Specifically, CPU 121 performs multivariate analysis (for example, the least squares method) using history data of the past image plane positions of the subject and time, and predicts the image plane position of the subject by obtaining an equation for a prediction curve. CPU 121 also calculates the predicted image plane position of the subject by substituting the time of still image capture into the obtained equation for the prediction curve.

[0136] In S6009, the CPU 121 changes the focus movement speed (image plane movement speed of the focus). Specifically, the CPU 121 changes the image plane movement speed of the focus by estimating the image plane movement speed of the subject from the subject travel direction prediction result obtained in S6002, the image plane predicted position of the subject obtained in S6008 (predicted image plane position of the subject), and history data. In the examples of Figs. 18(A) and 18(B) described above, the CPU 121 estimates that the image plane movement speed of the subject is decreasing, so it changes the image plane movement speed of the focus by decreasing the image plane movement speed of the focus. Also, in the examples of Figs. 18(C) and 18(D) described above, the CPU 121 changes the image plane movement speed of the focus by setting the image plane movement speed of the focus to 0 because the travel direction of the subject suddenly changes and does not move in the optical axis direction. In this way, in S6009, the CPU 121 changes the image plane movement speed of the focus by setting the image plane movement speed of the focus according to the travel direction of the subject.

[0137] In S6010, the CPU 121 determines whether the subject is within the focus movable range set in S6006, and if it is determined that the subject is within the focus movable range, the process proceeds to S6011. On the other hand, in S6010, if the CPU 121 determines that the subject is not within the focus movable range (i.e., outside the focus movable range), the CPU 121 does not move the focus and ends the predictive AF process. In S6011, the CPU 121 moves the focus to an image plane position (predicted image plane position of the subject) corresponding to the predicted subject position (moves the focus lens 105 to the predicted image plane position of the subject), and ends the predictive AF process. In S6012, the CPU 121 moves the focus to an image plane position (image plane position of the subject) corresponding to the subject position calculated from the focus detection result (defocus amount) (moves the focus lens 105 to the image plane position of the subject), and ends the predictive AF process.

[0138] Fig. 20 shows items that are changed during predictive calculation and focus control according to the traveling direction of the subject described in the first embodiment. During predictive calculation and focus control, CPU 121 changes items such as "number of history data items used in predictive calculation", "focus movable range", "focus movement speed", and "focus detection area" shown in Fig. 20 according to the traveling direction of the subject. Values ​​of these items shown in Fig. 20 are stored in the ROM of CPU 121 in the imaging device (camera 100) as a table according to changes in the traveling direction of the detected subject. CPU 121 changes the values ​​of these items by referring to the table stored in the ROM according to changes in the traveling direction of the detected subject.

[0139] In the present invention, the moving direction of the detected subject is estimated using the size, positional relationship, and aspect ratio of the detected subject area. In addition, in the present invention, future changes in the moving direction of the detected subject are predicted using information on the size, positional relationship, and aspect ratio of the subject area that change over time. These technical configurations of the present invention make it possible to obtain more accurate shooting results with less time lag from the focus detection timing to the image exposure timing, compared to when the same thing is done using the focus detection result.

[0140] For example, when trying to estimate that a subject approaches and then moves away using only the focus detection result (defocus amount), once the subject stops, it is difficult to estimate whether it will stop there or reverse and move away. However, in the present invention, the subject's pattern as a shooting scene characteristic is used to distinguish between multiple patterns, estimate the moving direction of the detected subject, and also predict the moving direction in the future, so that a shooting result with less time lag and higher accuracy can be obtained. For example, the multiple patterns are a pattern of "approaching motorcycle", a pattern of "sideways motorcycle", and a pattern of "moving away motorcycle".

[0141] As described above, in the first embodiment, when a focus area is detected, the focus area is selected as the focus detection area in which focus detection is performed preferentially over the subject detection area, but in the present invention, the method of setting the focus detection area is not limited to this.

[0142] For example, as a method of setting the focus detection area, a mode (second mode) in which the subject detection area is prioritized and a mode (first mode) in which the focus area is prioritized can be provided in CPU 121 as modes that can be set by the photographer. Specifically, in the first mode, both the subject detection area (first local area) and the focus area (second local area) are made selectable as the focus detection area, and the focus area is preferentially set as the focus detection area. In the second mode, the subject detection area (first local area) is set as the focus detection area. In this way, by providing CPU 121 as a local area selection means with the second mode in which the subject detection area is prioritized and the first mode in which the focus area is prioritized, it is possible to easily reflect the photographer's intention regarding the area to be focused on in the subject.

[0143] As described above, in the first embodiment, a configuration for detecting a focus area by area detection based on machine learning (focus area detection unit 142 configured by machine learning CNN and detecting a focus area) has been described. However, in the present invention, a configuration for detecting a focus area is not limited to the configuration described in the first embodiment.

[0144] For example, in the present invention, the focus area can be set using information such as the aspect ratio of the subject detection area, the size of the subject detection area, and the depth information of the subject using a defocus map (hereinafter referred to as "subject detection information"). When the subject is a person, if the size of the subject detection area is equal to or larger than a predetermined size, the position of the eyelashes can be estimated with respect to the pupil area detected as the subject detection area, and the estimated eyelash area can be set as the focus area. When the subject is a motorcycle, the inclination direction of the motorcycle body can be detected using a defocus map, and it is possible to switch between setting the focus area to the head, or to setting the focus area to an area estimated to be the body position of the subject detection area corresponding to the entire motorcycle. Similarly, the distance of the subject can be determined using the defocus information of the motorcycle body and the defocus information of the helmet (head) as the subject detection area. Furthermore, the aspect ratio of the subject detection area can be used to determine whether the body of a vehicle such as a motorcycle or a car is detected from the front or the side, and the focus area can be set.

[0145] As described above, by using a configuration that sets the focus area using subject detection information, there is no need to prepare a circuit that performs focus area detection using CNN (CNN for focus area detection) within the imaging device, and focus area detection can be achieved at low cost.

[0146] <Second embodiment> Next, a second embodiment of the present invention will be described with reference to Figures 21, 22(A), 22(B) and 23(A) to 23(D). The configuration of the second embodiment is similar to that of the first embodiment, but only the focus detection area setting process performed in the second embodiment is different from the focus detection area setting process performed in S4000 to S4002 of the first embodiment. In the focus detection area setting process performed in the second embodiment, a focus detection area is selected from candidate focus detection areas within a detected subject.

[0147] Below, a description of the configuration of the second embodiment that is similar to the configuration of the first embodiment will be omitted. Fig. 21 is a flowchart showing the flow of focus detection area setting processing in the second embodiment of the present invention.

[0148] In S4100, the CPU 121 displays a local region (including a focus region) from the entire subject (entire detected subject) detected in the acquired image. Figs. 22(A) and 22(B) show display examples of a local region (including a focus region). Figs. 22(A) and 22(B) show an example in which a motorcycle is detected as a detected subject. In Figs. 22(A) and 22(B), the region surrounded by dotted lines indicates the region that is a candidate for focus detection of the detected subject, and the region surrounded by solid lines indicates the candidate focus detection region. In Figs. 22(A) and 22(B), the helmet region, headlight region, body logo region, and muffler region surrounded by dotted lines are the regions that are candidates for focus detection of the detected subject.

[0149] In S4101, the CPU 121 as a designation means selects (designates) a focus detection candidate area from a local area (including a focus area) of a detected subject. The user designates a focus detection candidate area from the local area (including a focus area) shown in FIG. 22(A) and FIG. 22(B). In FIG. 22(A), the focus detection candidate area is the helmet area, and in FIG. 22(B), the focus detection candidate area is the body logo area. The focus detection candidate area is designated by changing and selecting the focus detection candidate area by touch operation with the user's finger or button operation. The focus detection candidate area may also be selected by the user's line of sight. The focus detection candidate area may be changed not only when Sw1 is turned on, but also before Sw1 is turned on or during continuous shooting when Sw2 is turned on. Alternatively, instead of displaying the local area in advance as performed in S4100, after the designated position is determined by a touch operation with the user's finger or a button operation, a local area close to the designated position may be selected on the camera 100 side, and the selected local area may be set as a focus detection candidate area.

[0150] As another method, a method of selecting (designating) a focus detection candidate area from a menu operation in the camera 100 will be described with reference to Figs. 23(A) to 23(D). Fig. 23(A) shows a setting screen for selecting a detected subject. In Fig. 23(A), a detected subject is selected from detectable subjects (for example, vehicles, animals, and people) (in the example of Fig. 23(A), a vehicle is selected as the detected subject). In Fig. 23(B), a type of vehicle is further selected from the selected detected subject, which is a vehicle (in the example of Fig. 23(B), a motorcycle is selected as the type of vehicle). In Fig. 23(C), the orientation of the motorcycle, which is the detected subject of the selected type, is specified (in the example of Fig. 23(C), the rightward orientation is selected). In Fig. 23(D), a focus detection candidate area is selected within the detected subject of the selected type (in the example of Fig. 23(D), a body logo part is selected as the focus detection candidate area). In Figs. 23(C) and 23(D), a different focus detection candidate area may be selected from a local area (including a focus area) for the orientation of the detected subject. As described above, the focus detection candidate area may be selected by operating the menu. Alternatively, an image of each detected subject may be selected, and local areas that will become the focus detection candidate area may be displayed and selected. Furthermore, a three-dimensional model of each detected subject may be displayed as an image, and the three-dimensional model may be made rotatable by a user operation, so that different focus detection candidate areas may be specified according to the orientation or posture of the detected subject.

[0151] The items that are candidates for local regions (including focus regions) may be changed according to the orientation of the detected subject. For example, if the front is selected as the orientation of the motorcycle in Fig. 23(C), only the helmet and headlight may be displayed as focus detection candidate regions, and if the rightward orientation is selected, the helmet, headlight, body logo, and muffler may be displayed as shown in Fig. 23(D).

[0152] Furthermore, the user may set items that are candidates for local regions (including focus regions) depending on the orientation of the detected subject.

[0153] In S4102, the CPU 121 records the focus detection candidate area designated by the designation means in S4101. The recording may be performed by storing in a memory in the CPU 121 the positional relationship on the screen of the focus detection candidate area designated from the local area (including the focus area) and the positional relationship in the optical axis direction from the defocus amount of each local area (including the focus area).

[0154] In S4103, the CPU 121 determines whether the focus detection candidate area designated by the designation means in S4101 is selectable, and if it is determined that the focus detection candidate area is selectable, the process proceeds to S4104. On the other hand, in S4103, if the CPU 121 determines that the focus detection candidate area is not selectable or if the focus detection candidate area is not selected, the process proceeds to S4105. Hereinafter, the determination conditions required for the CPU 121 to determine whether the focus detection candidate area is selectable will be simply referred to as "determination conditions". The determination conditions may be "the designated focus detection candidate area becomes invisible due to a change in the movement direction or posture of the detected subject, and cannot be detected" or "the designated focus detection candidate area can be detected, but the posture of the subject has changed at a timing different from that designated by the user". In the case of whether the posture of the subject has changed, the degree of agreement between the posture and movement direction of the subject at the time of recording in S4102 and the positional relationship with each local area may be compared and determined. Specifically, the XY direction on the screen is taken as the XY direction, and the optical axis direction is taken as the Z direction, and the correlation between the magnitude and direction of the XYZ direction vector at the time of recording is determined. If there is a correlation, it is determined that the attitude has not changed, and if there is no correlation, it is determined that the attitude has changed. The correlation method is to calculate the inner product of the vectors and obtain each angle of the two vectors. If each angle is less than a predetermined value, it is determined that there is a correlation, and if each angle is equal to or greater than a predetermined value, it is determined that there is no correlation. As described in FIG. 23(A) to FIG. 23(D), if a focus detection candidate area according to the orientation of the detected subject is selected, it is determined that the focus detection candidate area is selectable. On the other hand, if a focus detection candidate area according to the orientation of the detected subject is not selected, if the focus detection candidate area is not visible, or if it is not detected as a focus area according to the shooting scene, it is determined that the focus detection candidate area is not selectable.

[0155] In S4104, the CPU 121 sets the focus detection candidate area designated by the designation means as the focus detection area. When the setting of the focus detection area performed in S4104 is completed, the CPU 121 ends the focus detection area setting process and proceeds to S405 in Fig. 10. In S4105, the CPU 121 sets the local area automatically selected by the camera 100 as the focus detection area, rather than the focus detection candidate area designated by the designation means. When the setting of the focus detection area performed in S4105 is completed, the CPU 121 ends the focus detection area setting process and proceeds to S405 in Fig. 10. The automatic selection method by the camera 100 may be, for example, the closest local area to each local area, or a local area that is prioritized in advance for the shape, posture, or traveling direction of the subject.

[0156] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0157] 100 Cameras 107 Image sensor 121 CPU 126 Focus drive circuit 140 Object detection section 141 Dictionary data storage unit 142 Focus area detection unit

Claims

1. A detection means for detecting a plurality of types of vehicles; An acquisition means for acquiring information regarding at least one of a photographing direction relative to a vehicle or a tilt direction of the vehicle; A setting means for setting a region to be focused, The image processing device according to claim 1, wherein the setting means changes the area to be focused on in accordance with the type of vehicle detected by the detection means and the information acquired by the acquisition means.

2. 2. The image processing apparatus according to claim 1, wherein the types of vehicles detected by said detection means include motorcycles and cars.

3. 3. The image processing apparatus according to claim 2, wherein said detection means detects a head area of ​​a driver of a vehicle as the subject area.

4. 3. The image processing apparatus according to claim 2, wherein said detection means detects a body area of ​​a vehicle as the subject area.

5. The image processing device according to claim 3, characterized in that, when the subject is a motorcycle, the setting means sets the area of ​​the driver's head as the area to be focused in a scene photographed from the front with respect to the direction of travel or in a scene in which the body of the motorcycle is tilted forward.

6. 5. The image processing device according to claim 4, wherein when the subject is a motorcycle, in a shooting scene in which the body is tilted backward, the setting means sets the area of ​​the body as the area to be focused.

7. The image processing device according to claim 2, characterized in that, when the subject is a car, the setting means sets the area to be focused so that the entire car is included within the depth of field when the scene is shot from the front in the direction of travel, and when the scene is shot from above.

8. 3. The image processing device according to claim 2, wherein the setting means sets an area of ​​a side of a vehicle body as the area to be focused in a scene photographed from the side in a traveling direction when the subject is a vehicle.

9. The image processing device according to claim 1 , wherein the setting unit sets the area to be focused on the basis of items that are candidates for the area to be focused on according to a direction of the detected subject that is preset by a user.

10. The image processing device according to claim 1 , wherein the setting unit outputs a region to be focused on for a captured image by using dictionary data generated based on machine learning.

11. 2. The image processing apparatus according to claim 1, further comprising a display control means for controlling so that an indicator indicating the region to be focused is displayed on a display unit.

12. a detection step for detecting a plurality of types of vehicles; acquiring information regarding at least one of a photographing direction relative to the vehicle or a tilt direction of the vehicle; A setting step of setting a region to be focused, The image processing method according to claim 1, wherein the setting step changes the area to be focused on in accordance with the type of the vehicle and the information.

13. 12. A program for causing a computer to execute each unit of the image processing apparatus according to claim 1.

Images