Image processing device, image processing method, and imaging apparatus

Abstract

To provide an image processing device capable of suppressing the reduction of moving image quality due to switching of a main subject.SOLUTION: The image processing device obtains a confidence level of being the main subject for each of a first type of subject detected from an image on the basis of its posture. The image processing device acquires the degree of focusing for each of the first type of subject and a second type of subject detected from the image. The image processing device then determines the main subject from the first type of subject and the second type of subject detected on the basis of the confidence level and the degree of focusing.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an imaging device, and more particularly to a technique for determining a main subject from an image. [Background technology]

[0002] Patent Document 1 describes a technique for determining, as the main subject, the subject that is closest to the subject to be tracked among target subjects such as a person's face. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-66889 Summary of the Invention [Problem to be solved by the invention]

[0004] When a main subject is determined as described in Patent Document 1, the distance from the imaging device to the main subject can change significantly as the main subject changes. If the main subject changes frequently, the period when no subject is in focus becomes long, which causes a decrease in video quality.

[0005] In one aspect, the present invention provides an image processing device, an image processing method, and an imaging device that can suppress degradation of moving image quality due to switching of main subjects. [Means for solving the problem]

[0006] The above-mentioned object can be achieved by an image processing device comprising: a first detection means for detecting a first type of subject and a second type of subject different from the first type of subject from an image; a second detection means for detecting the posture of each of the first type of subject detected by the first detection means; a first acquisition means for acquiring a reliability that each of the first type of subject detected by the first detection means is a main subject based on the posture; a second acquisition means for acquiring a focus degree for each of the first type of subject and the second type of subject detected by the first detection means; and a determination means for determining the main subject from the first type of subject and the second type of subject detected by the first detection means based on the reliability and the focus degree. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device, an image processing method, and an imaging device that are capable of suppressing degradation of moving image quality due to switching of the main subject. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to an embodiment; [Figure 2] FIG. 1 is a diagram schematically illustrating a pixel array according to an embodiment. [Figure 3] 1A and 1B are a schematic plan view and a schematic cross-sectional view of a pixel according to an embodiment; [Figure 4] FIG. 1 is a diagram for explaining the correspondence between pixels of an image sensor and pupil intensity distribution according to an embodiment; [Figure 5] FIG. 10 is a diagram showing the relationship between the defocus amount and the image shift amount in the embodiment. [Figure 6] FIG. 1 is a diagram showing a focus detection area in an embodiment. [Figure 7] Flowchart for shooting mode operation in an embodiment [Figure 8] Flowchart for imaging and recording processing in an embodiment [Figure 9]Flowchart for subject tracking AF processing in an embodiment [Figure 10] Flowchart for subject detection and tracking processing in an embodiment [Figure 11] FIG. 10 is an explanatory diagram of posture information according to an embodiment; [Figure 12] 10 is a flowchart illustrating a main subject determination process according to an embodiment of the present invention. [Figure 13] FIG. 1 is a diagram illustrating an example of a neural network structure according to an embodiment. [Figure 14] FIG. 10 is a diagram illustrating a specific example of main subject determination processing in the embodiment. [Figure 15] FIG. 10 is a diagram illustrating a specific example of main subject determination processing in the embodiment. [Figure 16] FIG. 10 is a diagram illustrating a specific example of main subject determination processing in the embodiment. [Figure 17] 1 is a flowchart illustrating a focus adjustment process according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following embodiments, the present invention will be described with reference to an imaging device such as a digital camera. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] ●(Overall composition) 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing device according to this embodiment. The digital camera 100 (hereinafter simply referred to as camera 100) has an imaging optical system that includes a first lens group 101, an aperture 102, a second lens group 103, and a third lens group 104, and forms an optical image of a subject on the imaging surface of an image sensor 107.

[0012] The first lens group 101 is arranged at the front (closest to the subject) of the multiple lens groups included in the imaging optical system and is movable along the optical axis OA. The position of the first lens group 101 is controlled by a zoom actuator 111. The zoom actuator 111 moves the first lens group 101 and the second lens group 103 in conjunction with each other along the optical axis by, for example, driving a cam barrel (not shown).

[0013] The aperture 102 has an opening that can be adjusted by an aperture actuator 112, and functions as a mechanical shutter when capturing a still image. Note that a mechanical shutter separate from the aperture 102 and a shutter actuator that drives the mechanical shutter may also be provided.

[0014] The second lens group 103 moves along the optical axis OA integrally with the aperture 102 and in conjunction with the first lens group 101. The positions of the first lens group 101 and the second lens group 103 determine the angle of view (focal length) of the imaging optical system. The third lens group 105 is movable along the optical axis OA. The position of the third lens group 105 is controlled by a focus actuator 114. The position of the third lens group determines the focal length of the imaging optical system. The third lens group is called a focus lens.

[0015] The optical low-pass filter 106 is provided to reduce false colors and moire that occur in captured images. The image sensor 107 is a CMOS image sensor or a CCD image sensor having a rectangular pixel array (also called a pixel area) in which m pixels are arranged horizontally and n pixels are arranged vertically in a two-dimensional manner. Each pixel is provided with a color filter in a primary color Bayer array, for example, and an on-chip microlens. The image sensor 107 may also be a three-chip color image sensor.

[0016] In this embodiment, the imaging optical system, zoom actuator 111, aperture actuator 112, focus actuator 114, focus drive circuit 126, aperture drive circuit 128, and zoom drive circuit 129 are provided in the camera 100. However, these components may also be provided in a lens unit that is detachable from the camera 100.

[0017] The flash 115 is a light source that illuminates the subject. The flash 115 is equipped with a flashlight device using a xenon tube or a continuous-light-emitting LED (light-emitting diode). The AF (autofocus) auxiliary light source 116 projects a predetermined pattern image through a projection lens. This improves focus detection capability for low-brightness or low-contrast subjects.

[0018] The CPU 121 controls the overall operation of the imaging device 100. The CPU 121 includes an A / D converter, a D / A converter, a communication interface circuit, and the like. The CPU 121 loads a program stored in a ROM 135 into a RAM 136 and executes it to control each unit of the imaging device 100 and realize the functions of the imaging device 100, such as autofocus detection (AF), imaging, image processing, and recording. Some of the functions realized by the CPU 121 executing the program may be implemented by a hardware circuit separate from the CPU 121. Some of the circuits may also be reconfigurable circuits such as FPGAs. For example, to reduce the time required for focus detection calculations, which will be described later, some of the calculations may be executed by a dedicated hardware circuit.

[0019] The communication interface circuit included in the CPU 121 supports one or more standards for wired and wireless communication, and the imaging device 100 can communicate with external devices directly or via other devices through the communication interface circuit.

[0020] A flash control circuit 122 controls the lighting of the flash 115 in synchronization with the imaging operation. An assist light source drive circuit 123 controls the lighting of the AF assist light source 116 in synchronization with the focus detection process. An image sensor drive circuit 124 controls the imaging operation of the image sensor 107, and also A / D converts the signal obtained by the imaging operation and sends it to the CPU 121. An image processing circuit 125 can apply various image processing to image data, such as gamma conversion, color interpolation, encoding, decoding, evaluation value generation, and feature region detection.

[0021] The focus driving circuit 126 drives the focus actuator 114 based on a command including the driving amount and driving direction of the focus lens given from the CPU 121. This causes the third lens group 105 to move along the optical axis OA, changing the focal length of the imaging optical system.

[0022] The aperture drive circuit 128 drives the aperture actuator 112 to control the aperture diameter and opening / closing of the aperture 102. The zoom drive circuit 129 drives the zoom actuator 111 in response to, for example, a user instruction, and changes the focal length (angle of view) of the imaging optical system by moving the first lens group 101 and the second lens group 103 along the optical axis OA.

[0023] The display unit 131 has, for example, an LCD (liquid crystal display device). The display unit 131 displays information about the imaging mode of the imaging device 100, a preview image before imaging, a confirmation image after imaging, an in-focus state display image during focus detection, and the like. The operation unit 132 is configured with a power switch, a release switch, a zoom operation switch, an imaging mode selection switch, and the like. The release switch has two switches: SW1 that is turned on when pressed halfway, and SW2 that is turned on when pressed fully. The recording medium 133 is, for example, a semiconductor memory card that is detachable from the imaging device 100, and still image data and video data obtained by imaging are recorded on the recording medium 133.

[0024] Note that if the display unit 131 is a touch display, a touch panel or a combination of a touch panel and a GUI displayed on the display unit 131 may be used as the operation unit 132. For example, when a tap operation on the touch panel is detected during live view display, the image area corresponding to the tap position can be configured to perform focus detection as a focus detection area.

[0025] It is also possible to calculate contrast information of captured image data using the image processing circuit 125, and have the CPU 121 perform contrast AF. In contrast AF, contrast information is calculated sequentially while moving the focus lens group 105 to change the focal length of the imaging optical system, and the focus lens position at which the contrast information reaches its peak is set as the focal position.

[0026] In this way, the image capturing apparatus 100 can perform both image plane phase difference AF and contrast AF, and can selectively use one or a combination of both depending on the situation.

[0027] The subject detection unit 140 can be configured using, for example, a convolutional neural network (CNN). By configuring the CNN using parameters (dictionary data) generated by machine learning for each type of subject, the area of ​​a specific subject present in the image represented by the image data is detected. The subject detection unit 140 may be implemented using dedicated hardware configured to enable high-speed execution of CNN-based processing operations, such as a graphics processing unit (GPU) or a neural processing unit (NPU).

[0028] Machine learning for generating dictionary data can be performed by any known method, such as supervised learning. Specifically, a CNN can be trained using a data set in which, for each type of subject, input images are associated with whether or not a target subject is captured. The trained CNN or its parameters can be stored as dictionary data in the dictionary data storage unit 141. Note that CNN training may be performed on a device different from the camera 100. When a trained CNN is used for subject detection processing of a captured image, an image of the same size as the input image used for CNN training is cut out from the captured image and input to the CNN. By inputting the image to the CNN while sequentially changing the cutout position, the area in which the target subject is captured can be estimated.

[0029] Note that the object region may be detected using other methods, such as detecting an object region in an image and then determining the type of object in the object region using feature values ​​for each type of object. The configuration and learning method of the neural network can be changed depending on the detection method used.

[0030] The subject detection unit 140 can be implemented using any known method as long as it can output the number, position, size, and reliability of areas in the input image that are estimated to contain a predetermined type of subject.

[0031] The subject detection unit 140 can apply subject detection processing for multiple types of subjects to one frame of image data by repeatedly switching between dictionary data. The CPU 121 can determine which dictionary data to use for the subject detection processing from among the multiple dictionary data stored in the dictionary data storage unit 141 based on the priority of the subject types set in advance, the setting values ​​of the camera 100, etc.

[0032] The types of subjects may be, for example, but are not limited to, human bodies, human organs (faces, eyes, torsos, etc.), non-human subjects (animals, non-living objects (tools, vehicles, buildings, etc.)), etc. Separate dictionary data is prepared for subjects with different characteristics. In this case, as non-human subjects, objects that move between players in sports (between first type subjects), such as balls, pucks, and shuttlecocks, are detected.

[0033] The dictionary data for detecting a human body may be prepared separately as dictionary data for detecting a human body (contour) and dictionary data for detecting organs of the human body. The dictionary data for detecting organs of the human body may be prepared separately for each type of organ.

[0034] The posture acquisition unit 142 estimates the posture of the subject for each of the subject regions detected by the subject detection unit 140. It is assumed that the type of information regarding the posture of the subject that the posture acquisition unit 142 should acquire is predetermined depending on the type of subject. For example, if the subject is a human, the posture acquisition unit 142 acquires the positions of multiple joints as information regarding the posture of the subject.

[0035] The method for estimating the pose of a subject from an image of the subject region may be any known method, such as the method described in Cao, Zhe, et al., "Realtime multi-person 2D pose estimation using part affinity fields," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

[0036] ●(image sensor) The pixel array and pixel structure of the image sensor 107 will be described with reference to Figures 2 and 3. The left-right direction in Figure 2 is the x-direction (horizontal direction), the up-down direction is the y-direction (vertical direction), and the direction perpendicular to the x-direction and y-direction (direction perpendicular to the paper surface) is the z-direction (optical axis direction). In the example shown in Figure 2, the pixel (unit pixel) array of the image sensor 107 is shown as an area of ​​4 columns x 4 rows, and the sub-pixel array is shown as an area of ​​8 columns x 4 rows.

[0037] In the pixel group 200 of 2 columns x 2 rows, for example, a pixel 200R having a spectral sensitivity of a first color R (red) is arranged at the upper left position, a pixel 200G having a spectral sensitivity of a second color G (green) is arranged at the upper right and lower left position, and a pixel 200B having a spectral sensitivity of a third color B (blue) is arranged at the lower right position. Furthermore, each pixel (unit pixel) is divided into two in the x direction (Nx divisions) and one in the y direction (Ny divisions), resulting in a division number of 2 (division number N LF =Nx×Ny) first subpixel 201 and second subpixel 202 (the first to Nth subpixels LF It is composed of multiple sub-pixels (sub-pixels).

[0038] In the example shown in FIG. 2, each pixel of the image sensor 107 is divided into two sub-pixels arranged in the horizontal direction, and the number of divisions N is calculated from the image signal obtained by one image capture. LFIt is possible to generate a number of viewpoint images equal to the number of sub-pixels in the image sensor 107, and a captured image obtained by combining all the viewpoint images. Note that the pixels may be divided in two directions, and there is no limit to the number of divisions in each direction. Therefore, it can be said that the viewpoint images are images generated from signals of some of the sub-pixels among the plurality of sub-pixels, and the captured image is an image generated from signals of all the sub-pixels. In this embodiment, as an example, the pixel period P in the horizontal and vertical directions of the image sensor 107 is set to 6 μm, and the number of horizontal pixels N H =6000, number of vertical pixels N V = 4000. Therefore, the total number of pixels N = N H ×N V = 24 million. In addition, the horizontal period of the sub-pixels P S If the total number of sub-pixels is N S =N H ×(P / P S )×N V =48 million.

[0039] FIG. 3(a) shows a plan view of one pixel 200G of the image sensor 107 shown in FIG. 2, as viewed from the light receiving surface side (+z side) of the image sensor 107. The z-axis is set perpendicular to the plane of FIG. 3(a), with the positive direction of the z-axis defined as the front side. The y-axis is set perpendicular to the z-axis, with the positive direction of the y-axis being the up-down direction, and the x-axis is set perpendicular to the z-axis and y-axis, with the positive direction of the x-axis being the right side. FIG. 3(b) shows a cross-sectional view taken along the aa section line in FIG. 3(a) as viewed from the -y side.

[0040] As shown in Figures 3(a) and 3(b), in pixel 200G, a microlens 305 is formed on the light receiving surface side (+z direction) of each pixel, and incident light is condensed by this microlens 305. Furthermore, a plurality of photoelectric conversion units, namely, first photoelectric conversion unit 301 and second photoelectric conversion unit 302, are formed, which are divided into two in the x (horizontal) direction and one in the y (vertical) direction, with a division number of 2. The first photoelectric conversion unit 301 and second photoelectric conversion unit 302 correspond to the first subpixel 201 and second subpixel 202 in Figure 2, respectively. More generally, the photoelectric conversion unit of each pixel is divided into Nx divisions in the x direction and Ny divisions in the y direction, and the division number of the photoelectric conversion unit is N LF = Nx × Ny, the first to NthLF The photoelectric conversion units are 1st to Nth LF It corresponds to a sub-pixel.

[0041] The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn junction photodiodes, each consisting of a p-type well layer 300 and two divided n-type layers 301 and 302. If necessary, an intrinsic layer may be sandwiched between them to form a pin structure photodiode. In each pixel, a color filter 306 is formed between the microlens 305 and the first photoelectric conversion unit 301 or second photoelectric conversion unit 302. If necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit, or the color filter may be omitted.

[0042] Light incident on pixel 200G is collected by microlens 305 and further dispersed by color filter 306, after which it is received by first photoelectric conversion unit 301 and second photoelectric conversion unit 302, respectively. In first photoelectric conversion unit 301 and second photoelectric conversion unit 302, pairs of electrons and holes (positive holes) are generated according to the amount of received light, and after they are separated by a depletion layer, the electrons are accumulated. Meanwhile, the holes are discharged to the outside of image sensor 107 through a p-type well layer connected to a constant voltage source (not shown). The electrons accumulated in first photoelectric conversion unit 301 and second photoelectric conversion unit 302 are transferred to a capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0043] In this embodiment, the microlenses 305 correspond to the optical system in the image sensor 107. The optical system in the image sensor 107 may be configured to use microlenses as in this embodiment, or may be configured to use materials with different refractive indices, such as waveguides. The image sensor 107 may be a back-illuminated image sensor having circuits and the like on the surface opposite to the surface having the microlenses 305, or may be a stacked image sensor having some circuits, such as the image sensor drive circuit 124 and the image processing circuit 125. A material other than silicon may be used for the semiconductor substrate, and for example, an organic material may be used as the photoelectric conversion material.

[0044] ●(pupil division) Next, the pupil division function of the image sensor 107 of this embodiment will be described with reference to FIGS. 4 and 5(a). 4 shows a cross-sectional view of the aa cross section of pixel 200G arranged in the image sensor 107 of this embodiment shown in FIG. 3(a) as viewed from the +y side, and also shows the pupil plane at a position a distance Z in the z-axis direction (direction of the optical axis OA) from the image sensor 107's image sensor surface 600. Note that in FIG. 4, the x-axis and y-axis of the cross-sectional view are reversed compared to FIG. 3 in order to correspond to the coordinate axes of the exit pupil plane. The image sensor 107's image sensor surface 600 is disposed at the image plane of the image sensor optical system.

[0045] The pupil intensity distribution (first pupil intensity distribution) 501 has a substantially conjugate relationship, via the microlens 305, with the light receiving surface of the first photoelectric conversion unit 301, whose center of gravity is decentered in the -x direction. Therefore, the first pupil intensity distribution 501 corresponds to the pupil region that can receive light by the first subpixel 201. The center of gravity of the first pupil intensity distribution 501 is decentered to the +xp side on the pupil plane. Similarly, the pupil intensity distribution (second pupil intensity distribution) 502 has a substantially conjugate relationship, via the microlens 305, with the light receiving surface of the second photoelectric conversion unit 302, whose center of gravity is decentered in the +x direction.

[0046] Therefore, the second pupil intensity distribution 502 corresponds to the pupil region that can receive light by the second subpixel 202. The center of gravity of the second pupil intensity distribution 502 is decentered toward the -xp side on the pupil plane. Furthermore, the pupil intensity distribution 500 is the pupil region that can receive light by the entire pixel 200G when the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (the first subpixel 201 and the second subpixel 202) are all combined. In other words, the first pupil intensity distribution 501 is decentered toward the +xp side on the pupil plane with respect to the center of the pupil intensity distribution 500, and the second pupil intensity distribution 502 is decentered toward the -xp side on the pupil plane with respect to the center of the pupil intensity distribution 500.

[0047] In addition, in FIG. 4, pupil region 500 is a pupil region that can receive light in the entire pixel 200G, which is a combination of photoelectric conversion unit 301 and photoelectric conversion unit 302 (first subpixel 201 and second subpixel 202).

[0048] 5(a) is a schematic diagram showing the correspondence between the image sensor and pupil division of this embodiment. Light beams that have passed through different pupil partial regions, first pupil partial region 501 and second pupil partial region 502, are incident on each (imaging) pixel of the image sensor at different angles from the imaging surface 800, and are received by photoelectric conversion units 301 and 302, which are divided into 2 x 1 regions. Note that in this embodiment, the pupil region is divided into two in the horizontal direction, but the pupil may also be divided vertically as necessary.

[0049] The image sensor 107 has an array of imaging pixels each having a first sub-pixel 201 that receives a light beam that passes through a first pupil partial region 501 of the imaging optical system, and a second sub-pixel 202 that receives a light beam that passes through a second pupil partial region 502 of the imaging optical system that is different from the first pupil partial region. Therefore, the imaging pixel receives a light beam that passes through a pupil region 500 that is the combination of the first pupil partial region 501 and the second pupil partial region 502 of the imaging optical system.

[0050] Note that not all pixels in the image sensor 107 have multiple photoelectric conversion units (sub-pixels), but an image sensor pixel, a pixel having only a first sub-pixel, and a pixel having only a second sub-pixel may be individually arranged. Alternatively, a pixel having two sub-pixels and a pixel having one photoelectric conversion unit equivalent to two sub-pixels may be arranged.

[0051] In this embodiment, the image processing circuit 125 generates one piece of image data for phase-difference AF (first focus detection signal (image A)) by concatenating multiple signals obtained by adding together the outputs of the four first sub-pixels 201 for each repeating unit of the color filter shown in Fig. 2. The image processing circuit 125 also generates a second focus detection signal (image B) by concatenating multiple signals obtained by adding together the outputs of the four second sub-pixels 202 for each repeating unit of the color filter.

[0052] As described above, the first and second sub-pixels 201 and 202 are also called focus detection pixels because they are used to generate focus detection signals. By adding the outputs of the same type of sub-pixels present within a repeating unit of a color filter, a signal reflecting each of the R, G, and B color components can be obtained, and the signal can be treated as a luminance signal with little bias in spectral sensitivity. When capturing an image, a signal is obtained by adding the output of the first sub-pixel 201 and the output of the second sub-pixel 202 on a pixel-by-pixel basis.

[0053] ●(Relationship between defocus amount and image shift amount) The relationship between the amount of image shift between a pair of focus detection signals (image A and image B) generated by the image processing circuit 125 of this embodiment and the amount of defocus (degree of focus) of the optical system relative to the subject will be described. Fig. 5(b) shows a schematic relationship diagram between the amount of defocus and the amount of image shift between the first focus detection signal (image A) and the second focus detection signal (image B). The image sensor 107 is disposed on the imaging plane 800, and as described with reference to Figs. 4 and 5(a), the exit pupil of the imaging optical system is divided into a first pupil partial region 501 and a second pupil partial region 502.

[0054] The magnitude |d| of the defocus amount d is the distance from the imaging position of the subject to the imaging plane 800. When the defocus amount d is negative (d<0), it indicates a front-focus state where the imaging position of the subject is closer to the subject than the imaging plane 800, and when it is positive (d>0), it indicates a back-focus state where the imaging position of the subject is on the opposite side of the subject than the imaging plane 800. In the in-focus state where the imaging position of the subject is on the imaging plane 800, the magnitude of the defocus amount d is 0. Figure 5(a) shows an example where subject 801 is in an in-focus state (d=0) and subject 802 is in a front-focus state (d<0). The front-focus state (d<0) and the back-focus state (d>0) are collectively referred to as the defocus state (|d|>0).

[0055] In a front-focus state (d<0), a light beam from the subject 802 that passes through the first pupil partial region 501 (second pupil partial region 502) is collected once at a position closer to the subject than the imaging surface 800. The light beam then spreads to a width Γ1 (Γ2) around the center of gravity G1 (G2) of the light beam, forming a blurred image on the imaging surface 800. The blurred image is converted into an electrical signal by the first sub-pixel 201 (second sub-pixel 202) of each of the multiple pixels that receive the image. As described above, the phase-difference AF section generates a first focus detection signal (image A) from the signal of the first sub-pixel 201 and a first focus detection signal (image B) from the signal of the second sub-pixel 202. Therefore, image A (image B) is recorded at the center of gravity G1 (G2) on the imaging surface 800 as a subject image in which the subject 802 is blurred to a width Γ1 (Γ2).

[0056] The blur width Γ1 (Γ2) of the subject image increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. Similarly, the magnitude |p| of the image shift amount p of the subject image between the first focus detection signal and the second focus detection signal (= the difference G1-G2 in the center of gravity positions of the light beams) also increases roughly in proportion to the increase in the magnitude |d| of the defocus amount d. In the back-focus state (d>0), the relationship between the magnitude |d| of the defocus amount and the blur width of the subject image and the image shift amount p is the same, except that the direction of the image shift between image A and image B is opposite to that in the front-focus state. Therefore, as the defocus amount increases, the image shift amount between the image A and the image B increases.

[0057] ●(Focus detection area) The following describes the pixel areas of the image sensor 107 that are used to generate the first and second focus detection signals (focus detection areas). FIG. 6 shows an example of focus detection areas set in the effective pixel area 1000 of the image sensor 107, superimposed with focus detection area indicators displayed on the display unit 131 during focus detection. In this embodiment, a total of nine focus detection areas are set, three in the row direction and three in the column direction, but this is merely an example, and a greater or lesser number of focus detection areas may be set. The size, position, and spacing of the focus detection areas may also vary.

[0058] Furthermore, when every pixel in the effective pixel area 1000 has the first sub-pixel 201 and the second sub-pixel 22, as in the image sensor 107, the position and size of the focus detection area may be set dynamically. For example, a predetermined range may be set as the focus detection area, centered on a position specified by the user. In this embodiment, the focus detection area is set so that focus detection results can be obtained with higher resolution when acquiring a defocus map, which will be described later. For example, the effective pixel area 1000 is divided into 120 areas horizontally and 80 areas vertically, for a total of 9,600 areas, each of which is set as a focus detection area.

[0059] 6, the nth focus detection area in the row direction and the mth focus detection area in the column direction is represented as A(n,m), and the rectangular frame-shaped index representing the focus detection area of ​​A(n,m) is represented as I(n,m). Images A and B used to detect the defocus amount in that focus detection area are generated from signals obtained from the first sub-pixel 201 and the second sub-pixel 202 in that focus detection area. Furthermore, the index I(n,m) is usually displayed superimposed on a live view image.

[0060] ●(Shooting mode operation) The operation of the camera 100 in the imaging mode will be described with reference to the flowchart shown in Fig. 7. When the power is turned on and the startup process is completed, the camera 100 operates in the imaging mode.

[0061] In the imaging mode, the camera 100 continuously captures moving images and displays the captured moving images on the display unit 131, causing the display unit 131 to function as an EVF, while waiting for a user instruction to be input.

[0062] In S1, the CPU 121 starts driving the image sensor 107 via the image sensor drive circuit 124 in order to capture a moving image to be displayed on the display unit 131. Thereafter, the image sensor 107 outputs an analog image signal at a predetermined frame rate.

[0063] When the CPU 121 acquires one frame's worth of analog image signals from the image sensor 107, it applies correlated double sampling, A / D conversion, and the like to generate a digital image signal. The CPU 121 outputs the digital image signal to the signal processing circuit 125. The signal processing circuit 125 applies demosaic processing and the like to the digital image signal to generate image data for display. The signal processing circuit 125 writes the image data for display to, for example, a video memory area of ​​the RAM 136. The signal processing circuit 125 also generates an evaluation value to be used in AE processing from the digital image signal and outputs it to the CPU 121. Furthermore, the signal processing circuit 125 generates first and second focus detection signals for each of a plurality of focus detection areas based on signals read from pixels included in the focus detection area, and outputs the evaluation values ​​to the CPU 121.

[0064] When the first and second subpixels 201 and 202 are configured as separate pixels (do not share the same microlens), the pixel coordinates at which a signal is obtained from the first subpixel 201 are different from the pixel coordinates at which a signal is obtained from the second subpixel 202. Therefore, the signal processing circuit 125 generates first and second focus detection signals by interpolating the signals so that a signal pair of the first and second subpixels 201 and 202 exists at the same pixel position.

[0065] In S2, the CPU 121 supplies the display image data stored in the video memory area of ​​the RAM 135 to the display unit 131, and displays it as one frame of a live view image. The user can adjust the imaging range, exposure conditions, etc. while viewing the live view image displayed on the display unit 131. The CPU 121 determines the exposure conditions based on the evaluation value obtained from the signal processing circuit 125, and displays an image indicating the determined exposure conditions (shutter speed, aperture value, imaging ISO sensitivity) on the display unit 131, superimposed on the live view image.

[0066] Thereafter, the CPU 121 executes the operation of S2 each time the imaging of one frame is completed, thereby causing the display unit 131 to function as an EVF.

[0067] In S3, CPU 121 determines whether or not a half-press operation (SW1 ON) of the release switch included in operation unit 132 has been detected. If CPU 121 determines that SW1 ON has not been detected, it repeatedly executes S3. On the other hand, if it determines that SW1 ON has been detected, CPU 121 executes S300.

[0068] At S300, the CPU 121 executes subject tracking autofocus (AF) processing. At S300, the CPU 121 applies subject detection processing to image data for display and determines a focus detection area. The CPU 121 also executes predictive AF processing to prevent a decrease in AF accuracy due to the time lag between when the AF processing is executed and when the release switch is fully pressed (SW2 is turned on) is detected. Details of the operation at S300 will be described later.

[0069] In S5, the CPU 121 determines whether or not ON of SW2 is detected. If the CPU 121 determines that ON of SW2 is not detected, it executes S3. On the other hand, if it determines that ON of SW2 is detected, the CPU 121 executes the image capturing and recording process of S400. Details of the operation of S400 will be described later.

[0070] In S7, the CPU 121 determines whether or not it has detected that a main switch included in the operation unit 132 has been turned off. If it has determined that it has detected that the main switch has been turned off, the CPU 121 ends the operation in the imaging mode. On the other hand, if it has determined that it has not detected that the main switch has been turned off, the CPU 121 executes S3.

[0071] Although the subject detection process and AF process have been described as being executed in response to the detection of SW1 being turned on, they may be executed at a different timing. If the subject tracking AF process of the S300 is executed before the detection of SW1 being turned on, it becomes possible to start capturing images immediately by pressing the shutter button all the way down, omitting the halfway press operation.

[0072] ●(Image capture and recording processing) Next, the imaging and recording process executed by the CPU 121 in S400 of FIG. 7 will be described with reference to the flowchart shown in FIG. In S401, the CPU 121 determines exposure conditions (shutter speed, aperture value, imaging ISO sensitivity, etc.) through AE processing based on the evaluation value generated by the signal processing circuit 125. Then, the CPU 121 controls the operation of each unit so that a still image is captured according to the determined exposure conditions.

[0073] That is, the CPU 121 transmits the aperture value and shutter speed to the aperture drive circuit 128 to drive the aperture 102. The CPU 121 also controls the charge accumulation operation of the image sensor 107 via the image sensor drive circuit .

[0074] In S402, the CPU 121 reads out one frame's worth of analog image signals from the image sensor 107 via the image sensor drive circuit 124. Note that for at least the pixels within the focus detection area, the CPU 121 also reads out the signals of one of the first and second sub-pixels 201 and 202.

[0075] In S403, the CPU 121 A / D converts the signal read out in S402 to a digital image signal. The CPU 121 also applies defective pixel correction processing to the digital image signal using the image processing circuit 125. The defective pixel correction processing is a process that complements signals read out from pixels (defective pixels) from which normal signals cannot be read out, using signals read out from surrounding normal pixels.

[0076] In S404, the CPU 121 causes the image processing circuit 125 to generate a still image data file for recording and first and second focus detection signals. The image processing circuit 125 applies image processing, encoding processing, etc. to the digital image signal after the defective pixel correction processing to generate still image data for recording. The image processing may be, for example, demosaic (color interpolation) processing, white balance adjustment processing, gamma correction (tone correction) processing, color conversion processing, edge enhancement processing, etc. The image processing circuit 125 also applies encoding processing to the still image data in a format that corresponds to the format of the data file in which the still image data is stored.

[0077] In S405, the CPU 121 records, on the recording medium 133, an image data file that stores the still image data generated in S404 and the sub-pixel signals read from the focus detection area in S402.

[0078] In S406, the CPU 121 records device characteristic information as characteristic information of the camera 100 in the recording medium 133 in association with the image data file recorded in S405.

[0079] The device characteristic information includes, for example, the following information: Imaging conditions (aperture value, shutter speed, imaging sensitivity, etc.) Information about the image processing applied to the digital image signal by the image processing circuit 125 Information about the light-receiving sensitivity distribution of the imaging pixels and sub-pixels of the imaging element 107 Information about vignetting of the imaging light beam within the camera 100 Information on the distance from the mounting surface of the imaging optical system of the camera 100 to the imaging element 107 Information about the manufacturing tolerances of the Camera 100

[0080] Information on the light sensitivity distribution of the imaging pixels and sub-pixels (hereinafter simply referred to as light sensitivity distribution information) is information on the light sensitivity of the imaging element 107 according to the distance from the intersection of the imaging element 107 and the optical axis. Since the light sensitivity depends on the microlens 305 and photoelectric conversion units 301 and 302 of the pixel, the information may also be information on these. The light sensitivity distribution information may also be information on the change in sensitivity with respect to the angle of incidence of light.

[0081] In S407, the CPU 121 records the lens characteristic information as characteristic information of the imaging optical system in the recording medium 133 in association with the still image data file recorded in S405.

[0082] The lens characteristic information includes, for example, the following information: Exit pupil information Information about the frame of the lens barrel, etc. that blocks the light beam -Information about focal length and F-number at the time of shooting - Information about aberrations in the imaging optical system -Information about manufacturing errors in imaging optical systems Position of the focus lens 105 during imaging (subject distance)

[0083] Next, in S408, the CPU 121 associates the image-related information as information related to the still image data with the still image data file recorded in S405, and records the image-related information as information related to the still image data on the recording medium 133. The image-related information includes, for example, information related to the focus detection operation before image capture, information related to the movement of the subject, and information related to the focus detection accuracy.

[0084] In steps S406 to S408, the CPU 121 may also store the device characteristic information, lens characteristic information, and image-related information in the ROM 136 in association with the image data file recorded in step S405.

[0085] In S409, the CPU 121 causes the image processing circuit 125 to scale the still image data to generate image data for display, and causes the image data to be displayed on the display unit 131. This allows the user to check the captured image. When a predetermined display time has elapsed, the CPU 121 ends the image capturing and recording process.

[0086] ●(Subject tracking AF processing) Next, the subject tracking AF process in S300 of FIG. 7 will be described with reference to the flowchart shown in FIG.

[0087] In S301, the CPU 121 calculates the amount of image shift (phase difference) between the first and second focus detection signals generated in S2 for each of the multiple focus detection areas. The amount of image shift between the signals can be determined as the relative position at which the correlation between the signals is maximized. The CPU 121 calculates the amount of defocus as the degree of focus for each focus detection area from the calculated amount of image shift.

[0088] As described above, in this embodiment, the effective pixel area 1000 is divided into 120 areas horizontally and 80 areas vertically, resulting in a total of 9,600 areas, each of which is set as a focus detection area. The CPU 121 generates data (a defocus map) that associates the defocus amount calculated for each area with the position of the area. The CPU 121 stores the generated defocus map in, for example, the RAM 136.

[0089] In S302, CPU 121 executes subject detection processing using subject detection unit 140. Subject detection unit 140 detects the areas of one or more types of subjects, and outputs to CPU 121 the detection results including the type of subject, the position and size of the area, the reliability of detection, etc. for each detected area.

[0090] Furthermore, CPU 121 performs a process (tracking process) to detect the position of the subject in the current frame based on the result of the subject detection process in the current frame and the result of the subject detection process in a past frame. If the subject cannot be detected by the subject detection process using the trained CNN of subject detection unit 140, CPU 121 can estimate the position of the subject in the current frame by a tracking process using another method such as template matching. Details will be described later.

[0091] In S303, when the subject detection unit 140 detects a region of a subject (for example, a human) having joints as the subject region, the CPU 121 uses the posture acquisition unit 142 to acquire posture information of each subject region.

[0092] An example of posture information acquired by posture acquisition unit 142 will be described with reference to Fig. 11. Here, it is assumed that in the image of the current frame shown in Fig. 11(a), the regions of human subjects 901 and 902 and the region of ball 903 have been detected as subject regions.

[0093] Subject 901 is holding a ball 903 in both hands and is an important subject (main subject) in the captured scene. In this embodiment, the subject (main subject) that the photographer is likely to be focusing on is determined based on posture information acquired from the subject area. On the other hand, subject 902 is a non-main subject. A non-main subject is a subject that is not the main subject among the detected subjects.

[0094] 11(b) is a diagram showing an example of posture information of subjects 901 and 902, and the position and size of ball 903. Posture acquisition unit 142 detects the position of joint 911 of subject 901 as posture information of subject 901. Similarly, posture acquisition unit 142 detects the position of joint 912 of subject 902 as posture information.

[0095] FIG. 11(b) shows an example in which the positions of the top of the head, neck, shoulders, elbows, wrists, waist, knees, and ankles are detected as joint positions. However, the joint positions detected by the posture acquisition unit 142 may be some of these, or other joint positions may be detected. Furthermore, in addition to joint positions, information such as axes connecting joints may be detected as posture information. Any information related to the posture of the subject can be detected as posture information. Here, it is assumed that the posture acquisition unit 142 detects joint positions as posture information of a human subject.

[0096] The posture acquisition unit 142 detects the image coordinates (x, y) of the joints 911 and 912. Here, the image coordinates (x, y) are coordinates that indicate the pixel position in the image in a two-dimensional orthogonal coordinate system with the center pixel of the image or one of the four corner pixels as the origin (0, 0).

[0097] In addition, in S302, the subject detection unit 140 detects information including the image coordinates (x, y) of the center of gravity 413 and the size in the image (the number of pixels of the length indicated by the arrow 914) for the area detected as the ball subject 903.

[0098] In S304, the CPU 121 performs a main subject determination process. The CPU 121 determines a main subject from among the detected subjects using the defocus map generated in S301, the subject detection result obtained in S302, and the posture information obtained in S303. The main subject is a subject that is considered to be highly likely to be the subject that the photographer intends to focus on. Details of the main subject determination process will be described later.

[0099] In S305, CPU 121 sets the focus detection area so that it includes the area of ​​the main subject determined in S304. For example, of the 9600 focus detection areas that can be set, CPU 121 sets one or more focus detection areas that are included in the area of ​​the main subject and whose detected defocus amount satisfies a condition. The condition may be, for example, that a value indicating the reliability of the defocus amount is equal to or greater than a threshold and that a defocus amount indicating the closest subject distance has been obtained.

[0100] The setting of the focus detection area in S305 may be performed based on the defocus amount obtained in the next frame for all focus detection areas included in the area of ​​the main subject determined in S304.

[0101] In S306, the CPU 121 acquires the defocus amount for the focus detection area set in S305. The defocus amount acquired here may be the amount calculated in S301, or may be the amount newly calculated for a new frame.

[0102] In S307, CPU 121 executes predictive AF processing for each of the subject regions detected by subject detection unit 140 in S402. Predictive AF processing is processing for predicting the defocus amount of a subject region at the time of capturing the next frame. CPU 121 generates time-series data of the defocus amount for each subject region based on, for example, the defocus map generated in S401 for one or more past frames and the current frame. CPU 121 then obtains an equation for a prediction curve using multivariate analysis (for example, the least squares method) based on the time-series data of the defocus amount. CPU 121 predicts the defocus amount corresponding to the subject distance when capturing the next frame by substituting the capture time of the next frame into the obtained equation for the prediction curve. Note that time-series data of the position of the subject region may be generated to predict the three-dimensional position of the subject when capturing the next frame.

[0103] For example, the three-dimensional position (X, Y, Z) of the subject is expressed in an XYZ Cartesian coordinate system with the intersection of the imaging plane and the optical axis as the origin and the optical axis as the Z axis. The three-dimensional position of the subject when capturing the next frame can be predicted from the image coordinates (X, Y) of the subject area and the time-series data of the defocus amount Z.

[0104] For human subjects, the defocus amount corresponding to the subject distance when capturing the next frame may be predicted from the time-series data of joint positions. By using the time-series data, it is possible to estimate the position even when the joint position cannot be detected because it is hidden by another subject. Whether the subject is partially hidden or has fallen out of the frame can be determined from the number and positions of the joint positions that cannot be detected.

[0105] In S308, the CPU 121 executes focus adjustment processing for capturing the next frame based on the defocus amount predicted for the main subject in S307. The CPU 121 converts the defocus amount into a drive direction and drive amount for the focus lens, and controls the focus actuator 114 via the focus drive circuit 126 to drive the focus lens 105. Details will be described later. When the drive of the focus lens 105 is complete, the CPU 121 ends the subject tracking AF processing.

[0106] ●(Subject detection and tracking processing) Next, the subject detection and tracking process in S302 of FIG. 9 will be described in detail with reference to the flowchart shown in FIG.

[0107] In S2000, CPU 121 sets dictionary data to be used by subject detection unit 140 by determining the type of subject to be detected by subject detection unit 140. The type of subject to be detected can be determined based on predetermined priorities, settings of camera 100 (e.g., shooting mode), etc. For example, assume that dictionary data for "people," "vehicles," "animals," and "balls" is stored in dictionary data storage unit 141. Note that subject types may be categorized more finely. For example, dictionary data such as "dogs," "cats," "birds," and "cows" may be stored instead of "animals," and dictionary data such as "four-wheeled vehicles," "two-wheeled vehicles," "trains," and "airplanes" may be stored instead of "vehicles."

[0108] When a shooting mode for shooting a specific type of subject is set in camera 100, CPU 121 sets dictionary data for that type of subject. For example, when portrait mode or sports mode is set, dictionary data for "people" is set. When sports mode is set, dictionary data for "ball" is also set. When "panning mode" is set, dictionary data for "vehicle" is set.

[0109] If a shooting mode for shooting a specific type of subject is not set in camera 100, CPU 121 sets dictionary data for the subject according to a predetermined priority. For example, dictionary data for "people" and "animals" can be set.

[0110] The type of dictionary data and the method for determining the dictionary data to be set are not limited to the methods described here. One or more dictionary data may be set. When one dictionary data is set, subjects that can be detected using one dictionary data can be detected with a high frequency. When multiple dictionary data are set, multiple types of subjects can be detected by switching dictionaries for each frame. Note that multiple types of subjects may be detected for the same frame if the processing time allows. When detecting one type of subject per frame, the detection frequency of a type of subject with a first priority may be set higher than the detection frequency of a type of subject with a lower second priority. Here, it is assumed that dictionary data for "person" and "ball" are set. For convenience, the following description will be given assuming that the first and second types of subjects are detected for the same frame.

[0111] In S2001, CPU 121 uses subject detection unit 140 to apply a detection process for a first type of subject to the image of the current frame. Here, it is assumed that the first type of subject is a "person." Subject detection unit 140 applies the subject detection process to the image of the current frame using "person" dictionary data stored in dictionary data storage unit 141. Subject detection unit 140 outputs the detection result to CPU 121. At this time, CPU 121 may display the subject detection result on display unit 131. Furthermore, CPU 121 stores the detected subject area in RAM 136.

[0112] When "person" dictionary data is set, the subject detection unit 140 detects multiple types of regions related to a person with different granularities, such as a "whole body" region, a "face" region, and an "eye" region. It is desirable to detect local regions such as a person's eyes or face for use in focus detection and exposure control, but there is a possibility that they cannot be detected if the face is not facing forward or is hidden by other subjects. On the other hand, it is unlikely that the whole body will be completely undetectable. Therefore, multiple types of regions with different granularities are detected, increasing the possibility of detecting some region of a "person." Note that dictionary data can also be configured to detect multiple types of regions with different granularities for types of subjects other than people.

[0113] In S2002, CPU 121 uses object detection unit 140 to apply a detection process for a second type of object to the image of the current frame. Here, it is assumed that the second type of object is a "ball." Object detection unit 140 applies the object detection process to the image of the current frame using the "ball" dictionary data stored in dictionary data storage unit 141. Object detection unit 140 outputs the detection result to CPU 121. At this time, CPU 121 may cause display unit 131 to display the object detection result.

[0114] Instead of subject detection using the "ball" dictionary data, object detection processing may be combined with processing to recognize a ball from detected objects. Any known method, such as YOLO, may be used for object detection. YOLO is the method described in Redmon, Joseph, et al., "You only look once: Unified, real-time object detection," Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, or a successor version thereof. Note that "ball" also includes badminton shuttlecocks and ice hockey shuttlecocks. Sports equipment used by hand, such as rackets, bats, and golf clubs, may also be detected.

[0115] In S2003, the CPU 121 performs object tracking processing by applying template matching processing to the current frame using the object region most recently detected in S2001 as a template. The image of the object region itself may be used as the template, or information obtained from the object region, such as brightness information, color histogram information, or feature point information such as corners and edges, may be used as the template. Any known method may be used for the matching method and template update method. The result of the tracking processing may be the position and size of the region in the current frame that is most similar to the template.

[0116] The tracking process in S2003 may be performed only if a first type of subject is not detected in S2001. By detecting an area in the current frame that is similar to a previously detected subject area, stable subject detection and tracking process can be achieved. The CPU 121 ends the subject detection and tracking process when the tracking process ends.

[0117] ●(Main subject detection processing) Next, the main subject determination process in S304 of FIG. 9 will be described in detail with reference to the flowchart shown in FIG.

[0118] In S4000, the CPU 121 obtains the defocus amounts corresponding to the first and second types of object regions detected in the object detection and tracking process from the defocus map generated in S301. Then, the CPU 121 calculates the difference in the defocus amounts corresponding to the first and second types of object regions. If multiple first and second types of object regions are detected, the CPU 121 calculates the difference in the defocus amounts for each combination.

[0119] In S4001, CPU 121 calculates the probability or reliability that a first type of subject is the main subject. Here, a case where the probability that a first type of subject is the main subject in an image is calculated will be described, but other probabilities or reliability may also be calculated. For example, the reliability may be calculated as the reciprocal of the distance between the center of gravity of the first type of subject area and the center of gravity of the second type of subject area.

[0120] (How to calculate probability) Since the first type of subject is a person, a method for calculating the probability that the first type of subject is the main subject based on the joint positions and sizes will be described. Here, a method using a neural network will be described.

[0121] 13 is a diagram showing an example of the structure of a neural network. The neural network has an input layer 1001, a hidden layer 1002, and an output layer 1003. There may be multiple hidden layers 1002. Each layer has multiple neurons 1004, and the neurons 1004 in adjacent layers are connected to each other by synapses 1005.

[0122] The number of neurons 1004 in the input layer 1001 is equal to the dimension of the input data. The number of neurons in the output layer 1003 is equal to the number of answers. Here, the output layer has two neurons 1004 because a neural network is used to obtain two answers, namely, whether a certain type of object is the main object or not. A neural network that classifies inputs into two classes is used to determine whether a first type of object is the main object or not (whether the probability is high or low).

[0123] Let wij be the weight of the synapse 1005 connecting the i-th neuron 1004 in the input layer 1001 and the j-th neuron in the hidden layer 1002. In this case, the output zj of the j-th neuron in the hidden layer 1002 is given by the following equation:

number

[0124] In equation (1), xi represents the value input to the i-th neuron in the input layer 1001. Since all neurons in the input layer 1001 are connected to the j-th neuron in the hidden layer 1002, the input values ​​of each neuron are weighted and added together and input to the j-th neuron in the hidden layer 1002.

[0125] The jth neuron in the hidden layer 1002 outputs the value of the activation function h, which takes as its argument the value obtained by adding a bias bj to the input value. The bias bj is a parameter corresponding to the sensitivity of the neuron. The activation function h is a function that converts the input value into a value that represents the excitation state of the neuron. Here, ReLU (Rectified Linear Unit) is used, but other functions such as a sigmoid function may also be used.

[0126] The weight of the synapse 1005 connecting the j-th neuron in the hidden layer 1002 and the k-th neuron in the output layer 1003 is denoted by wkj, and the bias of the k-th neuron in the output layer 1003 is denoted by bk. In this case, the value yk output by the k-th neuron 1004 in the output layer 1003 is given by the following equation:

number

[0127] In equation (3), zj is the output value from the jth neuron in the hidden layer 1002, which is connected to the kth neuron in the output layer 1003. Furthermore, i and k are the neuron numbers in the output layer 1003, where i,k=1 or 2. The output yk of each neuron in the output layer 100 is normalized by the softmax function shown in equation (4) so ​​that the sum becomes 1. If y1 corresponds to the classification of the main subject and y2 corresponds to the classification of the non-main subject, then f(y1) represents the probability that it is the main subject, and f(y2) represents the probability that it is the non-main subject.

[0128] The input values ​​for the neural network are the coordinates of the person's joints and the coordinates and size of the ball. All weights and biases are then optimized through learning to minimize a loss function that uses the output probability and correct label. The correct label is a binary value: 1 for the main subject and 0 for non-main subject. The loss function can be any function that can measure the degree of agreement with the correct label, such as mean squared error. As an example, the binary cross-entropy shown below is used as the loss function here.

number

[0129] In equation (5), m is the index of the object to be learned. ym is equal to the probability value f(y1) output from the neuron k=1 in the output layer 1003. tm is the correct label (0 or 1).

[0130] By optimizing to reduce the value of equation (5), it is possible to learn weights and biases so that the correct label and the output probability value approach each other. The learned weights and biases are saved in the recording medium 133, and can be used by the CPU 121 by storing them in the RAM 136 as needed. Multiple types of learned weights and biases can be prepared depending on the scene, etc.

[0131] The CPU 121 can obtain the probability f(y1) that the subject area is the area of ​​the main subject by inputting the values ​​obtained from the subject area into a neural network to which learned weights and biases (results of machine learning performed in advance) have been applied.

[0132] When training a neural network, subject information (here, joint positions) immediately before an important action can be used as the state of the main subject. For example, in a ball-throwing sport, joint positions detected from an image of a person stretching their hand out to throw the ball can be used as one of the states of the main subject.

[0133] The reason for performing such learning is to enable the camera 100 to execute appropriate control for a subject that has taken an action that should be considered the main subject. For example, by automatically starting image recording when the probability value of the subject being the main subject exceeds a first predetermined value, it becomes possible to reliably record a scene that would not be captured in time even if the photographer manually gave a shooting instruction. Note that control may also be performed using information on the typical time from the state used for learning (the state of extending a hand holding a ball) to an important action (throwing the ball). This increases the likelihood of recording an image at the time of an important action, such as the moment the ball leaves the hand.

[0134] Here, an example has been described in which a determination as to whether an image is the main subject is made based on a probability obtained using a trained neural network. However, other machine learning techniques, such as support vector machines or decision trees, may be used as long as a similar determination can be made. Furthermore, a function that outputs a reliability or probability value based on a certain model may also be constructed, without being limited to machine learning.

[0135] For example, in the case of a sport using a ball, whether or not a person is the main subject may be determined based on the positional relationship between the person and the ball. For example, assuming that the closer the person is to the ball, the higher the probability that the person is the main subject, whether or not the person is the main subject may be determined based on the value of a monotonically decreasing function of the distance between the person and the ball. Note that whether or not a person is the main subject may also be determined using only the posture information of the subject, without using information about the ball.

[0136] Depending on the subject's action (e.g., passing, shooting, etc.), it may be better to use ball information in some cases, or not. For example, in the case of a shooting action, the distance between the person and the ball is large, but it may be desirable to make the shooting subject the main subject. In this case, using only the subject's posture information without using the distance between the ball and the person may result in a determination that is more in line with the photographer's intention. On the other hand, if the person is about to receive the ball, using the distance between the ball and the person in addition to the posture information may result in a determination that is more in line with the photographer's intention. In this way, it may be possible to determine whether or not to take ball information into account depending on the type of action predicted from the subject's posture information. Alternatively, data obtained by performing a predetermined transformation, such as a linear transformation, on the coordinates of each joint and the coordinates and size of the ball may be used as input data for the neural network.

[0137] Alternatively, time-series data on the posture of the person, the positions of the person and the ball, the amount of defocus of each subject, and the probability that each subject is the main subject may be used as input data to the neural network.In addition, predicted values ​​of the coordinates of the person's joints and the coordinates and size of the ball at the time of image capture may be used as input data to the neural network.

[0138] Furthermore, if the determination of the main subject frequently switches between subjects with large differences in defocus amounts, it is likely that this is not what the photographer intended. Therefore, when CPU 121 determines from the time-series data of the subject determined as the main subject that the main subject frequently switches between subjects with large defocus amounts, it can suppress the switching. CPU 121 can suppress the switching when the determination of the main subject switches (from A to B, from B to A) between two subjects A and B whose difference in defocus amount is equal to or greater than a threshold value two or more times within a period of a predetermined number of frames.

[0139] The CPU 121 can suppress switching by, for example, correcting the reliability (probability) of one of the subjects A and B (for example, the subject on the closer side) to be the main subject. Alternatively, an area including both subjects A and B may be used as the area of ​​the main subject. In this case, the CPU 121 can control the focal length and / or aperture value so that both subjects are in focus.

[0140] It is also possible to use predicted data when the amount of change per unit time in the position of the subject or joints is large (for example, above a threshold), and not use predicted data when the amount of change is small (below the same or a different threshold). By doing so, it is possible to maintain the accuracy of the reliability indicating the likelihood of being the main subject when the change in the subject's posture is small, and to detect changes in the main subject more quickly when the change in the subject's posture is large.

[0141] As described above, the CPU 121 calculates the reliability (probability) that a region of the first type of subject is the main subject. When multiple regions of the first type of subject are detected, the reliability is calculated for each region, and a representative value is set as the reliability of the first type of subject. The representative value may be, for example, the maximum value.

[0142] In S4002, the CPU 121 determines whether the reliability of the first type of subject calculated in S4001 is equal to or greater than a first predetermined value. If it is determined that it is equal to or greater than the first predetermined value, the CPU 121 executes S4005; if it is not, the CPU 121 executes S4003. As described above, the reliability (probability) takes a value of 0 to 100%, so the first predetermined value is also set within this range. Here, the first predetermined value is set to 80.

[0143] In S4003, CPU 121 determines whether the difference in defocus amount was able to be calculated in S4000, and if it is determined that it was able to be calculated, executes S4004, and if it is not determined that it was able to be calculated, executes S4007. The difference in defocus amount cannot be calculated when, for example, the first type of subject or the second type of subject is outside the imaging range or is only partially included in the imaging range due to movement of the subject or a change in the imaging range.

[0144] In S4004, the CPU 121 determines whether the difference in the defocus amount between the person, which is the first type of subject, and the ball, which is the second type of subject, is greater than a second predetermined value, and if it is determined that the difference is greater than the second predetermined value, executes S4006, and if it is not, executes S4007. The second predetermined value is in units of defocus amount, and can be 1Fδ or 1Fδ, where F is the aperture value and δ is the allowable circle of confusion diameter.

[0145] In S4005, CPU 121 determines, as the main subject, the subject having the highest reliability (probability) of being the main subject among the first type of subjects.

[0146] In S4006, CPU 121 determines that the second type of subject is the main subject. Here, it is assumed that one ball is detected, but if multiple balls are detected, CPU 121 determines that the subject with the highest reliability (probability) of being the main subject is the main subject.

[0147] S4006 is executed when the reliability (probability) that the first type of subject is the main subject is less than a first predetermined value, and the difference in defocus amount between the first type of subject and the second type of subject is greater than a second predetermined value. For example, in a sport where the ball is passed between people who are far apart in the depth direction, this condition applies. By setting the second type of subject (the ball) as the main subject, and controlling the focus to be continuously on the ball, the focus distance can be changed more smoothly than by immediately switching the main subject from the person who made the pass to the person who receives the pass.

[0148] In S4007, similarly to S4005, the CPU 121 determines, as the main subject, the subject having the highest reliability (probability) of being the main subject among the first type of subjects.

[0149] In S4008, CPU 121 determines that the first type of subject or the second type of subject is the main subject. S4008 is executed when the difference in defocus amount cannot be calculated. When only one type of subject is present, CPU 121 determines that the present type of subject is the main subject. When multiple subjects of one type are present, CPU 121 may determine that the subject with the highest reliability or a subject previously determined as the main subject is the main subject. Furthermore, when only a portion of one type of subject is included, CPU 121 may determine that the subject closest to the center of the image is the main subject, taking into account the framing of the photographer.

[0150] When the main subject is determined in any of S4005, S4006, S4007, and S4008, CPU 121 ends the main subject determination process.

[0151] A specific example of the main subject determination process will be described with reference to FIGS. 14 shows a time series of typical scenes in which a ball 903 is passed from person 902 playing basketball to person 901. It is assumed that there is a sufficient difference in the amount of defocus (distance from the camera) between people 901 and 902. Here, a sufficient difference in the amount of defocus may be such that when one person is in focus, the other is not clear.

[0152] 14(a) shows a state in which person 902 is making a preparation movement to pass ball 903. When the reliability of person 902 being the main subject is calculated using the positions and sizes of person 902's joints and the position and size of ball 903, person 902 is in a posture of about to pass the ball, and therefore the reliability obtained is higher than that of person 901 and is equal to or greater than a first predetermined value. Therefore, the flow chart shown in FIG. 12 proceeds from S4002 to S4005, and person 902 is determined to be the main subject.

[0153] 14(b) shows a state in which the ball 903 has left the person 902 and is heading towards the person 901, and the person 901 is not making any movement to receive the ball 903. Neither person 901 nor person 902 is in a position to pass or receive the ball 903. Therefore, the reliability of the people 901 and 902 as the main subject calculated for each person is lower than the first predetermined value.

[0154] Furthermore, as the ball 903 moves from the person 902 to the person 901, the difference between the defocus amount of the person 902, who was determined to be the main subject in the state of Fig. 14(a), and the defocus amount of the ball 903 becomes larger than the second predetermined value. Therefore, the flow chart shown in Fig. 12 proceeds to S4002, S4003, S4004, and S4006, and the ball 903, which is a second type of subject, is determined to be the main subject.

[0155] 14(c) shows a state in which the ball 903 has further moved and the person 901 is in the action of catching the ball 903. When the reliability of the person 901 being the main subject is calculated using the positions and sizes of the person 901's joints and the position and size of the ball 903, the reliability obtained is higher than that of the person 902 and is equal to or greater than a first predetermined value, because the person 901 is in a posture to receive the ball 903. Therefore, the flow proceeds from S4002 to S4005 in the flowchart shown in FIG. 12, and the person 901 is determined to be the main subject.

[0156] 15 shows a typical scene in chronological order when a ball 923 is passed from a person 922 playing basketball to a person 921. Unlike the case of FIG. 14, it is assumed that there is not a sufficient difference in the amount of defocus (distance from the camera) between the people 921 and 922.

[0157] 15(a) shows a state in which person 922 is making a preparation movement to pass ball 923. When the reliability of person 922 being the main subject is calculated using the positions and sizes of person 922's joints and the position and size of ball 923, person 922 is in a posture of about to pass the ball, and therefore the reliability obtained is higher than that of person 921 and is equal to or greater than a first predetermined value. Therefore, the flow chart shown in FIG. 12 proceeds from S4002 to S4005, and person 922 is determined to be the main subject.

[0158] 15(b) shows a state in which the ball 923 has left the person 922 and is heading towards the person 921, and the person 921 is not making any movement to receive the ball 923. Neither person 921 nor 922 is in a position to pass or receive the ball 923. Therefore, the reliability of the people 921 and 922 as the main subject calculated for each person is lower than the first predetermined value.

[0159] Furthermore, because there is not a sufficient difference in the defocus amounts (distances from the camera) between the people 921 and 922, the difference between the defocus amount of the person 922 determined to be the main subject in the state of Fig. 15(a) and the defocus amount of the ball 923 is equal to or less than the second predetermined value. Therefore, the flow chart shown in Fig. 12 proceeds to S4002, S4003, S4004, and S4007, and of the people 921 and 922, which are first type subjects, one that has a higher reliability as the main subject is determined to be the main subject.

[0160] 15(c) shows a state in which ball 923 has moved further and person 921 is making the motion of catching ball 923. When the reliability that person 921 is the main subject is calculated using the positions and sizes of person 921's joints and the position and size of ball 923, since person 921 is in a posture to receive ball 923, a reliability higher than that of person 922 and equal to or greater than a first predetermined value is obtained. Therefore, the flow proceeds from S4002 to S4005 in the flowchart shown in FIG. 12, and person 921 is determined to be the main subject.

[0161] 16 shows a time series of typical scenes in which a person 923 playing basketball shoots a ball 903 into a goal 925 located beyond a person 924. It is assumed that there is a sufficient difference in the amount of defocus (distance from the camera) between the people 924 and 923.

[0162] 16(a) shows a state in which person 923 is making a preparation to shoot ball 903. When the reliability of person 923 being the main subject is calculated using the positions and sizes of person 923's joints and the position and size of ball 903, person 923 is in a posture of about to shoot, and therefore the reliability obtained is higher than that of person 924 and is equal to or greater than the first predetermined value. Therefore, the flow chart shown in FIG. 12 proceeds from S4002 to S4005, and person 923 is determined to be the main subject.

[0163] FIG. 16(b) shows a state in which the ball 903 has left the person 923 and is heading toward the goal 925, with the person 923 maintaining the shooting posture. In this state, because the ball has left the person 923, the reliability calculated for the person 923 falls below the first predetermined value. However, because the person 923 is maintaining the shooting posture, the reliability calculated for the person 923 becomes higher than the reliability calculated for the person 924. Therefore, in S4007, the person 923 is determined to be the main subject until the difference in the defocus amount between the person 923 and the ball 903 becomes larger than the second predetermined value. Thereafter, when the difference in the defocus amount between the person 923 and the ball 903 becomes larger than the second predetermined value, the ball 903 is determined to be the main subject.

[0164] When person 923 releases his shooting stance and person 924 takes an action such as reaching out to grab ball 903, the reliability that person 924 is the main subject becomes greater than that of person 923. Therefore, when the reliability that person 924 is the main subject exceeds a first predetermined value, or even if it does not exceed the first predetermined value, if the difference in the amount of defocus between person 924 and ball 903 becomes equal to or less than a second predetermined value, person 924 will be determined to be the main subject.

[0165] Consider a case where person 923 has released his shooting stance and ball 903 has been determined as the main subject, and person 924 does not make any movement such as reaching out to grab ball 903. In this case, when the difference in the amount of defocus between person 924 and ball 903 becomes equal to or less than a second predetermined value, the main subject may change to person 923 or 924, and then the main subject may change to ball 903. Therefore, even if the difference in the amount of defocus between person 924 and ball 903 becomes equal to or less than the second predetermined value, ball 903 may continue to be determined to be the main subject, thereby suppressing the change of the main subject.

[0166] Finally, the focus adjustment process in S308 of FIG. 9 will be described in detail with reference to the flowchart shown in FIG. In S5000, the CPU 121 determines whether or not the main subject will change based on the result of the main subject determination process in S404, and if it is determined that the main subject will change, it executes S5001, and if not, it executes S5003.

[0167] In S5001, the CPU 121 determines whether the operating mode of the camera 100 is video mode. Video mode is an operating mode in which video for recording is captured and recorded, and is different from the state in which video for live view display is captured in still image mode. If the CPU 121 determines that the operating mode is video mode, it executes S5002; if not, it executes S5003.

[0168] In S5002, CPU 121 controls the driving of focus lens 105 so as to focus on the new main subject. When operating in video mode, video is recorded even while the focus lens is being driven, so CPU 121 adjusts the driving speed of focus lens 105 so that the in-focus distance does not pass over the new main subject.

[0169] For example, when the main subject switches from a person to a ball, when the focus lens 105 is driven to focus on the ball, the focus lens 105 is driven at a speed that takes into account the deceleration capability so that the focus can be on the ball without passing the ball. In this way, a sudden change in the focus distance can be suppressed, and the focus distance can be switched with quality.

[0170] Furthermore, the focus adjustment speed may be changed depending on whether or not a distance range includes a subject of the type to be detected. For example, consider a case where the main subject switches from a first type of subject to a second type of subject. In this case, the drive speed of focus lens 105 in the distance range where the first type of subject is present is set slower than the drive speed of focus lens 105 in the distance range where neither the first type of subject nor the second type of subject is present. Alternatively, the drive speed of focus lens 105 in the distance range where neither the first type of subject nor the second type of subject is present is set faster than the drive speed of focus lens 105 in the distance range where the first type of subject is present.

[0171] In the distance range where the first and second types of subjects do not exist, driving the focus lens 105 quickly has little effect on the quality of the video. Therefore, priority is given to shortening the time required to switch the focused subject. In this case, too, the focus lens is driven so that the focused distance does not pass the distance of the second type of subject.

[0172] Such adjustment of the drive speed of the focus lens 105 may also be performed in an operation mode other than the video mode.

[0173] In S5003, CPU 121 drives focus lens 105 to focus on the current main subject or a new main subject. Because the camera is not operating in video mode, no video is recorded while focus lens 105 is being driven. Therefore, CPU 121 drives focus lens 105 as fast as possible within the range that allows the camera to focus on the new main subject without passing the distance of the new main subject.

[0174] When the focus control for the new main subject is completed in S5002 or S5003, the CPU 121 ends the focus adjustment process.

[0175] The image processing device according to this embodiment acquires a reliability of each first-type subject detected from an image as a main subject based on the posture. The image processing device then determines the main subject from the first-type subject and the second-type subject based on the degree of focus (defocus amount) of each of the first-type subject and the second-type subject detected from the image and the reliability of the first-type subject. This makes it possible to switch the main subject to the second-type subject before the main subject is switched between first-type subjects with different defocus amounts. As a result, it is possible to prevent the quality of video from deteriorating due to the switching of the main subject.

[0176] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0177] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0178] 100... digital camera, 107... image sensor, 121... CPU, 126... focus drive circuit, 140... subject detection unit, 142... attitude acquisition unit

Claims

1. a first detection means for detecting a first type of subject and a second type of subject different from the first type of subject from an image; a second detection means for detecting a posture of each of the first type of subjects detected by the first detection means; a focus degree acquisition means for acquiring a focus degree for each of the first type of subject and the second type of subject detected by the first detection means; a determination means for determining a main subject from the first type of subject and the second type of subject detected by the first detection means, based on the posture and the degree of focus.

2. The method further comprises a reliability obtaining means for obtaining a reliability of each of the first type of subjects detected by the first detection means as a main subject based on the posture, The image processing apparatus according to claim 1 , wherein the determining means determines the main subject further based on the reliability.

3. 3. The image processing apparatus according to claim 2, wherein the determining means determines, from among the detected subjects of the first type, a subject whose reliability is equal to or greater than a first predetermined value, to be a main subject.

4. 4. The image processing device according to claim 2 or 3, characterized in that, when there is no subject among the detected first type of subjects whose reliability is equal to or higher than a first predetermined value, the determination means determines a main subject from the first type of subjects and the second type of subjects detected by the first detection means based on the difference between the focus degree of the first type of subject and the focus degree of the second type of subject.

5. 5. The image processing device according to claim 4, wherein the determination means determines a main subject from the second type of subject when a difference between a focus degree of the first type of subject and a focus degree of the second type of subject is greater than a second predetermined value.

6. 5. The image processing device according to claim 4, wherein the determination means determines a main subject from the first type of subject based on the reliability when a difference between a focus degree of the first type of subject and a focus degree of the second type of subject is equal to or less than a second predetermined value.

7. 7. The image processing device according to claim 4, wherein, when the difference in the degree of focus cannot be obtained, the determination means determines a main subject from among the first type of subject and the second type of subject detected from the image.

8. 7. The image processing device according to claim 4, wherein, when the difference in the degree of focus cannot be obtained, the determination means determines that, of the first type of subject and the second type of subject, the subject closest to the center of the image is the main subject.

9. 9. The image processing apparatus according to claim 2, wherein the reliability obtaining means obtains the reliability by using information on the second type of subject in addition to the posture.

10. 9. The image processing device according to claim 2, wherein the reliability obtaining means obtains the reliability using a positional relationship between the first type of subject and the second type of subject in addition to the orientation.

11. 11. The image processing apparatus according to claim 1, wherein the first type of subject is a person, and the second detection means detects the posture based on information about joints.

12. 12. The image processing device according to claim 11, wherein the posture is a posture for performing an action related to the second type of subject.

13. 13. The image processing apparatus according to claim 1, wherein the second type of subject is an object that moves between the first type of subjects.

14. A first detection means for detecting a first type of subject and a second type of subject different from the first type of subject from an image; a focus degree acquisition means for acquiring a focus degree for each of the first type of subject and the second type of subject detected by the first detection means; and a control means for switching the subject to be focused on from the first type of subject to the second type of subject based on a difference in distance in a depth direction between the first type of subject and the second type of subject, or a difference in a degree of focus between the first type of subject and the second type of subject, after focusing on the first type of subject.

15. A second detection means for detecting a posture of each of the first type of subjects detected by the first detection means; a reliability acquisition unit that acquires a reliability of each of the first type of subjects detected by the first detection unit as a main subject based on the posture, 15. The image processing apparatus according to claim 14, wherein the control means focuses on a subject of the first type detected, the subject having the reliability equal to or greater than a first predetermined value.

16. The image processing device described in Claim 15, characterized in that the control means determines a subject to be focused from the first type of subject based on the reliability when the difference in distance in the depth direction between the first type of subject and the second type of subject, or the difference in the degree of focus between the first type of subject and the second type of subject, is equal to or less than a second predetermined value.

17. The image processing device described in Claim 14, characterized in that the control means switches the subject to be focused on from the first type of subject to the second type of subject when, after focusing on the first type of subject, the difference in distance in the depth direction between the first type of subject and the second type of subject, or the difference in the degree of focus between the first type of subject and the second type of subject, is greater than a second predetermined value.

18. A second detection means for detecting a posture of each of the first type of subjects detected by the first detection means; a reliability acquisition unit that acquires a reliability of each of the first type of subjects detected by the first detection unit as a main subject based on the posture, 15. The image processing apparatus according to claim 14, wherein the control means further switches the subject to be focused from the first type of subject to the second type of subject based on the reliability.

19. The image processing device described in Claim 18, characterized in that when there is no subject among the detected first type of subject whose reliability is equal to or greater than a predetermined first specified value, the control means determines a subject to be focused from the first type of subject and the second type of subject detected by the first detection means based on the difference between the degree of focus of the first type of subject and the degree of focus of the second type of subject.

20. The image processing device described in Claim 19, characterized in that when the difference in the degree of focus cannot be obtained, the control means focuses on the subject closest to the center of the image among the first type of subject and the second type of subject.

21. The image processing device according to claim 18, wherein the reliability acquisition means acquires the reliability using information on the second type of subject in addition to the posture.

22. The image processing device described in Claim 18, characterized in that the reliability acquisition means acquires the reliability using the positional relationship between the first type of subject and the second type of subject in addition to the posture.

23. The first type of subject is a person, 15. The image processing apparatus according to claim 14, further comprising: second detection means for detecting a posture of each of the first type of subjects detected by the first detection means based on information about joints.

24. The image processing device according to claim 23, wherein the posture is a posture for performing an action related to the second type of subject.

25. The image processing device of claim 14, wherein the second type of subject is an object that moves between the first type of subjects.

26. A first detection means for detecting a person and a ball from an image; a focus degree acquisition means for acquiring a focus degree for each of the person and the ball detected by the first detection means; and a control means for switching the subject to be focused on from the person to the ball based on a difference in distance in the depth direction between the person and the ball, or a difference in the degree of focus between the person and the ball, after focusing on the person.

27. ​​A second detection means for detecting a posture of each of the plurality of persons detected by the first detection means; a reliability acquisition unit that acquires a reliability of each of the plurality of persons detected by the first detection unit as a main subject based on the posture of the person, 27. The image processing apparatus according to claim 26, wherein the control means focuses on a person, of the plurality of persons, whose reliability is equal to or greater than a first predetermined value.

28. The image processing device described in Claim 27, characterized in that the control means determines a person to be focused on from among the multiple people based on the reliability when the difference in distance in the depth direction between the person and the ball, or the difference in the degree of focus between the person and the ball, is equal to or less than a predetermined second specified value.

29. The image processing device described in Claim 26, characterized in that the control means, after focusing on the person, switches the subject to be focused on from the person to the ball when the difference in distance in the depth direction between the person and the ball, or the difference in the degree of focus between the person and the ball, is greater than a second predetermined value determined in advance.

30. A second detection means for detecting a posture of each of the plurality of subjects detected by the first detection means; a reliability acquisition unit that acquires a reliability of each of the plurality of persons detected by the first detection unit as a main subject based on the posture of the person, 27. The image processing apparatus according to claim 26, wherein said control means, after focusing on the person, switches the subject to be focused on from the person to the ball based on the reliability.

31. The image processing device described in Claim 30, characterized in that when there is no person among the multiple detected people whose reliability is equal to or higher than a predetermined first specified value, the control means determines the subject to be focused on from the people and the ball based on the difference between the degree of focus of the people and the degree of focus of the ball.

32. The image processing device described in Claim 31, characterized in that when the difference in the degree of focus cannot be obtained, the control means focuses on a subject between a person and a ball which is closer to the center of the image.

33. The image processing device described in Claim 27, characterized in that the reliability acquisition means acquires the reliability using information about the ball in addition to the posture.

34. The image processing device according to claim 27, characterized in that the reliability acquisition means acquires the reliability using the positional relationship between the person and the ball in addition to the posture.

35. The image processing device as described in Claim 26, further comprising a second detection means for detecting a posture of each of the multiple persons detected by the first detection means based on information about the person's joints.

36. The image processing device described in Claim 35, characterized in that the posture is a posture for performing an action related to a ball.

37. An image processing device as described in any one of claims 1 to 13, further comprising a control means for focusing an imaging optical system on the determined main subject.

38. An imaging means for capturing an image; The image processing device according to claim 1 , which uses an image captured by the imaging means; and a control unit for focusing an imaging optical system on a main subject determined by the image processing device.

39. An image processing method executed by an image processing device, comprising: Detecting a first type of subject and a second type of subject different from the first type of subject from an image; detecting a posture of each of the detected first type subjects; acquiring a focus degree for each of the detected first type of subject and the detected second type of subject; determining a main subject from the detected first type of subject and the detected second type of subject based on the posture and the degree of focus.

40. An image processing method executed by an image processing device, comprising: Detecting a first type of subject and a second type of subject different from the first type of subject from an image; acquiring a focus degree for each of the detected first type of subject and the detected second type of subject; and after focusing on the first type of subject, switching the subject to be focused on from the first type of subject to the second type of subject based on a difference in distance in a depth direction between the first type of subject and the second type of subject, or a difference in a degree of focus between the first type of subject and the second type of subject.

41. An image processing method executed by an image processing device, comprising: Detecting a person and a ball from an image; acquiring a focus degree for each of the detected person and the detected ball; This image processing method includes: after focusing on the person, switching the subject to be focused on from the person to the ball based on a difference in distance in the depth direction between the person and the ball or a difference in the degree of focus between the person and the ball.

42. A program for causing a computer to function as each of the means possessed by an image processing device described in any one of claims 1 to 37.

Images