Focusing device and focusing method, and imaging device

Abstract

To provide a focus adjustment device that can suppress the deterioration of video quality caused by switching the main subject. [Solution] The focusing device detects a first type of subject and a second type of subject that is different from the first type of subject from the image, and focuses on the subject to be focused from among the detected subjects. When the subject to be focused switches from a first subject of the first type to a second subject of the first type, the focusing device performs a first control, which focuses on the third subject and then on the second subject, or a second control, which focuses on the second subject without focusing on the third subject, depending on the difference in depth between the second subject and a third subject of the second type, or the difference in the degree of focus between the second subject and the third subject.

Description

[Technical Field]

[0001] The present invention relates to a focusing device, a focusing method, and an imaging device. [Background technology]

[0002] Patent Document 1 describes a technique for determining the primary subject from among target subjects such as a person's face, which is closest to the subject being tracked. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2018-66889 [Overview of the project] [Problems that the invention aims to solve]

[0004] As described in Patent Document 1, when the main subject is determined, the distance from the imaging device to the main subject can change significantly depending on the switching of the main subject. If the switching of the main subject occurs frequently, the period during which no subject is in focus becomes longer, which can cause a decrease in the quality of the video.

[0005] In one embodiment, the present invention provides a focus adjustment device, a focus adjustment method, and an imaging device that can suppress the deterioration of video quality due to the switching of the main subject. [Means for solving the problem]

[0006] In one embodiment, the present invention provides a focus adjustment device comprising: a first detection means for detecting a first type of subject and a second type of subject of a different type from the first type of subject from an image; and a focus adjustment means for focusing on a subject to be focused from among the subjects detected by the first detection means, wherein when the subject to be focused switches from a first subject of the first type to a second subject of the first type, the focus adjustment means performs a first control, which focuses on the third subject and then focuses on the second subject, or a second control, which focuses on the second subject without focusing on the third subject, depending on the difference in depth distance between the second subject and a third subject of the second type, or the difference in the degree of focus between the second subject and the third subject. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to provide a focus adjustment device, a focus adjustment method, and an imaging device that can suppress the deterioration of video quality due to the switching of the main subject. [Brief explanation of the drawing]

[0008] [Figure 1] Block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to the embodiment. [Figure 2] A schematic diagram showing the pixel arrangement in the embodiment. [Figure 3] schematic plan view and schematic cross-sectional view of a pixel in the embodiment [Figure 4] A diagram illustrating the correspondence between the pixels of the image sensor in the embodiment and the pupil intensity distribution. [Figure 5] A diagram showing the relationship between the amount of defocus and the amount of image displacement in the embodiment. [Figure 6] Diagram showing the focus detection region in the embodiment. [Figure 7] Flowchart of the operation of the shooting mode in the embodiment [Figure 8] Flowchart relating to imaging and recording processes in the embodiment [Figure 9]Flowchart of subject tracking AF processing in an embodiment [Figure 10] Flowchart of subject detection and tracking processing in an embodiment [Figure 11] Explanatory diagram of pose information in an embodiment [Figure 12] Flowchart of main subject determination processing in an embodiment [Figure 13] Diagram showing an example of the structure of a neural network in an embodiment [Figure 14] Diagram related to a specific example of main subject determination processing in an embodiment [Figure 15] Diagram related to a specific example of main subject determination processing in an embodiment [Figure 16] Diagram related to a specific example of main subject determination processing in an embodiment [Figure 17] Flowchart of focus adjustment processing in an embodiment

Best Mode for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] In the following embodiments, the present invention will be described with respect to the case where it is implemented in an imaging device such as a digital camera. However, an imaging function is not essential for the present invention, and it can be implemented with any electronic device. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented with other electronic devices.

[0011] ●(Overall structure) Figure 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing apparatus according to this embodiment. The digital camera 100 (hereinafter simply referred to as camera 100) includes a first lens group 101, an aperture 102, a second lens group 103, and a third lens group 105, and has an imaging optical system that forms an optical image of a subject on the imaging surface of an image sensor 107.

[0012] The first lens group 101 is positioned furthest forward (on the subject side) of the multiple lens groups included in the imaging optical system and is movable along the optical axis OA. The position of the first lens group 101 is controlled by the zoom actuator 111. The zoom actuator 111 moves the first lens group 101 and the second lens group 103 in conjunction in the optical axis direction by, for example, driving a cam cylinder (not shown).

[0013] The aperture 102's opening amount can be adjusted by the aperture actuator 112, and it functions as a mechanical shutter when capturing still images. Alternatively, a separate mechanical shutter and a shutter actuator for driving the mechanical shutter may be provided.

[0014] The second lens group 103 moves along the optical axis OA, integrally with the aperture 102 and in conjunction with the first lens group 101. The angle of view (focal length) of the imaging optical system is determined by the positions of the first lens group 101 and the second lens group 103. The third lens group 105 is movable along the optical axis OA. The position of the third lens group 105 is controlled by the focus actuator 114. The position of the third lens group determines the focal distance of the imaging optical system. The third lens group is called the focus lens.

[0015] The optical low-pass filter 106 is provided to reduce false colors and moiré patterns that occur in the captured image. The image sensor 107 is a CMOS image sensor or CCD image sensor having a rectangular pixel array (also called a pixel area) in which, for example, m pixels in the horizontal direction and n pixels in the vertical direction are arranged in two dimensions. Each pixel is provided with, for example, a color filter according to the primary color Bayer array and an on-chip microlens. The image sensor 107 may also be a three-chip color image sensor.

[0016] In this embodiment, the imaging optical system, zoom actuator 111, aperture actuator 112, focus actuator 114, focus drive circuit 126, aperture drive circuit 128, and zoom drive circuit 129 are provided on the camera 100. However, these components may also be provided on a lens unit that is detachable from the camera 100.

[0017] The flash 115 is a light source that illuminates the subject. The flash 115 is equipped with a flash illumination device using a xenon tube or a continuously emitting LED (light-emitting diode). The AF (autofocus) auxiliary light source 116 projects a predetermined pattern image through a projection lens. This improves the focus detection capability for low-luminance or low-contrast subjects.

[0018] The CPU 121 controls the operation of the entire imaging device 100. The CPU 121 includes an A / D converter, a D / A converter, and a communication interface circuit. The CPU 121 controls various parts of the imaging device 100 by reading programs stored in the ROM 135 into the RAM 136 and executing them, thereby realizing the functions of the imaging device 100, such as autofocus detection (AF), imaging, image processing, and recording. In addition, some of the functions realized by the CPU 121 by executing programs may be implemented by hardware circuits separate from the CPU 121. Furthermore, reconfigurable circuits such as FPGAs may be used in some circuits. For example, in order to shorten the time required for calculations for focus detection, which will be described later, some of the calculations may be performed by dedicated hardware circuits.

[0019] The communication interface circuit provided by the CPU 121 supports one or more standards for wired and wireless communication. The imaging device 100 can communicate with external devices directly or via other devices through the communication interface circuit.

[0020] The flash control circuit 122 controls the illumination of the flash 115 in synchronization with the imaging operation. The auxiliary light source drive circuit 123 controls the illumination of the AF auxiliary light source 116 in synchronization with the focus detection process. The image sensor drive circuit 124 controls the imaging operation of the image sensor 107 and performs A / D conversion on the signals obtained from the imaging operation and transmits them to the CPU 121. The image processing circuit 125 can apply various image processing operations to the image data, such as gamma conversion, color interpolation, encoding, decoding, evaluation value generation, and feature region detection.

[0021] The focus drive circuit 126 drives the focus actuator 114 based on a command provided by the CPU 121, which includes the amount and direction of drive of the focus lens. As a result, the third lens group 105 moves along the optical axis OA, and the focusing distance of the imaging optical system changes.

[0022] The aperture drive circuit 128 drives the aperture actuator 112 to control the aperture diameter and opening / closing of the aperture 102. The zoom drive circuit 129 drives the zoom actuator 111, for example in response to user instructions, to move the first lens group 101 and the second lens group 103 along the optical axis OA, thereby changing the focal length (angle of view) of the imaging optical system.

[0023] The display unit 131 has, for example, an LCD (liquid crystal display device). The display unit 131 displays information related to the imaging mode of the imaging device 100, a preview image before imaging, a confirmation image after imaging, or an image showing the focus status when focus is detected. The operation unit 132 is configured to include a power switch, a release switch, a zoom operation switch, and an imaging mode selection switch. The release switch has two switches: SW1 which is ON when half-pressed and SW2 which is ON when fully pressed. The recording medium 133 is, for example, a semiconductor memory card that can be attached to and removed from the imaging device 100, and still image data and video data obtained by imaging are recorded on the recording medium 133.

[0024] If the display unit 131 is a touch display, a touch panel or a combination of the touch panel and a GUI displayed on the display unit 131 may be used as the operation unit 132. For example, if a tap operation on the touch panel is detected during live view display, the system can be configured to detect the focus using the image area corresponding to the tap position as the focus detection area.

[0025] Furthermore, the image processing circuit 125 can calculate contrast information from the captured image data, and the CPU 121 can perform contrast AF. In contrast AF, the focus lens group 105 is moved to sequentially calculate contrast information while changing the focus distance of the imaging optical system, and the focus lens position where the contrast information peaks is set as the focus position.

[0026] Thus, the imaging device 100 is capable of performing both image plane phase-detection AF and contrast AF, and can be selectively used or combined depending on the situation.

[0027] The subject detection unit 140 can be configured, for example, using a CNN (Convolutional Neural Network). By configuring the CNN using parameters (dictionary data) generated by machine learning for each type of subject, it detects the region of a specific subject present in the image represented by the image data. The subject detection unit 140 may also be implemented using dedicated hardware configured to execute CNN-based calculations at high speed, such as a GPU (Graphics Processing Unit) or an NPU (Neural Processing Unit).

[0028] Machine learning to generate dictionary data can be performed using any known method, such as supervised learning. Specifically, a CNN can be trained using a dataset in which input images are associated with whether or not the target subject is present, for each type of subject. The trained CNN or its parameters can be stored as dictionary data in the dictionary data storage unit 141. Note that CNN training may be performed on a device other than the camera 100. When using the trained CNN for subject detection processing of captured images, an image of the same size as the input image used during CNN training is extracted from the captured image and input to the CNN. By sequentially changing the extraction position and inputting it to the CNN, the region in which the target subject is present can be estimated.

[0029] Furthermore, after detecting object regions in an image, other methods may be used to detect the object region, such as determining what type of subject it is using feature quantities specific to each type of subject. The configuration and training method of the neural network can be changed depending on the detection method adopted.

[0030] The subject detection unit 140 can be implemented by any known method, as long as it can output the number, location, size, and confidence level of areas in the input image that are estimated to contain a predetermined type of subject.

[0031] The subject detection unit 140 can apply subject detection processing for multiple types of subjects to a single frame of image data by repeatedly applying subject detection processing while switching dictionary data. The CPU 121 can determine which dictionary data to use for subject detection processing from among the multiple dictionary data stored in the dictionary data storage unit 141, based on a pre-set priority for the types of subjects and the settings of the camera 100.

[0032] The types of subjects may include, but are not limited to, the human body, human organs (face, eyes, torso, etc.), or non-human subjects (animals, inanimate objects (tools, vehicles, buildings, etc.)). Separate dictionary data will be prepared for subjects with different characteristics. Here, as examples of non-human subjects, we will detect objects that move between competitors (between subjects of the first type) in sports, such as balls, pucks, and shuttles.

[0033] Dictionary data for detecting the human body may be prepared separately for detecting the human body (outline) and for detecting the organs of the human body. Dictionary data for detecting the organs of the human body may be prepared separately for each type of organ.

[0034] The posture acquisition unit 142 estimates the posture of each subject area detected by the subject detection unit 140. The information that the posture acquisition unit 142 should acquire regarding the subject's posture is predetermined according to the type of subject. For example, if the subject is a human, the posture acquisition unit 142 will acquire the positions of multiple joints as information regarding the subject's posture.

[0035] The method for estimating the subject's pose from the image of the subject area may be any known method. For example, the method described in Cao, Zhe, et al., "Realtime multi-person 2D pose estimation using part affinity fields," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017 can be used.

[0036] ●(Image sensor) The pixel arrangement and pixel structure of the image sensor 107 will be described with reference to Figures 2 and 3. In Figure 2, the left-right direction is the x-direction (horizontal direction), the up-down direction is the y-direction (vertical direction), and the direction perpendicular to the x and y directions (perpendicular to the plane of the paper) is the z-direction (optical axis direction). In the example shown in Figure 2, the pixel (unit pixel) arrangement of the image sensor 107 is shown in a 4x4 grid, and the sub-pixel arrangement is shown in an 8x4 grid.

[0037] In a 2x2 pixel array of 200 pixels, for example, pixel 200R, which has spectral sensitivity to the first color R (red), is located in the upper left; pixel 200G, which has spectral sensitivity to the second color G (green), is located in the upper right and lower left; and pixel 200B, which has spectral sensitivity to the third color B (blue), is located in the lower right. Furthermore, each pixel (unit pixel) is divided into 2 parts in the x direction (Nx division) and 1 part in the y direction (Ny division), resulting in a total of 2 divisions (N divisions). LF The first subpixel 201 and the second subpixel 202 (from the first subpixel to the Nth subpixel) (=Nx × Ny) LF It is composed of multiple subpixels.

[0038] In the example shown in Figure 2, each pixel of the image sensor 107 is divided into two subpixels arranged horizontally, thereby dividing the image signal obtained in a single image capture into N divisions. LFAn equal number of perspective images and a captured image obtained by synthesizing all the perspective images can be generated. Note that the pixels may be divided in two directions, and there is no limit to the number of divisions for each direction. Therefore, the perspective image can be an image generated from the signals of some of the plurality of sub-pixels, and the captured image can be an image generated from the signals of all the sub-pixels. In this embodiment, as an example, the pixel periods P in the horizontal and vertical directions of the image sensor 107 are 6 μm, the horizontal pixel number N H = 6000, and the vertical pixel number N V = 4000. Therefore, the total number of pixels N = N H × N V = 24 million. Also, if the horizontal period P S of the sub-pixels is 3 μm, the total number of sub-pixels N S = N H × (P / P S ) × N V = 48 million.

[0039] FIG. 3(a) shows a plan view of one pixel 200G of the image sensor 107 shown in FIG. 2 when viewed from the light-receiving surface side (+z side) of the image sensor 107. The z-axis is set in the direction perpendicular to the plane of FIG. 3(a), and the front side is defined as the positive direction of the z-axis. Also, the y-axis is set in the up-down direction perpendicular to the z-axis, and the upward direction is defined as the positive direction of the y-axis, and the x-axis is set in the left-right direction perpendicular to the z-axis and the y-axis, and the rightward direction is defined as the positive direction of the x-axis. FIG. 3(b) shows a cross-sectional view when viewed from the -y side along the a-a cutting line in FIG. 3(a).

[0040] As shown in FIGS. 3(a) and 3(b), a microlens 305 is formed on the light-receiving surface side (+z direction) of each pixel, and incident light is condensed by this microlens 305. Further, a plurality of photoelectric conversion units including a first photoelectric conversion unit 301 and a second photoelectric conversion unit 302, which are divided into two in the x (horizontal) direction and one in the y (vertical) direction, are formed. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 respectively correspond to the first sub-pixel 201 and the second sub-pixel 202 in FIG. 2. More generally described, the photoelectric conversion unit of each pixel is divided into Nx in the x direction and Ny in the y direction, and when the number of divisions N LF = Nx × Ny, the first to NLF The photoelectric conversion unit is the 1st to Nth LF Corresponds to sub-pixels.

[0041] The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn-junction photodiodes, each composed of a p-type well layer 300 and two divided n-type layers 301 and 302. If necessary, an intrinsic layer may be sandwiched between them to form a PIN structure photodiode. A microlens 305 and a color filter 306 are formed between the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 in each pixel. If necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit, or the color filter may be omitted.

[0042] Light incident on pixel 200G is focused by microlens 305 and further spectrally separated by color filter 306 before being received by first photoelectric conversion unit 301 and second photoelectric conversion unit 302, respectively. In the first photoelectric conversion unit 301 and second photoelectric conversion unit 302, electron-hole pairs are generated according to the amount of light received, and after separation in the depletion layer, electrons are accumulated. Meanwhile, holes are discharged to the outside of the image sensor 107 through a p-type well layer connected to a constant voltage source (not shown). The electrons accumulated in the first photoelectric conversion unit 301 and second photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via a transfer gate and converted into a voltage signal.

[0043] In this embodiment, the microlens 305 corresponds to the optical system in the image sensor 107. The optical system in the image sensor 107 may be configured to use a microlens as in this embodiment, or it may be configured to use materials with different refractive indices, such as waveguides. Furthermore, the image sensor 107 may be a back-illuminated image sensor with circuits etc. on the side opposite to the side having the microlens 305, or it may be a stacked image sensor with some circuits such as the image sensor drive circuit 124 and the image processing circuit 125. In addition, materials other than silicon may be used as the semiconductor substrate, and for example, organic materials may be used as the photoelectric conversion material.

[0044] ●(pupil division) Next, the pupil division function of the image sensor 107 in this embodiment will be described with reference to Figures 4 and 5(a). Figure 4 shows a cross-sectional view of the aa cross-section of pixels 200G arranged on the image sensor 107 of this embodiment shown in Figure 3(a), viewed from the +y side, and the pupil plane located at a distance Z away from the imaging surface 600 of the image sensor 107 in the z-axis direction (direction of the optical axis OA). In Figure 4, the x and y axes of the cross-sectional view are inverted compared to Figure 3 in order to correspond with the coordinate axes of the exit pupil plane. The imaging surface 600 of the image sensor 107 is located at the imaging plane of the imaging optical system.

[0045] The pupil intensity distribution (first pupil intensity distribution) 501 is in a substantially conjugate relationship with the light-receiving surface of the first photoelectric conversion unit 301, whose centroid is eccentric in the -x direction, via the microlens 305. Therefore, the first pupil intensity distribution 501 corresponds to the pupil region that can be received by the first sub-pixel 201. The centroid of the first pupil intensity distribution 501 is eccentric on the pupil surface towards the +xp side. Similarly, the pupil intensity distribution (second pupil intensity distribution) 502 is in a substantially conjugate relationship with the light-receiving surface of the second photoelectric conversion unit 302, whose centroid is eccentric in the +x direction, via the microlens 305.

[0046] Therefore, the second pupil intensity distribution 502 corresponds to the pupil region that can receive light with the second sub-pixel 202. The centroid of the second pupil intensity distribution 502 is eccentric on the pupil plane towards -xp. Furthermore, the pupil intensity distribution 500 is the pupil region that can receive light with the entire pixel 200G when the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (first sub-pixel 201 and second sub-pixel 202) are combined. In other words, the first pupil intensity distribution 501 is eccentric on the pupil plane towards +xp with respect to the center of the pupil intensity distribution 500, and the second pupil intensity distribution 502 is eccentric on the pupil plane towards -xp with respect to the center of the pupil intensity distribution 500.

[0047] Furthermore, in Figure 4, the pupil region 500 is the pupil region that can receive light with the entire pixel 200G, which includes the photoelectric conversion unit 301 and the photoelectric conversion unit 302 (first sub-pixel 201 and second sub-pixel 202).

[0048] Figure 5(a) shows a schematic diagram illustrating the correspondence between the image sensor and the pupil division in this embodiment. The light beam passing through the different pupil regions, the first pupil region 501 and the second pupil region 502, is incident on each (imaging) pixel of the image sensor from the imaging surface 800 at different angles and is received by the 2×1 divided photoelectric conversion units 301 and 302. In this embodiment, the pupil region is divided into two horizontally, but if necessary, the pupil may be divided vertically.

[0049] The image sensor 107 has an arrangement of imaging pixels, each having a first sub-pixel 201 that receives a light beam passing through the first pupil region 501 of the imaging optical system, and a second sub-pixel 202 that receives a light beam passing through the second pupil region 502 of the imaging optical system, which is different from the first pupil region. Therefore, the imaging pixels receive a light beam passing through the pupil region 500, which is the sum of the first pupil region 501 and the second pupil region 502 of the imaging optical system.

[0050] Furthermore, not all pixels of the image sensor 107 have multiple photoelectric conversion units (sub-pixels); instead, an imaging pixel, a pixel having only a first sub-pixel, and a pixel having only a second sub-pixel may be individually arranged. Alternatively, a pixel having two sub-pixels and a pixel having one photoelectric conversion unit equivalent to two sub-pixels may be arranged.

[0051] In this embodiment, the image processing circuit 125 generates a single phase-difference AF image data (first focus detection signal (image A)) by concatenating multiple signals obtained by adding the outputs of four first sub-pixels 201 for each repeating unit of the color filter shown in Figure 2. The image processing circuit 125 also generates a second focus detection signal (image B) by concatenating multiple signals obtained by adding the outputs of four second sub-pixels 202 for each repeating unit of the color filter.

[0052] Thus, the first and second sub-pixels 201 and 202 are used to generate the focus detection signal and are therefore also called focus detection pixels. By adding the outputs of the same type of sub-pixels present within the repeating unit of the color filter, a signal reflecting the R, G, and B color components is obtained, and it can be treated as a luminance signal with less bias in spectral sensitivity. When obtaining an image, the signal obtained is the sum of the output of the first sub-pixel 201 and the output of the second sub-pixel 202 on a pixel-by-pixel basis.

[0053] ●(Relationship between defocus amount and image displacement amount) This section describes the relationship between the amount of image shift of a pair of focus detection signals (image A and image B) generated by the image processing circuit 125 of this embodiment and the amount of defocus (degree of focus) of the optical system relative to the subject. Figure 5(b) shows a schematic relationship between the amount of defocus and the amount of image shift between the first focus detection signal (image A) and the second focus detection signal (image B). An image sensor 107 is arranged on the imaging surface 800, and as described with reference to Figures 4 and 5(a), the exit pupil of the imaging optical system is divided into a first pupil region 501 and a second pupil region 502.

[0054] The magnitude of the defocus amount d, |d|, is the distance from the image formation position of the subject to the image sensor 800. When the defocus amount d is negative (d<0), it means the image formation position of the subject is on the subject side of the image sensor 800, which is a front-focus state. When it is positive (d>0), it means the image formation position of the subject is on the opposite side of the image sensor 800, which is a back-focus state. When the image formation position of the subject is on the image sensor 800, which is a focused state, the magnitude of the defocus amount d is 0. In Figure 5(a), subject 801 is in focus (d=0) and subject 802 is shown as an example of a front-focus state (d<0). The front-focus state (d<0) and the back-focus state (d>0) together are called a defocus state (|d|>0).

[0055] In the front-focus state (d<0), the light beam from the subject 802 that has passed through the first pupil region 501 (second pupil region 502) is once focused at a position closer to the subject than the imaging plane 800. Then, it spreads out with a width Γ1 (Γ2) centered on the centroid position G1 (G2) of the light beam, resulting in a blurred image on the imaging plane 800. The blurred image is converted into an electrical signal by the first sub-pixel 201 (second sub-pixel 202) in each of the multiple pixels that receive the light. As described above, the phase-detection AF unit generates the first focus detection signal (image A) from the signal of the first sub-pixel 201 and the first focus detection signal (image B) from the signal of the second sub-pixel 202. Therefore, image A (image B) is recorded on the imaging plane 800 at the centroid position G1 (G2) as a subject image with the subject 802 blurred with a width Γ1 (Γ2).

[0056] The blur width Γ1 (Γ2) of the subject image increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. Similarly, the magnitude of the image displacement p (=difference in the centroid position of the light beam G1-G2) of the subject image between the first focus detection signal and the second focus detection signal |p| also increases roughly in proportion to the increase in the magnitude of the defocus amount d |d|. In the case of back focus (d>0), the relationship between the magnitude of the defocus amount |d|, the blur width of the subject image, and the image displacement p is the same, except that the direction of image displacement between image A and image B is opposite to that of the front focus state. Therefore, as the amount of defocus increases, the amount of image displacement between image A and image B also increases.

[0057] ●(Focus detection area) The pixel area of ​​the image sensor 107 used to generate the first focus detection signal and the second focus detection signal (focus detection area) will be described. Figure 6 shows an example of a focus detection area set in the effective pixel area 1000 of the image sensor 107, superimposed with the indicator of the focus detection area displayed on the display unit 131 when focus is detected. In this embodiment, a total of nine focus detection areas are set, three in the row direction and three in the column direction, but this is just one example, and more or fewer focus detection areas may be set. Also, the size, position, and spacing of the focus detection areas may differ.

[0058] Furthermore, in cases where every pixel in the effective pixel area 1000, such as in the image sensor 107, has a first sub-pixel 201 and a second sub-pixel 202, the position and size of the focus detection area may be dynamically set. For example, a predetermined range may be set as the focus detection area, centered on a position specified by the user. In this embodiment, the focus detection area is set so that a higher resolution focus detection result can be obtained when acquiring the defocus map described later. For example, each of the 9600 regions obtained by dividing the effective pixel area 1000 into 120 horizontally and 80 vertically is set as the focus detection area.

[0059] In Figure 6, the focus detection region at the nth row and mth column is represented as A(n,m), and the rectangular frame-shaped index representing the focus detection region A(n,m) is represented as I(n,m). The signals obtained from the first sub-pixel 201 and the second sub-pixel 202 within the focus detection region generate images A and B, which are used to detect the amount of defocus in that focus detection region. The index I(n,m) is usually superimposed on the live view image.

[0060] ●(Operation in imaging mode) The operation of camera 100 in imaging mode will be explained using the flowchart shown in Figure 7. When camera 100 is powered on and the startup process is completed, it operates in imaging mode.

[0061] In imaging mode, the camera 100 continuously captures video and displays the obtained video on the display unit 131, allowing the display unit 131 to function as an EVF while waiting for user input.

[0062] In S1, the CPU 121 starts driving the image sensor 107 via the image sensor drive circuit 124 in order to capture video for display on the display unit 131. Thereafter, the image sensor 107 outputs an analog image signal at a predetermined frame rate.

[0063] The CPU 121 acquires an analog image signal for one frame from the image sensor 107, and then applies correlated double sampling, A / D conversion, etc., to generate a digital image signal. The CPU 121 outputs the digital image signal to the image processing circuit 125. The image processing circuit 125 applies demosaicing and other processes to the digital image signal to generate image data for display. The image processing circuit 125 writes the image data for display to, for example, the video memory area of ​​RAM 136. The image processing circuit 125 also generates evaluation values ​​used for AE processing from the digital image signal and outputs them to the CPU 121. Furthermore, for each of the multiple focus detection areas, the image processing circuit 125 generates first and second focus detection signals based on signals read from pixels included in the focus detection area and outputs them to the CPU 121.

[0064] Furthermore, if the first and second sub-pixels 201 and 202 are configured as separate pixels (not sharing the same microlens), the pixel coordinates from which a signal is obtained from the first sub-pixel 201 will be different from the pixel coordinates from which a signal is obtained from the second sub-pixel 202. Therefore, the image processing circuit 125 interpolates the signals so that the signal pairs of the first and second sub-pixels 201 and 202 exist at the same pixel position to generate the first and second focus detection signals.

[0065] In S2, the CPU 121 supplies display image data stored in the video memory area of ​​RAM 136 to the display unit 131, which displays it as a single frame of live view image. The user can adjust the imaging range, exposure conditions, etc., while viewing the live view image displayed on the display unit 131. The CPU 121 determines the exposure conditions based on evaluation values ​​obtained from the image processing circuit 125, and displays an image showing the determined exposure conditions (shutter speed, aperture value, imaging ISO sensitivity) superimposed on the live view image on the display unit 131.

[0066] From this point onward, the CPU 121 executes operation S2 each time an image frame is captured. As a result, the display unit 131 functions as an EVF.

[0067] In S3, the CPU 121 determines whether a half-press operation (turning SW1 on) of the release switch included in the operation unit 132 has been detected. If the CPU 121 determines that SW1 is not turned on, it repeatedly executes S3. On the other hand, if it determines that SW1 is turned on, the CPU 121 executes S300.

[0068] In S300, CPU121 performs subject-tracking autofocus (AF) processing. In S300, CPU121 applies subject detection processing to the image data for display and determines the focus detection area. In addition, CPU121 performs predictive AF processing to suppress the decrease in AF accuracy caused by the time difference between executing AF processing and detecting the full press operation of the shutter release switch (SW2 on). Details of the operation in S300 will be described later.

[0069] In S5, CPU121 determines whether or not SW2 is turned on. If CPU121 determines that SW2 is not turned on, it executes S3. On the other hand, if it determines that SW2 is turned on, CPU121 executes the imaging and recording process in S400. Details of the operation in S400 will be described later.

[0070] In S7, the CPU 121 determines whether or not it has detected that the main switch included in the control unit 132 is off. If the CPU 121 determines that it has detected that the main switch is off, it terminates the operation of the imaging mode. On the other hand, if the CPU 121 determines that it has not detected that the main switch is off, it executes S3.

[0071] Although subject detection and AF processing were explained as being performed in response to the detection of SW1 being on, they can also be performed at a different time. If the subject tracking AF processing of the S300 is performed before the on status of SW1 is detected, it becomes possible to skip the half-press operation and start imaging immediately with a full-press operation.

[0072] ● (Imaging and recording processing) Next, using the flowchart shown in Figure 8, we will explain the imaging and recording process performed by the CPU 121 in S400 of Figure 7. In S401, the CPU 121 determines the exposure conditions (shutter speed, aperture value, imaging ISO sensitivity, etc.) through AE processing based on evaluation values ​​generated by the image processing circuit 125. The CPU 121 then controls the operation of each part to capture a still image according to the determined exposure conditions.

[0073] Specifically, the CPU 121 transmits the aperture value and shutter speed to the aperture drive circuit 128 to drive the aperture 102. The CPU 121 also controls the charge accumulation operation of the image sensor 107 through the image sensor drive circuit 124.

[0074] In step S402, the CPU 121 reads out one frame of analog image signals from the image sensor 107 via the image sensor drive circuit 124. Additionally, for pixels within the focus detection area, the signal from either the first or second sub-pixel 201 or 202 is also read out.

[0075] In S403, the CPU 121 performs A / D conversion on the signal read in S402 to obtain a digital image signal. The CPU 121 also applies defective pixel correction processing to the digital image signal using the image processing circuit 125. Defective pixel correction processing is a process that compensates for signals read from pixels from which normal signals cannot be read (defective pixels) by using signals read from surrounding normal pixels.

[0076] In step S404, the CPU 121 instructs the image processing circuit 125 to generate a still image data file for recording and first and second focus detection signals. The image processing circuit 125 applies image processing and encoding to the digital image signal after defective pixel correction processing to generate still image data for recording. Image processing may include, for example, demosaicing (color interpolation), white balance adjustment, gamma correction (gradation correction), color conversion, and edge enhancement. The image processing circuit 125 also applies encoding processing to the still image data in a manner appropriate to the format of the data file that stores the still image data.

[0077] In S405, the CPU 121 records an image data file containing the still image data generated in S404 and the sub-pixel signals read from the focus detection area in S402 onto the recording medium 133.

[0078] In S406, the CPU 121 records the device characteristic information, which is characteristic information of the camera 100, on the recording medium 133, associating it with the image data file recorded in S405.

[0079] Device characteristic information includes, for example, the following information: • Imaging conditions (aperture value, shutter speed, ISO sensitivity, etc.) • Information regarding the image processing applied to the digital image signal by the image processing circuit 125. • Information regarding the light-receiving sensitivity distribution of the imaging pixels and sub-pixels of the image sensor 107. • Information regarding vignetting of the imaging light beam within Camera 100 • Information on the distance from the mounting surface of the imaging optical system in camera 100 to the image sensor 107. • Information regarding manufacturing tolerances for Camera 100

[0080] Information regarding the light-receiving sensitivity distribution of imaging pixels and sub-pixels (hereinafter simply referred to as light-receiving sensitivity distribution information) is information regarding the light-receiving sensitivity of the image sensor 107 according to the distance from the intersection point of the image sensor 107 and the optical axis. Since the light-receiving sensitivity depends on the microlenses 305 and photoelectric conversion units 301, 302 of the pixel, the information may also be information regarding these. Furthermore, the light-receiving sensitivity distribution information may also be information regarding the change in sensitivity with respect to the angle of incidence of light.

[0081] In S407, the CPU 121 records lens characteristic information, which is characteristic information of the imaging optical system, onto the recording medium 133, corresponding to the still image data file recorded in S405.

[0082] Lens characteristic information includes, for example, the following: • Information regarding the exit pupil • Information regarding the frame of the telescope tube, etc., that emits the light beam. • Information on focal length and F-number at the time of imaging • Information regarding aberrations in the imaging optical system • Information regarding manufacturing tolerances of imaging optics. • Position of the focus lens 105 during image capture (distance from subject)

[0083] Next, in S408, the CPU 121 records image-related information, which is information about the still image data, on the recording medium 133, corresponding to the still image data file recorded in S405. The image-related information includes, for example, information about the focus detection operation before imaging, information about the movement of the subject, and information about the focus detection accuracy.

[0084] In addition, in S406-S408, the CPU 121 may also save device characteristic information, lens characteristic information, and image-related information in the ROM 135 in association with the image data file recorded in S405.

[0085] In step S409, the CPU 121 generates display image data by scaling the still image data in the image processing circuit 125 and displays it on the display unit 131. This allows the user to view the captured image. After a predetermined display time has elapsed, the CPU 121 terminates the imaging and recording process.

[0086] ●(Subject tracking AF processing) Next, the subject-tracking AF process in the S300 shown in Figure 7 will be explained using the flowchart shown in Figure 9.

[0087] In S301, the CPU 121 calculates the amount of image shift (phase difference) between the first and second focus detection signals generated in S2 for each of the multiple focus detection regions. The amount of image shift between signals can be determined as the relative position where the correlation between the signals is maximized. From the calculated amount of image shift, the CPU 121 calculates the amount of defocus, which is the degree of focus for each focus detection region.

[0088] As described above, in this embodiment, the effective pixel area 1000 is divided into 120 horizontally and 80 vertically, resulting in a total of 9600 areas, each of which is set as a focus detection area. The CPU 121 generates data (defocus map) that associates the defocus amount calculated for each area with the position of the area. The CPU 121 stores the generated defocus map, for example, in RAM 136.

[0089] In S302, the CPU 121 performs subject detection processing using the subject detection unit 140. The subject detection unit 140 detects areas of one or more types of subjects and outputs a detection result to the CPU 121 for each detected area, including the type of subject, the location and size of the area, and the reliability of the detection.

[0090] Furthermore, the CPU 121 performs a process to detect the position of the subject in the current frame (tracking process) based on the results of the subject detection process in the current frame and the results of the subject detection process in past frames. If the subject cannot be detected by the subject detection process using the pre-trained CNN in the subject detection unit 140, the CPU 121 can estimate the position of the subject in the current frame by tracking using other methods such as template matching. Details will be described later.

[0091] In S303, if the subject detection unit 140 detects a region of a subject with joints (for example, a human), the CPU 121 acquires posture information for each subject region using the posture acquisition unit 142.

[0092] An example of posture information acquired by the posture acquisition unit 142 will be explained using Figure 11. Here, in the image of the current frame shown in Figure 11(a), the regions of the human subjects 901 and 902 and the region of the ball 903 are detected as subject regions.

[0093] Subject 901 is holding a ball 903 with both hands and is an important subject (main subject) in the imaging scene. In this embodiment, the subject that the imager is most likely to be focusing on (main subject) is determined based on the posture information acquired from the subject area. On the other hand, subject 902 is a non-main subject. A non-main subject is a subject that is not the main subject among the detected subjects.

[0094] Figure 11(b) shows an example of the posture information of subjects 901 and 902, as well as the position and size of the ball 903. The posture acquisition unit 142 detects the position of the joints 911 of subject 901 as posture information for subject 901. Similarly, the posture acquisition unit 142 detects the position of the joints 912 of subject 902 as posture information.

[0095] Figure 11(b) shows an example of detecting the positions of the top of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles as joint positions. However, the posture acquisition unit 142 may detect only some of these joint positions, or it may detect other joint positions. In addition to joint positions, information such as the axes connecting the joints may also be detected as posture information. Any information related to the posture of the subject can be detected as posture information. Here, we assume that the posture acquisition unit 142 detects joint positions as posture information for a human subject.

[0096] The posture acquisition unit 142 detects the image coordinates (x,y) of joints 911 and 912. Here, the image coordinates (x,y) represent the pixel position in the image in a two-dimensional Cartesian coordinate system where the origin (0,0) is the central pixel or one of the four corner pixels of the image.

[0097] Furthermore, in S302, the subject detection unit 140 detects information about the region detected as the ball subject 903, including the image coordinates (x,y) of the centroid 413 and its size in the image (number of pixels of the length indicated by the arrow 914).

[0098] In S304, CPU 121 performs primary subject determination processing. Using the defocus map generated in S301, the subject detection results obtained in S302, and the posture information obtained in S303, CPU 121 determines the primary subject from among the detected subjects. The primary subject is the subject that the imager is most likely to intend to focus on. Details of the primary subject determination processing will be described later.

[0099] In S305, the CPU 121 sets the focus detection area to include the area of ​​the main subject determined in S304. For example, the CPU 121 sets one or more focus detection areas from among the 9600 configurable focus detection areas that are included in the area of ​​the main subject and whose detected defocus amount satisfies certain conditions. The conditions may be, for example, that the value indicating the reliability of the defocus amount is greater than or equal to a threshold, and that a defocus amount indicating the nearest subject distance is obtained.

[0100] In addition, the setting of the focus detection area in S305 may be based on the amount of defocus obtained in the next frame for all focus detection areas included in the area of ​​the main subject determined in S304.

[0101] In S306, CPU121 obtains the defocus amount for the focus detection area set in S305. The defocus amount obtained here may be the one calculated in S301, or it may be the defocus amount recalculated for a new frame.

[0102] In S307, the CPU 121 performs predictive AF processing for each subject area detected by the subject detection unit 140 in S302. Predictive AF processing is the process of predicting the amount of defocus in the subject area at the time of imaging the next frame. The CPU 121 generates time-series data of the amount of defocus for each subject area based on the defocus map generated in S301 for, for example, one or more past frames and the current frame. Then, the CPU 121 obtains the equation of the prediction curve using multivariate analysis (e.g., least squares method) based on the time-series data of the amount of defocus. The CPU 121 predicts the amount of defocus corresponding to the subject distance at the time of imaging of the next frame by substituting the imaging time of the next frame into the obtained prediction curve equation. Alternatively, time-series data of the position of the subject area may be generated to predict the 3D position of the subject at the time of imaging of the next frame.

[0103] For example, the three-dimensional position (X,Y,Z) of the subject is represented in an XYZ Cartesian coordinate system where the intersection of the imaging plane and the optical axis is the origin, and the optical axis is the Z-axis. From the image coordinates (X,Y) of the subject region and the time-series data of the defocus amount Z, the three-dimensional position of the subject at the time of the next frame can be predicted.

[0104] For human subjects, the amount of defocus corresponding to the subject distance at the time of shooting the next frame can be predicted from time-series data of joint positions. By using time-series data, it is possible to estimate the position even if the joint position could not be detected because it was hidden by other subjects. Whether part of the subject is hidden or the subject has gone out of frame can be determined by the number and location of the undetectable joint positions.

[0105] In S308, the CPU 121 performs focus adjustment processing for the next frame image based on the amount of defocus predicted in S307 for the main subject. The CPU 121 converts the amount of defocus into the drive direction and drive amount of the focus lens, and controls the focus actuator 114 through the focus drive circuit 126 to drive the focus lens 105. Details will be described later. Once the driving of the focus lens 105 is complete, the CPU 121 terminates the subject tracking AF processing.

[0106] ● (Subject detection and tracking processing) Next, we will explain the details of the subject detection and tracking process in S302 of Figure 9 using the flowchart shown in Figure 10.

[0107] In S2000, the CPU 121 sets the dictionary data to be used by the subject detection unit 140 by determining the type of subject to be detected by the subject detection unit 140. The type of subject to be detected can be determined based on predetermined priorities, camera settings (e.g., shooting mode), etc. For example, the dictionary data storage unit 141 may store dictionary data for "person," "vehicle," "animal," and "ball." The types of subjects may be classified more finely. For example, instead of "animal," dictionary data for "dog," "cat," "bird," "cow," etc. may be stored, or instead of "vehicle," dictionary data for "four-wheeled vehicle," "two-wheeled vehicle," "railway," "airplane," etc. may be stored.

[0108] The CPU 121 sets dictionary data for a specific type of subject if the camera 100 is set to a shooting mode intended for that type of subject. For example, if portrait mode or sports mode is set, it sets the dictionary data for "people". If sports mode is set, it also sets the dictionary data for "balls". Also, if "panning mode" is set, it sets the dictionary data for "vehicles".

[0109] If the camera 100 does not have a shooting mode set for a specific type of subject, the CPU 121 sets the subject dictionary data according to a predetermined priority. For example, it can set dictionary data for "people" and "animals".

[0110] The type of dictionary data and the method for determining which dictionary data to set are not limited to the methods described here. One or more dictionary data sets may be used. If only one dictionary data set is used, subjects detectable by that single dictionary data can be detected with high frequency. If multiple dictionary data sets are set, multiple types of subjects can be detected by switching dictionaries for each frame. Furthermore, if processing time allows, multiple types of subjects may be detected for the same frame. When detecting one type of subject per frame, the detection frequency of subjects with a first priority may be set higher than that of subjects with a lower second priority. Here, we assume that dictionary data for "person" and "ball" are set. For convenience, the following explanation assumes the detection of both the first and second types of subjects for the same frame.

[0111] In S2001, the CPU 121 uses the subject detection unit 140 to apply a first type of subject detection process to the image of the current frame. Here, the first type of subject is assumed to be "person". The subject detection unit 140 applies the subject detection process to the image of the current frame using the "person" dictionary data stored in the dictionary data storage unit 141. The subject detection unit 140 outputs the detection result to the CPU 121. At this time, the CPU 121 may also display the subject detection result on the display unit 131. The CPU 121 also saves the detected subject area to the RAM 136.

[0112] When the "person" dictionary data is set, the subject detection unit 140 detects multiple types of regions related to a person with different granularities, such as the "whole body" region, the "face" region, and the "eyes" region. While it is desirable to detect local areas such as the eyes and face of a person for use in focus detection and exposure control, there is a possibility that they cannot be detected if the face is not facing forward or is hidden by other subjects. On the other hand, it is unlikely that the whole body will not be detected at all. Therefore, multiple types of regions with different granularities are detected to increase the probability that some region of the "person" can be detected. The dictionary data can also be configured to detect multiple types of regions with different granularities for subjects other than people.

[0113] In S2002, the CPU 121 uses the subject detection unit 140 to apply a second type of subject detection process to the image of the current frame. Here, the second type of subject is assumed to be a "ball". The subject detection unit 140 uses the "ball" dictionary data stored in the dictionary data storage unit 141 to apply the subject detection process to the image of the current frame. The subject detection unit 140 outputs the detection result to the CPU 121. At this time, the CPU 121 may also display the subject detection result on the display unit 131.

[0114] Instead of using a "ball" dictionary data for subject detection, a combination of object detection processing and processing to recognize a ball from the detected objects may be used. Any known method such as YOLO can be used for object detection. YOLO is the method or a successor version described in Redmon, Joseph, et al., "You only look once: Unified, real-time object detection," Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. Note that "ball" also includes badminton shuttles and ice hockey pucks. In addition, sports equipment used by hand, such as rackets, bats, and golf clubs, may also be detected.

[0115] In S2003, CPU121 performs subject tracking by applying template matching to the current frame, using the subject region most recently detected in S2001 as a template. The image of the subject region itself may be used as the template, or information obtained from the subject region, such as brightness information, color histogram information, and feature point information such as corners and edges, may be used as the template. Any known method can be used for matching and updating the template. The result of the tracking process may be the position and size of the region in the current frame that is most similar to the template.

[0116] The tracking process in S2003 may be executed only if no subject of the first type was detected in S2001. By detecting a region similar to a previously detected subject region in the current frame, stable subject detection and tracking can be achieved. The CPU 121 terminates the subject detection and tracking process upon completion of the tracking process.

[0117] ● (Main subject detection process) Next, using the flowchart shown in Figure 12, we will explain the details of the main subject determination process in S304 of Figure 9.

[0118] In S4000, CPU121 obtains the defocus amount corresponding to the first and second types of subject regions detected in the subject detection and tracking process from the defocus map generated in S301. Then, CPU121 calculates the difference in defocus amounts corresponding to the first and second types of subject regions. If multiple instances of at least one of the first and second types of subject regions are detected, the difference in defocus amounts is calculated for each combination.

[0119] In S4001, CPU121 calculates the probability or confidence level that a first type of subject is the main subject. Here, we will explain the case where the probability that a first type of subject is the main subject in the image is calculated, but other probabilities or confidence levels may also be calculated. For example, as a confidence level, the reciprocal of the distance between the centroid position of the first type of subject area and the centroid position of the second type of subject area may be calculated.

[0120] (Method for calculating probability) Since the first type of subject is a person, we will explain a method for calculating the probability that the first type of subject is the main subject based on joint position and size. Here, we will describe a method using a neural network.

[0121] Figure 13 shows an example of the structure of a neural network. The neural network has an input layer 1001, a hidden layer 1002, and an output layer 1003. There may be multiple hidden layers 1002. Each layer contains multiple neurons 1004, and neurons 1004 in adjacent layers are connected to each other by synapses 1005.

[0122] The number of neurons 1004 in the input layer 1001 is equal to the dimension of the input data. Also, the number of neurons in the output layer 1003 is equal to the number of answers. Here, since we use a neural network to obtain two possible answers, whether a certain type of subject is the main subject or not, there are two neurons 1004 in the output layer. Using a neural network that classifies the input into two classes, we determine whether the first type of subject is the main subject or not (whether the probability is high or low).

[0123] Let wij be the weight of synapse 1005 connecting the i-th neuron 1004 in input layer 1001 and the j-th neuron in hidden layer 1002. Then, the output zj of the j-th neuron in hidden layer 1002 is given by the following equation.

number

[0124] In equation (1), xi represents the value input to the i-th neuron of input layer 1001. Since all neurons in input layer 1001 are connected to the j-th neuron of hidden layer 1002, the input values ​​of each neuron are weighted and added together before being input to the j-th neuron of hidden layer 1002.

[0125] The j-th neuron in the hidden layer 1002 outputs a value of activation function h, which takes the input value plus a bias bj as its argument. The bias bj is a parameter corresponding to the neuron's sensitivity. The activation function h is a function that converts the input value into a value representing the neuron's excitation state. Here, we will use ReLU (Rectified Linear Unit), but other functions such as the sigmoid function may also be used.

[0126] Let wkj be the weight of synapse 1005 connecting the j-th neuron in the hidden layer 1002 and the k-th neuron in the output layer 1003, and let bk be the bias of the k-th neuron in the output layer 1003. Then the value yk output by the k-th neuron 1004 in the output layer 1003 is given by the following formula.

number

[0127] In equation (3), zj is the output value from the j-th neuron of the hidden layer 1002, which is connected to the k-th neuron of the output layer 1003. i and k are the neuron numbers of the output layer 1003, where i,k = 1 or 2. The output yk of each neuron in the output layer 1003 is normalized by the softmax function shown in equation (4) so ​​that its sum is 1. If y1 corresponds to the classification of primary subject and y2 to the classification of non-primary subject, then f(y1) represents the probability of being the primary subject and f(y2) represents the probability of being the non-primary subject.

[0128] The input values ​​for the neural network are the coordinates of the joints of the person, and the coordinates and size of the ball. All weights and biases are then optimized through learning to minimize a loss function that uses the output probability and the correct label. The correct label is a binary value, 1 for the main subject and 0 for non-main subjects. The loss function can be any function that measures the degree of agreement with the correct label, such as the mean squared error. Here, as an example, the binary cross-entropy shown below will be used as the loss function.

number

[0129] In equation (5), m is the index of the subject to be trained. ym is equal to the probability value f(y1) output from the k=1 neuron in output layer 1003. tm is the correct label (0 or 1).

[0130] By optimizing the value of equation (5) to be small, the weights and biases can be learned so that the correct label and the output probability value are close together. The learned weights and biases are stored in the recording medium 133, and the CPU 121 can store them in RAM 136 and use them as needed. Multiple types of learned weights and biases can be prepared depending on the scene, etc.

[0131] CPU121 can obtain the probability f(y1) that a given subject area is the main subject area by inputting values ​​obtained from the subject area into a neural network to which pre-trained weights and biases (results of prior machine learning) have been applied.

[0132] Furthermore, during neural network training, subject information (in this case, joint positions) of the state immediately preceding an important action can be used as the state of the primary subject. For example, in a sport involving throwing a ball, the joint positions detected from an image of a person extending their hand forward to throw the ball can be used as one of the states of the primary subject during training.

[0133] The reason for performing this type of learning is to enable camera 100 to perform appropriate control over subjects that have performed actions that should be considered the main subject. For example, by automatically starting image recording when the probability value of a subject being the main subject exceeds a predetermined first value, it becomes possible to reliably record scenes that the camera operator would not be able to capture even if they manually gave shooting instructions. Alternatively, control may be performed using information on the typical time from the state used for learning (a hand holding a ball extended) to the important action (throwing the ball). This increases the likelihood of recording images at the moment an important action occurs, such as the moment the ball leaves the hand.

[0134] This section describes an example of determining whether something is the main subject based on probabilities obtained using a pre-trained neural network. However, other machine learning methods such as support vector machines or decision trees may be used if similar determination can be achieved. Furthermore, it is not limited to machine learning; a function that outputs a confidence level or probability value based on a certain model may also be constructed.

[0135] For example, in sports involving a ball, whether a person is the main subject can be determined based on the relative positions of the person and the ball. For instance, one could assume that the closer a person is to the ball, the higher the probability that the person is the main subject, and then determine whether they are the main subject using a monotonically decreasing function of the distance between the person and the ball. Alternatively, one could determine whether a person is the main subject using only the subject's posture information, without using information about the ball.

[0136] Depending on the subject's actions (e.g., passing, shooting), it may be better to use ball information or not. For example, in the case of a shooting motion, the distance between the person and the ball increases, but there is a possibility that the subject who shot should be the main subject. In this case, using only the subject's posture information, without using the distance between the ball and the person, may result in a determination that better matches the photographer's intention. On the other hand, when the subject is about to receive the ball, using the distance between the ball and the person in addition to posture information may result in a determination that better matches the photographer's intention. Thus, it may be possible to decide whether or not to consider information about the ball depending on the type of action that can be expected from the subject's posture information. In addition, data obtained by applying predetermined transformations such as linear transformations to the coordinates of each joint, or the coordinates and size of the ball, may be used as input data for the neural network.

[0137] Alternatively, time-series data of the person's posture, the positions of the person and the ball, the degree of defocus for each subject, and the probability of the subject being the main subject may be used as input data for the neural network. Furthermore, the coordinates of the person's joints at the time of imaging, and predicted values ​​of the ball's coordinates and size may also be used as input data for the neural network.

[0138] Furthermore, if the determination of the main subject frequently switches between subjects with a large difference in the amount of defocus, it is likely that this is contrary to the imager's intention. Therefore, the CPU 121 can suppress the switching if it determines from the time-series data of the subject determined to be the main subject that the main subject is frequently switching between subjects with a large difference in the amount of defocus. The CPU 121 can suppress the switching if, within a predetermined number of frames, the determination of the main subject switches (from A to B, and from B to A) two or more times between two subjects A and B whose difference in the amount of defocus is above a threshold.

[0139] The CPU 121 can suppress switching by, for example, correcting the confidence (probability) obtained for one of subjects A or B (for example, the closer subject) to be the main subject. Alternatively, the region containing both subjects A and B may be used as the region of the main subject. In this case, the CPU 121 can control the focusing distance and / or aperture value so that both subjects are in focus.

[0140] Predictive data may be used when the rate of change per unit time of the subject's or joint's position is large (for example, above a threshold), and not used when the rate of change is small (below the same or a different threshold). This allows for maintaining the accuracy of the confidence level indicating the main subject when the subject's posture changes are small, and for detecting changes in the main subject more quickly when the subject's posture changes are large.

[0141] As explained above, the CPU 121 calculates the confidence level (probability) of the primary subject for each region of the first type of subject. If multiple regions of the first type of subject are detected, the confidence level is calculated for each region, and the representative value is set as the confidence level of the first type of subject. The representative value may be, for example, the maximum value.

[0142] In S4002, CPU121 determines whether the confidence level of the first type of subject calculated in S4001 is equal to or greater than a first predetermined value. If it is determined to be equal to or greater than the first predetermined value, it executes S4005; otherwise, it executes S4003. As mentioned above, the confidence level (probability) can take values ​​from 0 to 100%, so the first predetermined value is also set within this range. Here, the first predetermined value is set to 80.

[0143] In S4003, CPU121 determines whether the difference in the amount of defocus could be calculated in S4000. If it is determined that it could be calculated, it executes S4004; otherwise, it executes S4007. The difference in the amount of defocus cannot be calculated if, for example, the subject moves or the imaging range changes, causing the first type of subject or the second type of subject to move out of the imaging range or be only partially included in the imaging range.

[0144] In S4004, the CPU 121 determines whether the difference in the amount of defocus between the first type of subject, a person, and the second type of subject, a ball, is greater than a second predetermined value. If it is determined to be greater than the second predetermined value, it executes S4006; otherwise, it executes S4007. The unit of the second predetermined value is the amount of defocus, and if F is the aperture value and δ is the allowable circle of confusion diameter, it can be set to 1Fδ.

[0145] In S4005, CPU121 determines that the subject with the highest confidence (probability) among the first type of subject is the primary subject.

[0146] In S4006, CPU121 determines the second type of subject as the primary subject. Here, it is assumed that one ball is detected, but if multiple balls are detected, CPU121 determines the subject with the highest confidence (probability) to be the primary subject as the primary subject.

[0147] S4006 is executed when the confidence (probability) that the first type of subject is the main subject is less than a first predetermined value, and the difference in the amount of defocus between the first type of subject and the second type of subject is greater than a second predetermined value. For example, in sports where a ball is passed, the situation where the ball is passed between people who are far apart in the depth direction meets this condition. By making the second type of subject (the ball) the main subject, the focus distance can be changed more smoothly than if the main subject were to be immediately switched from the person who passed the ball to the person who received the pass, by controlling the camera to keep the ball in focus.

[0148] In S4007, CPU121, similar to S4005, determines the subject with the highest confidence (probability) among the first type of subject to be the primary subject.

[0149] In S4008, CPU121 determines that either the first type of subject or the second type of subject is the primary subject. S4008 is executed when the difference in the amount of defocus cannot be calculated. If only one type of subject exists, CPU121 determines that the existing subject of that type is the primary subject. If multiple subjects of the same type exist, CPU121 may determine the subject with the highest confidence level or a subject that has been previously determined to be the primary subject as the primary subject. Furthermore, if only a portion of the same type of subject is included, CPU121 may take into account the imager's framing and determine the subject closest to the center of the image as the primary subject.

[0150] If the main subject is determined in any of S4005, S4006, S4007, or S4008, the CPU 121 terminates the main subject determination process.

[0151] A concrete example of the main subject determination process will be explained using Figures 14 to 16. Figure 14 shows a typical scene in chronological order where a basketball 903 is passed from person 902 to person 901. It is assumed that there is a sufficient difference in the amount of defocus (distance from the camera) between people 901 and 902. Here, a sufficient difference in the amount of defocus means that one person is in focus while the other is not sharp.

[0152] Figure 14(a) shows person 902 in a preparatory motion to pass ball 903. When the confidence level that person 902 is the main subject is calculated using the positions and sizes of person 902's joints and the positions and sizes of ball 903, a higher confidence level than that of person 901 and above the first predetermined value is obtained because person 902 is in a posture to pass the ball. Therefore, the flowchart shown in Figure 12 proceeds from S4002 to S4005, and person 902 is determined to be the main subject.

[0153] Figure 14(b) shows a state where ball 903 is moving away from person 902 and towards person 901, and person 901 is not in any position to receive ball 903. Neither person 901 nor 902 is in a position to pass or receive ball 903. Therefore, the confidence level of the main subject calculated for people 901 and 902 will be lower than the first predetermined value.

[0154] Furthermore, as the ball 903 moves from person 902 to 901, the difference between the amount of defocus of person 902, which was determined to be the main subject in the state shown in Figure 14(a), and the amount of defocus of the ball 903 becomes greater than the second predetermined value. Therefore, following the flowchart shown in Figure 12, the process proceeds through S4002, S4003, S4004, and S4006, and the ball 903, which is the second type of subject, is determined to be the main subject.

[0155] Figure 14(c) shows the ball 903 moving further, with person 901 in the motion of receiving the ball 903. When the confidence level that person 901 is the main subject is calculated using the positions and sizes of person 901's joints and the positions and sizes of the ball 903, since person 901 is in the posture of receiving the ball 903, the confidence level is higher than that of person 902 and is above the first predetermined value. Therefore, the flowchart shown in Figure 12 proceeds from S4002 to S4005, and person 901 is determined to be the main subject.

[0156] Figure 15 shows a typical scene in chronological order when the ball 923 is passed from person 922 to person 921 while playing basketball. Unlike in Figure 14, it is assumed that there is no significant difference in the amount of defocus (distance from the camera) between people 921 and 922.

[0157] Figure 15(a) shows person 922 in a preparatory motion to pass ball 923. When the confidence level that person 922 is the main subject is calculated using the positions and sizes of person 922's joints and the positions and sizes of ball 923, a higher confidence level than that of person 921 and above the first predetermined value is obtained because person 922 is in a posture to pass the ball. Therefore, the flowchart shown in Figure 12 proceeds from S4002 to S4005, and person 922 is determined to be the main subject.

[0158] Figure 15(b) shows a situation where ball 923 is moving away from person 922 and towards person 921, and person 921 is not in any position to receive ball 923. Neither person 921 nor 922 is in a position to pass or receive ball 923. Therefore, the confidence level of the main subject calculated for people 921 and 922 will be lower than the first predetermined value.

[0159] Furthermore, since there is not a sufficient difference in the amount of defocus (distance from the camera) between person 921 and 922, the difference between the amount of defocus of person 922, which is determined to be the main subject in the state shown in Figure 15(a), and the amount of defocus of ball 923 is less than or equal to the second predetermined value. Therefore, following the flowchart shown in Figure 12, the process proceeds through S4002, S4003, S4004, and S4007, and of the first type of subject, person 921 and 922, the one with the higher confidence level as the main subject is determined to be the main subject.

[0160] Figure 15(c) shows the ball 923 moving further, with person 921 in the motion of receiving the ball 923. When the confidence level that person 921 is the main subject is calculated using the positions and sizes of person 921's joints and the positions and sizes of the ball 923, since person 921 is in the posture of receiving the ball 923, the confidence level is higher than that of person 922 and is greater than or equal to the first predetermined value. Therefore, the flowchart shown in Figure 12 proceeds from S4002 to S4005, and person 921 is determined to be the main subject.

[0161] Figure 16 shows a typical time-series sequence of scenes in which a person 923 playing basketball shoots the ball 903 into a goal 925 located in front of person 924. It is assumed that there is a sufficient difference in the amount of defocus (distance from the camera) between people 924 and 923.

[0162] Figure 16(a) shows person 923 in a preparatory motion to shoot ball 903. When the confidence level that person 923 is the main subject is calculated using the positions and sizes of person 923's joints and the positions and sizes of ball 903, a higher confidence level than that of person 924 and above the first predetermined value is obtained because person 923 is in a posture to shoot. Therefore, the flowchart shown in Figure 12 proceeds from S4002 to S4005, and person 923 is determined to be the main subject.

[0163] Figure 16(b) shows a state where ball 903 is moving away from person 923 and towards goal 925, while person 923 maintains the shooting posture. In this state, because the ball is away from person 923, the confidence level calculated for person 923 falls below the first predetermined value. However, because person 923 maintains the shooting posture, the confidence level calculated for person 923 is higher than the confidence level calculated for person 924. Therefore, person 923 is determined to be the main subject in S4007 until the difference between the defocus amount of person 923 and ball 903 becomes greater than the second predetermined value. After that, when the difference between the defocus amount of person 923 and ball 903 becomes greater than the second predetermined value, ball 903 is determined to be the main subject.

[0164] If person 923 releases their shooting stance and person 924 reaches out to grab ball 903, the confidence that person 924 is the main subject will exceed that of person 923. Therefore, if the confidence that person 924 is the main subject exceeds the first predetermined value, or even if it does not exceed that value but the difference in the amount of defocus between person 924 and ball 903 becomes less than or equal to the second predetermined value, person 924 will be determined to be the main subject.

[0165] Consider the case where person 923 has released their shooting stance and ball 903 has been determined to be the main subject, and person 924 does not take any action such as reaching out to grab ball 903. In this case, when the difference in the amount of defocus between person 924 and ball 903 falls below a second predetermined value, the main subject may switch to person 923 or 924, and then possibly to ball 903. Therefore, even if the difference in the amount of defocus between person 924 and ball 903 falls below a second predetermined value, it may be possible to continue determining ball 903 as the main subject and suppress the switching of the main subject.

[0166] Finally, the details of the focus adjustment process in S308 of Figure 9 will be explained using the flowchart shown in Figure 17. In S5000, CPU121 determines whether the main subject will switch based on the result of the main subject determination process in S304. If it determines that the main subject will switch, it executes S5001; otherwise, it executes S5003.

[0167] In S5001, the CPU 121 determines whether the camera 100 is in video mode. Video mode is an operating mode in which video is captured and recorded for recording purposes, and is different from the state in still image mode in which video is captured for live view display. If the CPU 121 determines that the operating mode is video mode, it executes S5002; otherwise, it executes S5003.

[0168] In S5002, CPU121 controls the drive of the focus lens 105 to focus on the new main subject. When operating in video mode, video is recorded even while the focus lens is being driven, so CPU121 adjusts the drive speed of the focus lens 105 so that the focus distance does not pass the new main subject.

[0169] For example, when the main subject switches from a person to a ball, the focus lens 105 is driven at a speed that takes deceleration into account so that it can focus on the ball without overshooting the ball's distance. In this way, abrupt changes in the focus distance are suppressed, and a smooth transition of the focus distance can be achieved.

[0170] Furthermore, the focus adjustment speed may be changed depending on whether or not a subject of the type to be detected exists within that distance range. For example, consider the case where the main subject switches from a first type of subject to a second type of subject. In this case, the drive speed of the focus lens 105 in the distance range where the first type of subject exists is made slower than the drive speed of the focus lens 105 in the distance range where neither the first type of subject nor the second type of subject exists. Alternatively, the drive speed of the focus lens 105 in the distance range where neither the first type of subject nor the second type of subject exists is made faster than the drive speed of the focus lens 105 in the distance range where the first type of subject exists.

[0171] In distance ranges where neither the first nor the second type of subject exists, driving the focus lens 105 quickly has little impact on the video quality. Therefore, priority is given to reducing the time required to switch the focus subject. In this case as well, the focus lens is driven so that the focus distance does not exceed the distance of the second type of subject.

[0172] Note that this adjustment of the drive speed of the focus lens 105 may be performed even in operating modes other than video mode.

[0173] In S5003, CPU 121 drives the focus lens 105 to focus on the current main subject or a new main subject. Since it is not operating in video mode, no video is recorded while the focus lens 105 is being driven. Therefore, CPU 121 drives the focus lens 105 as fast as possible within the range that allows it to focus on the new main subject without overshooting the distance to the new main subject.

[0174] When focusing control to a new main subject is completed in S5002 or S5003, the CPU 121 terminates the focus adjustment process.

[0175] The image processing device according to this embodiment acquires a confidence level that each of the first types of subjects detected from the image is the main subject based on its posture. The image processing device then determines the main subject from the first and second types of subjects based on the degree of focus (amount of defocus) of the first and second types of subjects detected from the image, and the confidence level of the first type of subject. As a result, it becomes possible to switch the main subject to the second type of subject before the main subject switches between the first types of subjects which have different amounts of defocus. Consequently, it is possible to suppress a decrease in the quality of the video due to the switching of the main subject.

[0176] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0177] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0178] 100…Digital camera, 107…Image sensor, 121…CPU, 126…Focus drive circuit, 140…Subject detection unit, 142…Attitude acquisition unit

Claims

1. A first detection means for detecting a first type of subject and a second type of subject that is different from the first type of subject from an image, The system includes a focus adjustment means for focusing on a subject to be focused on among the subjects detected by the first detection means, The focus adjustment means adjusts when the subject to be focused changes from a first subject of the first type to a second subject of the first type. Depending on the difference in depth between the second subject and the third subject of the second type, or the difference in the degree of focus between the second subject and the third subject, A focus adjustment device characterized by performing a first control, which involves focusing on the third subject and then focusing on the second subject, or a second control, which involves focusing on the second subject without focusing on the third subject.

2. The system further includes a first acquisition means for acquiring the degree of focus for each subject detected by the first detection means, The focus adjustment device according to claim 1, characterized in that the distance difference in the depth direction and the difference in the degree of focus are calculated based on the degree of focus obtained by the first acquisition means.

3. The focus adjustment device according to claim 1, characterized in that the distance difference in the depth direction is calculated as the difference between the amount of defocus as the degree of focus of the second subject and the amount of defocus as the degree of focus of the third subject.

4. The focus adjustment device according to any one of claims 1 to 3, characterized in that the focus adjustment means performs the first control when the difference in distance in the depth direction or the difference in the degree of focus is greater than the second threshold.

5. The focus adjustment device according to claim 4, characterized in that the focus adjustment means performs the second control when the distance difference in the depth direction or the difference in the degree of focus is less than or equal to the second threshold.

6. The focus adjustment device according to claim 4 or 5, characterized in that the second threshold is set based on the aperture value and the allowable circle of confusion diameter related to the capture of the image.

7. The focusing device according to any one of claims 1 to 5, characterized in that the first type of subject is a person, and the second type of subject is an object moving between people.

8. A posture detection means for detecting the posture of each of the first types of subjects detected by the first detection means, Based on the aforementioned posture, a second acquisition means obtains the confidence level of the main subject for each of the first types of subjects detected by the first detection means, The focus adjustment device according to any one of claims 1 to 7, further comprising the above.

9. The focus adjustment device according to claim 8, wherein the first type of subject is a person, and the posture detection means detects the posture based on joint information.

10. The focusing device according to claim 8 or 9, characterized in that the subject to be focused is determined from among the first type of subjects detected by the first detection means, the first type of subject whose reliability is equal to or greater than the first threshold.

11. The focus adjustment device according to any one of claims 8 to 10, characterized in that the second acquisition means acquires the reliability using information of the second type of subject detected by the first detection means, in addition to the posture.

12. The focus adjustment device according to any one of claims 8 to 10, characterized in that the second acquisition means acquires the reliability using the positional relationship between the first type of subject and the second type of subject detected by the first detection means, in addition to the posture.

13. The focus adjustment device according to any one of claims 8 to 12, characterized in that the focus adjustment means performs the first control when the reliability of the first subject is less than a first threshold and the difference in distance in the depth direction or the difference in the degree of focus is greater than a second threshold.

14. The focus adjustment device according to any one of claims 1 to 13, characterized in that when the operating mode of the focus adjustment device is in video mode and the subject to be focused on changes, the drive speed of the focus lens is adjusted so that the focus distance does not exceed the distance of the new subject to be focused on.

15. The focus adjustment device according to claim 14, characterized in that the focus adjustment means makes the drive speed of the focus lens in a distance range where neither the first type of subject nor the second type of subject exists faster than the drive speed of the focus lens in a distance range where either the first type of subject or the second type of subject exists.

16. The focus adjustment device according to claim 1, characterized in that, if the difference in distance in the depth direction or the difference in the degree of focus cannot be obtained, the subject closer to the center of the image among the first type of subject and the second type of subject will be used as the subject to be focused.

17. The focus adjustment device according to claim 1, characterized in that, if the subject is not detected in the image of the current frame, the first detection means estimates the position of the subject in the current frame by tracking processing based on the region of a subject that was previously detected.

18. Image sensor and A focus adjustment device according to any one of claims 1 to 17, which uses an image acquired using the image sensor, An imaging device characterized by having the following features.

19. A focusing method performed by a focusing device, A first detection step of detecting a first type of subject and a second type of subject of a different type from the first type of subject from an image, The system includes a focus adjustment step which involves focusing on the subject to be focused from among the subjects detected in the first detection step, The aforementioned focus adjustment step is, When the subject to be focused switches from a first subject of the first type to a second subject of the first type, Depending on the difference in depth between the second subject and the third subject of the second type, or the difference in the degree of focus between the second subject and the third subject, A focusing method characterized by comprising the steps of: performing a first control to focus on the second subject after focusing on the third subject; or performing a second control to focus on the second subject without focusing on the third subject.

20. A program for causing a computer to function as each of the means of the focus adjustment device described in any one of claims 1 to 17.

Images