Imaging device, method for controlling, and program

Abstract

To properly set a focus detection region in a feature amount region of an object, with a motion direction of the object taken into consideration, even when a local region of the object cannot be detected for various different reasons.SOLUTION: The present invention includes: an acquisition unit for taking an image of an object and acquiring image data; a feature amount region detection unit for detecting a feature amount region in the image data acquired by the acquisition unit; a local region detection unit for detecting a local region to a local site of the object in the image data acquired by the acquisition unit; a motion direction detection unit for detecting a motion direction of the object; and a setting unit for setting a focal point detection region. The setting unit sets the focal point detection region in the motion direction of the object detected by the motion direction detection unit for a feature amount region detected by the feature amount detection unit when the local region detection unit does not detect the local region.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an imaging device having an autofocus function, and a control method and program thereof. [Background technology]

[0002] Conventionally, there has been known an autofocus technology in which an imaging device detects a subject and performs distance measurement calculations to automatically adjust the focus. For example, the imaging device described in Patent Document 1 is capable of performing optimal distance measurement calculations according to the subject by determining the detection direction of a focus detection signal according to the result of detecting the part of the subject.

[0003] In recent years, subjects are not limited to people. It is now possible to detect a wide variety of subjects, such as animals such as dogs, cats, birds, and horses, and vehicles such as airplanes and motorcycles. Furthermore, depending on the type of subject, it is possible to detect not only the "feature region" of the subject, but also the "local region" of the subject, such as the head and eyes of a person or animal, or the cockpit and driver of a vehicle. A feature region is a rectangular region that includes the features of the subject in the captured image, and a local region is a rectangular detection region that corresponds to a local part of the subject. It is desirable for the captured image to be in focus on such a local region of the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-192774 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, according to Patent Document 1, when continuously capturing images of a moving subject, if a feature region of the subject is detected but a local region is not detected, automatic focus detection is simply performed within the feature region.

[0006] When performing focus detection within the feature region of the subject, for example, if the flapping wings of a flying bird or the wings of an airplane are used as the focus detection target, the following problems may occur. For example, when a bird flaps its wings, the wings move violently, so if these are used as the focus detection target, a so-called perspective conflict occurs, causing variation in the focus detection results and reducing the focus accuracy. The wings of an airplane also rotate at high speed, so a similar problem occurs. However, even if the feature region of the subject has been detected, it is considered desirable to perform focus detection on local parts of the subject, such as the head and pupils. This is because the focus accuracy does not decrease when focus detection is performed on local areas of local parts of the subject.

[0007] An object of the present invention is to provide an imaging device, a control method and a program that can appropriately set a focus detection area taking into account the direction of movement of a feature area of ​​a subject when a feature area of ​​a subject is detected but a local area is not detected. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the present invention comprises an acquisition unit that images a subject and acquires image data, a feature area detection unit that detects a feature area contained in the image data acquired by the acquisition unit, a local area detection unit that detects a local area for a local portion of the subject contained in the image data acquired by the acquisition unit, a motion direction detection unit that detects the movement direction of the subject, and a setting unit that sets a focus detection area which is an autofocus detection area, wherein when a local area is not detected by the local area detection unit, the setting unit sets the focus detection area in the movement direction of the subject detected by the movement direction detection unit for the feature area detected by the feature area detection unit. Effect of the Invention

[0009] According to the present invention, when a feature region of a subject is detected but a local region is not detected due to various environmental factors, the orientation of the subject, etc., it is possible to appropriately set a focus detection region by taking into account the direction of movement of the feature region of the subject. [Brief description of the drawings]

[0010] [Figure 1] 1 is a configuration diagram of an imaging device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating the configuration of a subject tracking unit according to an embodiment of the present invention. [Diagram 3] 4 is a flowchart showing an overall imaging operation according to an embodiment of the present invention. [Figure 4] 4 is a flowchart showing "subject tracking" according to an embodiment of the present invention. [Diagram 5] 5 is a flowchart showing a "focus detection area setting process" according to an embodiment of the present invention. [Figure 6] 11A and 11B are explanatory diagrams of example regions and examples of setting of focus detection regions when feature amount regions and local regions of a subject are detected; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0012] <Figure 1: Configuration diagram of imaging device 100> 1 is a diagram showing the configuration of an imaging device 100 according to an embodiment of the present invention. The imaging device 100 of this embodiment is capable of capturing moving images and still images, and is configured to be capable of recording the image information in an image recording medium 157 or the like. The components of the imaging device 100 are configured to be capable of transmitting and receiving required information to each other via a bus 160. The operation of each component of the imaging device 100 is realized by a CPU 151 executing a program.

[0013] The imaging device 100 includes an imaging lens 101, a zoom control unit 113, an aperture control unit 105, a focus control unit 133, an imaging element 141, an imaging signal processing unit 142, an imaging control unit 143, and a position and orientation change acquisition unit 162. The imaging device 100 also includes a CPU 151, an image processing unit 152, an image compression unit 153, a RAM 154, a flash memory 155, a display unit 150, a subject tracking unit 161, an operation unit 156, an image recording medium 157, a power management unit 158, and a battery 159. The image recording medium 157 is detachable from the imaging device 100. Each component will be described below.

[0014] <Imaging lens 101 (lens unit)> The imaging lens 101 has a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed second group lens 121, a focus lens 131, a zoom motor 112, an aperture motor 104, and a focus motor 132. The fixed first group lens 102, the zoom lens 111, the aperture 103, the fixed second group lens 121, and the focus lens 131 constitute an "imaging optical system." For convenience, the fixed first group lens 102, the zoom lens 111, the fixed second group lens 121, and the focus lens 131 are illustrated as a single lens, but each lens may have a configuration including multiple lenses. The imaging lens 101 may be an interchangeable lens that is detachable from the imaging device 100.

[0015] <Zoom control unit 113: Aperture control unit 105: Focus control unit 133> The zoom control unit 113 drives and controls the zoom motor 112 that drives the zoom lens 111 to change and adjust the focal length (angle of view) of the imaging lens 101. The aperture control unit 105 drives and controls the aperture motor 104 that drives the aperture 103 to change the aperture diameter of the aperture 103. The focus control unit 133 calculates the defocus amount and defocus direction of the imaging lens 101 based on the phase difference between a pair of focus detection signals (image A (signal A) and image B (signal B)) obtained from the imaging element 141. The focus control unit 133 converts the calculated defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132, respectively. Then, the focus control unit 133 drives and controls the focus motor 132 based on the converted drive amount and drive direction. In this way, the imaging lens 101 is controlled to a focused state.

[0016] In this way, the focus control unit 133 performs automatic focus detection (autofocus: AF) using a "phase difference detection method." Note that the focus control unit 133 may also perform automatic focus detection (AF) using a "contrast detection method" based on a contrast evaluation value of an image signal obtained from the image sensor 141. There are no particular limitations on the method of automatic focus detection (autofocus).

[0017] <Imaging element 141: imaging signal processing unit 142: imaging control unit 143> The image sensor 141 has a plurality of pixels arranged in a matrix. For example, a large number of pixels are arranged in a matrix with "m rows" in the horizontal direction and "n columns" in the vertical direction ("m" and "n" are integers of 2 or more), and each pixel has two photoelectric conversion elements. An image of a subject generated by the imaging lens 101 is formed on the image sensor 141. The image of the subject formed on the image sensor 141 is converted into an electrical signal (image signal) by the photoelectric conversion element of each of the plurality of pixels of the image sensor 141.

[0018] The imaging control unit 143 controls the reading of image signals from the imaging element 141 in accordance with instructions from the CPU 151. The imaging signal processing unit 142 receives the image signals read from the imaging element 141. The imaging signal processing unit 142 performs signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing on the supplied image signals and outputs the processed signals to the imaging control unit 143. The imaging control unit 143 stores the image signals received from the imaging signal processing unit 142 in the RAM 154.

[0019] <Position and Posture Change Acquisition Unit 162> The position and orientation change acquisition unit 162 is realized by a position and orientation sensor such as a gyro, an acceleration sensor, an electronic compass, etc., and acquires information indicating a change in the position and orientation with respect to the shooting scene of the imaging device 100. The information indicating the position and orientation change acquired by the position and orientation change acquisition unit 162 is stored in the RAM 154 and referred to by the subject tracking unit 161.

[0020] <Image processing unit 152> The image processing unit 152 performs preset image processing on the image signal stored in the RAM 154. The image processing performed by the image processing unit 152 includes development processing such as white balance adjustment processing, color interpolation processing (demosaic processing), and gamma correction processing, as well as signal format conversion processing, scaling processing, and the like. The processing performed by the image processing unit 152 is not limited to these. The image processing unit 152 also generates information on subject brightness for performing automatic exposure control (AE). When performing AF using a contrast detection method, the image processing unit 152 also generates an "AF evaluation value." Information on a specific subject region may be supplied by the subject tracking unit 161 and used when the image processing unit 152 performs white balance adjustment processing and the like.

[0021] The image processing unit 152 stores the image signal that has been subjected to the various processes described above in the RAM 154. Furthermore, when the CPU 151 displays the image data stored in the RAM 154, the image processing unit 152 executes the above-mentioned scaling process so as to fit the display size of the display unit 150. Thereafter, the CPU 151 writes the scaled image data into a VRAM area formed in the RAM 154, whereby the image information and other required information are displayed on the display unit 150.

[0022] <Image compression unit 153: Image recording medium 157> When recording image data stored in RAM 154, CPU 151 generates a data file according to the recording format by, for example, adding a predetermined header to the image data. At this time, image compression unit 153 executes encoding and compression processing on the data file generated by CPU 151 as necessary. Image compression unit 153 records the encoded and compressed data on image recording medium 157, such as a smart card. Note that image recording medium 157 is not limited to a smart card, and may be any device that is detachable from imaging device 100 and capable of recording data in a non-volatile manner, such as a USB memory.

[0023] <Display section 150> The display unit 150 reads out and displays image data for display from a VRAM area formed in the RAM 154. The display unit 150 is realized by a display device such as an LCD or an organic EL display. The display unit 150 displays moving images captured by the imaging device 100 during moving image capture (shooting standby state, during moving image recording, etc.). This allows the display unit 150 to function as an electronic viewfinder (EVF). Note that the moving images and their frame images displayed when the display unit 150 functions as an EVF are referred to as "live view images" or "through images."

[0024] Furthermore, when a still image is captured by the imaging device 100, the display unit 150 displays the still image captured immediately before for a predetermined period of time so that the user can check the imaging result. The CPU 151 can superimpose an indicator indicating the position of the current subject area (for example, a "rectangular frame" surrounding the subject area) on the display image of the display unit 150. These display operations performed by the display unit 150 are realized under the control of the CPU 151.

[0025] <Operation unit 156> The operation unit 156 allows the user to give instructions to the imaging device 100, and is realized by a switch, a button, a key, a touch panel, etc. The instructions given by the user operating the operation unit 156 are notified to the CPU 151 via the bus 160, and the CPU 151 controls each unit to realize an operation according to the notified instructions.

[0026] <Flash memory 155: Power management unit 158: Battery 159> Battery 159 is managed by power management unit 158 ​​and supplies power to the entire imaging device 100. Flash memory 155 non-volatilely stores programs executed by CPU 151, various data such as parameters necessary for the execution process of the programs, GUI data, setting values ​​already set by the user, and the like.

[0027] <cpu151:ram154> The CPU 151 includes one or more processors, and for example, loads a program stored in a flash memory 155 into the RAM 154 and executes the loaded program. This allows the various units shown in FIG. 1 to be controlled to control the entire imaging device 100. Other processors such as a DSP or MPU can be used instead of the CPU 151. For example, when an instruction is given to switch from a power-off state to a power-on state by operating the operation unit 156, the CPU 151 loads a program stored in the flash memory 155 into the RAM 154 and executes it. Also, the program may be recorded in a non-volatile manner in a semiconductor recording medium such as a ROM instead of the flash memory 155.

[0028] Furthermore, the CPU 151 executes AE processing to automatically determine exposure conditions (shutter speed, accumulation time, aperture value, ISO sensitivity, etc.) based on luminance information of the subject. The luminance information of the subject can be acquired from, for example, the image processing unit 152. The CPU 151 can determine the exposure conditions based on the area of ​​a specific subject, such as a person's face. The CPU 151 controls the aperture control unit 105 to fix the aperture during video shooting, and performs exposure control according to the electronic shutter speed (accumulation time) and the magnitude of the read gain of the image signal from the image sensor 141. The CPU 151 notifies the imaging control unit 143 of the determined accumulation time and the magnitude of the gain. In response to this, the imaging control unit 143 controls the operation of the image sensor 141 so that shooting is performed according to the notified exposure conditions.

[0029] <Subject tracking section 161> The subject tracking unit 161 detects a specific subject from a captured image and continuously tracks the detected subject in multiple image data. The subject tracking unit 161 has at least the following three types of functions. That is, it has a "subject detection function" that detects the position and size (dimensions) of a rectangular area circumscribing the subject, a "subject tracking function" that continuously searches for and tracks areas similar to the detected area, and a "motion vector detection function" that determines a motion vector between image data.

[0030] The processing results of the subject tracking unit 161 can be used, for example, to automatically set a "focus detection area" which is a detection area for autofocus. As a result, a tracking AF function for a specific subject area can be realized. In addition, it is also possible to perform AE processing based on the luminance information of the focus detection area, and image processing (gamma correction processing, white balance adjustment processing, etc.) based on the pixel values ​​of the focus detection area. In addition, it is also possible to perform image blur correction by using the motion vector between image data calculated by the subject tracking unit 161.

[0031] Specifically, the image processing unit 152 refers to the motion vector between image data calculated by the subject tracking unit 161 and calculates the blur component of the imaging device 100. To correct this blur component, the CPU 151 drives and controls the zoom control unit 113, focus control unit 133, etc. to drive the lenses in the imaging lens 101. This makes it possible to perform image blur correction. Alternatively, a configuration may be adopted in which the imaging element 141 is provided with a drive mechanism (not shown), and the drive mechanism is driven to control the position of the imaging element 141 itself so as to perform blur correction of the imaging device 100.

[0032] <Figure 2: Configuration diagram of the subject tracking unit 106> Fig. 2 is a detailed configuration diagram of the subject tracking unit 106 in Fig. 1. The subject tracking unit 106 has a subject detection unit 201, a tracking feature amount region registration unit 202, a motion vector calculation unit 203, a subject motion vector calculation unit 204, a local region motion vector calculation unit 205, a reliability calculation unit 206, a weight map creation unit 207, and a tracking unit 208.

[0033] <Subject detection unit 201> The subject detection unit 201 (feature region detection unit, local region detection unit) is sequentially supplied with time-series image data from the image processing unit 152, detects specific subjects of the imaging target contained in each image data, and identifies the "feature region" and "local region" containing the detected specific subject. As a result of subject detection, the subject detection unit 201 outputs type information of the type of subject, "subject position information" indicating the position information of the subject region within the image data, and "reliability information" indicating the reliability of the detection accuracy. Examples of subject types include "people, animals, birds, horses, vehicles, airplanes," etc. A local region is a rectangular region in the image data that corresponds to a local part of the subject, such as the head, pupil, or cockpit, and circumscribes the local part.

[0034] <Tracking feature area registration unit 202> The tracking feature amount region registration unit 202 registers a feature amount region of the image data of the subject region detected by the subject detection unit 201 .

[0035] <Motion vector calculation unit 203: object motion vector calculation unit 204: local area motion vector calculation unit 205> The motion vector calculation unit 203 (motion direction detection unit) calculates the "motion vector" of the subject between image data that is sequentially supplied. More specifically, the motion vector calculation unit 203 calculates the motion direction of the subject from the difference between feature areas detected in the sequentially supplied image data. The subject motion vector calculation unit 204 calculates the overall motion vector of the feature areas registered by the tracking feature area registration unit 202. The local area motion vector calculation unit 205 calculates the motion vector of a local area in the feature area registered by the tracking feature area registration unit 202.

[0036] <Reliability calculation unit 206: Weight map creation unit 207: Tracking unit 208> The reliability calculation unit 206 calculates the reliability of the motion vector calculated by the motion vector calculation unit 203. The weight map creation unit 207 creates a weight map of the feature area registered by the tracking feature area registration unit 202 using the motion vector of the local area calculated by the local area motion vector calculation unit 205 and the reliability calculated by the reliability calculation unit 206. The tracking unit 208 uses the weight map created by the weight map creation unit 207 to search for a feature area having a high similarity to the feature area registered by the tracking feature area registration unit 202 as a subject area from the image data supplied sequentially. The search result is information such as the subject area, reliability, and motion vector of the subject in the image data, and is used by various processing blocks such as the CPU 151.

[0037] <Figure 3: Flowchart showing the overall imaging operation> 3 is a flowchart showing the overall process of the imaging operation, which will be described with reference to FIG. 3. The imaging operation of the imaging device 100 of this embodiment, such as subject detection, subject tracking, motion vector detection between images, focus detection area setting, focus detection processing, lens driving, etc.

[0038] Each process in the flowchart shown in Fig. 3 is executed by the CPU 151 itself or each unit that receives an instruction from the CPU 151. Movie shooting is executed during shooting standby or during movie recording. Note that the resolution of images (frames) handled during shooting standby and during movie recording differs in detail. On the other hand, the contents of subject detection, subject tracking, and subject and background motion vector detection processes are basically the same in both cases, so they will not be distinguished below.

[0039] First, in S301, the CPU 151 determines whether or not the power switch of the imaging device 100 is ON. If the CPU 151 determines that the power switch is not ON (No), the CPU 151 ends the process. On the other hand, if the CPU 151 determines that the power switch is ON (Yes), the CPU 151 advances the process to S302.

[0040] Next, in S302, the CPU 151 controls each unit to execute imaging processing for “one frame” and advances the process to S303. Here, a pair of parallax images and a captured image for one screen are generated, and the CPU 151 stores these in the RAM 154.

[0041] Next, in S303, CPU 151 controls subject tracking section 161 to cause subject tracking section 161 to execute "subject detection processing, local area presence / absence determination processing, subject type detection processing, subject tracking processing, and motion vector detection processing for detecting motion vectors between images." Details of the processing will be described later. Subject tracking section 161 notifies CPU 151 of the results of each processing. Specifically, subject tracking section 161 notifies CPU 151 of "the position, size, motion vector and subject type of the subject area, and local area presence / absence determination results," etc., and CPU 151 stores the notified information in RAM 154 and proceeds to processing at S304.

[0042] Next, in S304, the CPU 151 sets a "focus detection area" based on information such as the "position, size, motion vector, and type of object area" obtained in S303, and proceeds to S305. Details of the process will be described later. Here, the "focus detection area" is the area targeted by the autofocus that is the target area for distance measurement when the autofocus is used.

[0043] Next, in S305, the CPU 151 drives and controls the focus control unit 133 to execute focus detection processing. That is, the focus control unit 133 generates an "A image" by connecting together a plurality of "A signals" obtained from a plurality of pixels arranged in the same row of the image sensor 141 in a plurality of pixels included in a focus detection area of ​​a pair of parallax images previously stored in the RAM 154. Similarly, the focus control unit 133 generates a "B image" by connecting together a plurality of "B signals" arranged in the same row of the image sensor 141.

[0044] Then, the focus control unit 133 calculates the correlation amount between the "image A" and the "image B" while shifting the relative positions of the "image A" and the "image B", and obtains the relative position at which the similarity between the "image A" and the "image B" is highest as the "phase difference (shift amount)" between the "image A" and the "image B". Furthermore, the focus control unit 133 converts the obtained "phase difference" into the defocus amount and the defocus direction, and proceeds to the process at S306.

[0045] Next, in S306, focus control unit 133 drives and controls focus motor 132 in accordance with the lens drive amount and drive direction corresponding to the defocus amount and defocus direction calculated in S305, to move focus lens 131. When this lens drive process ends, CPU 151 returns the process to S301.

[0046] Thereafter, until CPU 151 determines in S301 that the power switch is not ON (until S301 returns "No"), CPU 151 repeatedly executes the processes of S302 to S306. This allows a subject area to be searched for in multiple pieces of time-series image data, achieving a subject tracking function. Note that, although the subject tracking process is executed for each frame in FIG. 3, it may be executed for multiple frames to reduce the processing load, power consumption, and the like.

[0047] <Figure 4: Flow of subject tracking processing> Fig. 4 is a flowchart showing the subject tracking process. The process of subject tracking unit 161 will be described with reference to Fig. 4. Each process in the flowchart of Fig. 4 is executed by CPU 151 itself or each unit that receives an instruction from CPU 151. First, in S401, input image data is supplied from imaging control unit 143 to subject tracking unit 161 under the control of CPU 151. The following description will focus on the operation of each unit in Fig. 2.

[0048] Next, in S402, the subject detection unit 201 sets multiple "evaluation areas" with different center positions and sizes for the image data input from the imaging control unit 143, and detects subjects from each of the set "evaluation areas." Any known method may be used for the subject detection method. For example, the subject can be automatically detected using a "feature extraction process" of a specific subject by CNN (Convolutional Neural Networks), which is one aspect of deep learning. Alternatively, the user may specify the "subject" by performing a touch operation using the operation unit 156 on the subject displayed on the display unit 150. The subject specified by the user is grasped by the CPU 151.

[0049] Next, in S403, the tracking feature area registration unit 202 registers, as a "feature area", the subject area detected by the subject detection unit 201, or the subject area of ​​the previous frame detected by the tracking unit 208 described below and stored in RAM 154. In this embodiment, the feature area registration method employs the result of the subject detection unit 201 for the first frame, and employs the processing result of the tracking unit 208 for the previous frame for the second frame and onward.

[0050] Next, in S404, the motion vector calculation unit 203 uses the current frame and the image of the previous frame from the images sequentially supplied in S401 to calculate a motion vector (first motion vector) for each pixel. Any known method may be used to calculate the first motion vector. In this embodiment, the "Lucas Kanade method" is adopted.

[0051] Let the brightness Y of coordinates (x, y) in the frame at time "t" be (x, y, t), and the brightness of the pixel after movement in the frame "Δt" later be "Y(x + Δx, y + Δy, t + Δt)." Then, by solving the following "Equation 1," "Δx" and "Δy" are calculated as the "motion vector" of each pixel.

[0052]

number

[0053] The "first motion vector" calculated here indicates the amount of movement of successive captured images per unit time, i.e., the amount of movement of the imaging device. For example, image processing unit 152 can perform image blur correction control by driving the lens with reference to the value of this first motion vector.

[0054] Next, in S405, subject motion vector calculation unit 204 calculates a "motion vector (second motion vector)" of the entire subject. Specifically, subject motion vector calculation unit 204 calculates a "motion vector (second motion vector) of the subject" from the "first motion vector" calculated in S404 and the feature area registered in S403. Then, in S406, tracking unit 208 tracks the subject from the images sequentially supplied using the feature area registered in tracking feature area registration unit 202 by S403. The above is the process executed by subject tracking unit 161.

[0055] <Figure 5: Flow of focus detection area setting process> Fig. 5 is a flowchart showing the focus detection area setting process. The focus detection area setting process will be described with reference to Fig. 5. Each process in the flowchart of Fig. 5 is executed by the CPU 151 itself or each unit that receives an instruction from the CPU 151. In S303 described above, the CPU 151 is notified by the subject tracking unit 161 of information on whether or not a "local area" of the subject has been detected, and this information is stored in the RAM 154.

[0056] First, in S501, CPU 151 determines whether or not a local area of ​​the subject exists by referring to "local area" presence / absence determination information stored in RAM 154. If CPU 151 determines that a local area of ​​the subject has been detected (Yes), the process proceeds to S507. On the other hand, if CPU 151 determines that a local area of ​​the subject has not been detected (No), the process proceeds to S502.

[0057] As described above, in S303, CPU 151 is notified of information such as the position and size of the subject area by subject tracking section 161, and the notified information is stored in RAM 154. Therefore, in S502, CPU 151 refers to the information stored in RAM 154 and determines whether or not a "subject detection area" exists based on the position and size of the subject area. If CPU 151 determines that a "subject area" has been detected (Yes), the process proceeds to S503. On the other hand, if CPU 151 determines that a "subject area" has not been detected (No), the process proceeds to S506. Therefore, the process proceeds to S503 when a "subject area" has been detected and a "local area" has not been detected.

[0058] Next, before performing the process of S503, in S303, CPU 151 is notified of the type of the subject from subject tracking unit 161 and stores it in RAM 154. Therefore, in S503, CPU 151 determines the type of the subject stored in RAM 154. Here, a determination is made of a subject having a local area in the moving direction of the subject. Such subjects are, for example, a bird, an airplane, etc. If CPU 151 determines that the subject has a local area in the moving direction of the subject (see reference symbol "A"), it advances the process to S504. On the other hand, if CPU 151 determines that the subject does not have a local area in the moving direction of the subject (see reference symbol "B"), it advances the process to S506. That is, in S503, if CPU 151 determines that the type of the subject is, for example, a bird, an airplane, etc. (A), it advances the process to S504, whereas if it determines that the type of the subject is not, for example, a bird, an airplane, etc. (B), it advances the process to S506.

[0059] Next, before executing S504, in S303, CPU 151 is notified of the "motion vector" of the subject from subject tracking section 161 and stores it in RAM 154. In S504, CPU 151 refers to the "motion vector" stored in RAM 154 to determine whether or not the subject is moving. If CPU 151 determines that the subject is moving (Yes), it advances the process to S505. On the other hand, if CPU 151 determines that the subject is not moving (No), it advances the process to S506.

[0060] In S505, the CPU 151 performs the following process since it has determined that the subject is moving and has a "local area" in the direction of movement of the subject, although the "local area" of the subject has not been detected, and the subject is moving and has a feature area in the direction of movement of the subject. That is, the CPU 151 (setting unit) sets a focus detection area in the feature area of ​​the subject in the direction of movement of the subject. This makes it possible to set a focus detection area near the local area of ​​the subject even if the local area of ​​the subject is not detected. In addition, by setting a focus detection area near the local area of ​​the subject, it is possible to prevent variation in focus detection caused by, for example, the vigorous flapping of a bird's wings or the high-speed rotation of an airplane's wings.

[0061] In addition, in S506, the CPU 151 sets the focus detection area to the entire feature area of ​​the subject in the following cases: (1) A local area of ​​the subject has not been detected and the feature area of ​​the subject has not been detected (No in S501, No in S502). (2) The subject has a feature area of ​​the subject but no local area in the moving direction of the subject (No in S501, Yes in S502, B in S503). (3) The subject has a local area (however, the local area itself has not been detected) in the moving direction of the subject but is not moving (No in S501, Yes in S502, A in S503, No in S504). In these cases, since there is not necessarily a local area in the moving direction, the focus detection area is set to the entire feature area of ​​the subject. Note that the CPU 151 may not set the focus detection area when it is determined that the motion vector of the subject is not detected by the motion vector calculation unit 203 (motion direction detection unit) (i.e., the motion direction cannot be detected).

[0062] Then, in S507, since the local region of the subject has been detected, the CPU 151 sets a focus detection region in the local region of the subject.

[0063] <Figure 6: Examples of feature regions, local regions, and focus detection region settings> Fig. 6 shows an example of a region and a focus detection region setting example when a feature region and a local region of a subject are detected. Fig. 6(a) shows a feature region 601 of a subject as a rectangle. Fig. 6(b) shows a bird's head, which is a local region 602 of the subject, as a rectangle. Fig. 6(c) and Fig. 6(d) show focus detection regions 603 and 604 as rectangles. In this case, the focus detection results vary due to rapid movement of the wings, etc., and the focus accuracy decreases.

[0064] Fig. 6(e) shows an example in which a local region of the subject is not detected but a feature region 605 of the subject is detected, and the subject is determined to have moved and to have a local region in the direction of movement. Fig. 6(e) shows an example in which a focus detection region 606 is set in the direction of movement of the subject within the feature region 605 of the subject.

[0065] In this way, even if a local area of ​​the subject is not detected, the focus detection area can be set near the local area of ​​the subject. Also, by setting the focus detection area near the local area of ​​the subject, it is possible to prevent variation in focus detection even if a perspective conflict occurs in autofocus due to, for example, the violent movement of the wings of a bird in flight or the high-speed rotation of the wings of an airplane.

[0066] <Summary and Effects of the Embodiment> According to the present embodiment, a focus detection area is set in the direction of movement of the subject in a "feature area" included in image data obtained by capturing an image of the subject formed by an imaging optical system. The reason for setting the focus detection area in the direction of movement of the subject within the "feature area" is that localized parts of a subject, such as a flying bird or airplane, exist in the direction of movement of the subject.

[0067] When a local region of the subject is detected, the focus detection region can be set as the local region, and there is no particular problem. However, due to various environmental factors, the orientation of the subject, and the like, there are cases where a local region for a local part of the subject cannot be detected even though the feature region of the subject is detected. In such cases, the focus detection region is set to a rectangular region within the "feature region" of the subject, in which the local region (local part) of the subject is assumed to be located in the region in the direction of the subject's movement detected by the motion vector calculation unit 203.

[0068] In this way, when the feature region of the subject is detected but the local region of the subject is not detected, the following effect can be obtained by setting the focus detection region in the feature region of the subject in the direction of the subject's movement. That is, even when the focus detection region cannot be set for the local region of the local part of the subject, appropriate auto focus detection can be performed for the focus detection region set in the feature region of the subject by referring to the movement direction of the subject. This can be understood as being able to set a desirable focus detection region by avoiding the local part of the subject where the auto focus detection varies. As a result, an imaging device capable of autofocus photography by setting an appropriate focus detection region can be realized.

[0069] <Variation 1> When a user moves imaging device 100 in the yaw direction to follow the subject while viewing the subject displayed in live view on display unit 150, position and orientation change acquisition unit 162 acquires information on the movement in the yaw direction (i.e., panning) of imaging device 100. Then, position and orientation change acquisition unit 162 acquires movement information (panning information) of imaging device 100 in the yaw direction and notifies CPU 151. In response to this, CPU 151 can also detect the movement direction of the subject by detecting the panning information of imaging device 100 instead of motion vector calculation unit 203.

[0070] <Variation 2> Also, if a local region is not detected, a learning model such as the one described below can be used. A "trained model" is constructed by learning a large number of sets of training data in which the feature region and movement direction of the subject are input to a learning model composed of a neural network (NN) or the like, and the output label is a "local region" for which a focus detection region can be set. This allows sufficient weighting processing between synapses.

[0071] Also, a new "setting unit (second setting unit)" is provided that sets a focus detection area in a "local area" that is inferred and output by inputting the feature area detected by the subject detection unit 201 and the motion direction of the subject calculated by the motion vector calculation unit 203 into a "learned model." With this, even if a local area is not detected, a local area in which a focus detection area can be set can be obtained if the feature area and motion direction of the subject are detected.

[0072] As a learning model, if a sufficiently large number of various types of teacher data can be learned, it is sufficient to use a NN having a basic input layer, intermediate layer, and output layer, and a sigmoid function is exemplified as an activation function. However, the learning model is not limited to this. For example, a convolutional NN (CNN) with teacher data, which has a "hidden layer" between the input layer and the output layer and has multiple layers in which the hidden layer is convoluted and pooling layers alternately, may be used to improve accuracy. In this case, a normalized linear function such as a "ReLU" function is adopted as the activation function, and the number of layers of the convolutional layer and the pooling layer can be set appropriately. The number of layers can be set so that the robustness of the local region output is suitable for implementation in the imaging device. Even in this case, it is preferable to learn a sufficiently large number of various types of teacher data. Regardless of which learning model is adopted, it is sufficient to make it so that one local region with the highest evaluation value (score) is output from the output layer.

[0073] <Application Examples> Furthermore, the main part of the embodiment of the present invention can be incorporated into various electronic devices having an imaging function. Examples include, but are not limited to, 3G mobile phones, smartphones, tablet terminals, notebook PCs, digital cameras, video cameras, etc. Any electronic device having an imaging function can be incorporated.

[0074] <Additional Note> The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an acquisition unit that captures an image of a subject and acquires image data; a feature region detection unit that detects a feature region included in the image data acquired by the acquisition unit; a local area detection unit that detects a local area for a local site of the subject included in the image data acquired by the acquisition unit; a motion direction detection unit for detecting a motion direction of the subject; a setting unit that sets a focus detection area that is a detection area for automatic focusing on the subject, The setting unit is an imaging device characterized in that, when a local region is not detected by the local region detection unit, the focus detection region is set in the direction of movement of the subject detected by the movement direction detection unit with respect to the feature region detected by the feature region detection unit. (Configuration 2) The setting unit is The imaging device described in configuration 1, characterized in that when a local region is not detected by the local region detection unit, the focus detection region is set in an area within the feature region detected by the feature region detection unit, in the movement direction of the subject detected by the movement direction detection unit, where the local region of the subject is expected to be located. (Configuration 3) The setting unit further The imaging device described in configuration 1 or 2, characterized in that when the local area detection unit does not detect a local area and when the motion direction detection unit cannot detect the movement direction of the subject, the focus detection area is set to the entire feature area detected by the feature area detection unit. (Configuration 4) The present invention further includes a trained model that trains a plurality of sets of training data in which a feature area and a movement direction of a subject are input and a local area in which the focus detection area can be set is used as an output label, The imaging device described in configuration 1 or 2, further comprising, instead of the setting unit, a second setting unit that sets the focus detection area to a local area that is inferred and output by inputting the feature area detected by the feature area detection unit and the motion direction of the subject detected by the motion direction detection unit into the trained model. (Configuration 5) The motion direction detection unit 3. The imaging device according to configuration 1 or 2, wherein the direction of movement of the subject is calculated from a difference between feature amount regions detected in image data supplied sequentially. (Configuration 6) The setting unit further 3. The imaging device according to configuration 1 or 2, wherein the focus detection area is not set when the motion direction of the subject is not detected by the motion direction detection section. (Configuration 7) 3. The imaging device according to configuration 1 or 2, wherein the imaging device detects the direction of movement of a subject by detecting panning of the imaging device itself. (Configuration 8) The setting unit further 2. The imaging device according to configuration 1, wherein, when a local area is detected by the local area detection section, the focus detection area is set to the detected local area. (Configuration 9) The imaging device according to configuration 1, 2 or 8, wherein the local region is a rectangular region in image data for a local part of a subject, including a head, an eye, and a cockpit. (method) A control method for an imaging device, comprising: an acquisition step of capturing an image of a subject and acquiring image data; a feature region detection step of detecting a feature region included in the image data acquired by the acquisition step; a local area detection step of detecting a local area for a local site of the subject included in the image data acquired by the acquisition step; a motion direction detection step of detecting a motion direction of the subject; A setting step of setting a focus detection area which is a detection area for autofocus, The setting step includes: a step of setting the focus detection area in the direction of movement of the subject detected in the motion direction detection step with respect to the feature area detected in the feature area detection step when a local area is not detected in the local area detection step. (program) A program for causing a computer to execute a control method for an imaging device, The control method includes: an acquisition step of capturing an image of a subject and acquiring image data; a feature region detection step of detecting a feature region included in the image data acquired by the acquisition step; a local area detection step of detecting a local area for a local site of the subject included in the image data acquired by the acquisition step; a motion direction detection step of detecting a motion direction of a subject; A setting step of setting a focus detection area which is a detection area for autofocus, The setting step includes: a step of setting the focus detection area in the direction of movement of the subject detected in the movement direction detection step for the feature area detected in the feature area detection step when a local area is not detected in the local area detection step.

[0075] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above-mentioned embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. For example, the present invention can be realized by supplying a program that realizes one or more functions of the above-mentioned embodiment to a system or device via a network or a recording medium, and having a computer processor of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0076] 100 Imaging device 101 Imaging lens 102 Fixed 1 group lens 103 Aperture 104 Aperture motor 105 Aperture control section 111 Zoom Lens 112 Zoom motor 113 Zoom control section 121 Fixed 3-group lens 131 Focus Lens 132 Focus Motor 133 Focus control section 141 Image sensor 142 Imaging signal processing section 143 Imaging control section 150 Display section 151 CPU 152 Image processing section 153 Image Compression Unit 154 RAM 155 Flash Memory 156 Operation section 157 Image recording media 158 Power management section 159 Battery 160 Bus 161 Subject tracking section 162 Position and Posture Change Acquisition Unit

Claims

1. an acquisition unit that captures an image of a subject and acquires image data; a feature region detection unit that detects a feature region included in the image data acquired by the acquisition unit; a local area detection unit that detects a local area for a local site of the subject included in the image data acquired by the acquisition unit; a motion direction detection unit for detecting a motion direction of the subject; A setting unit that sets a focus detection area that is a detection area for automatic focusing, The setting unit is an imaging device characterized in that, when a local region is not detected by the local region detection unit, the focus detection region is set in the direction of movement of the subject detected by the movement direction detection unit with respect to the feature region detected by the feature region detection unit.

2. The setting unit is 2. The imaging device according to claim 1, characterized in that, when a local region is not detected by the local region detection unit, the focus detection region is set in an area within the feature region detected by the feature region detection unit, in the direction of movement of the subject detected by the movement direction detection unit, in which a local region of the subject is assumed to be present.

3. The setting unit further 3. The imaging device according to claim 1, wherein when a local region is not detected by the local region detection unit and when a movement direction of a subject is not detected by the movement direction detection unit, the focus detection area is set to the entire feature region detected by the feature region detection unit.

4. The present invention further includes a trained model that trains a plurality of sets of training data in which a feature area and a movement direction of a subject are input and a local area in which the focus detection area can be set is used as an output label, The imaging device according to claim 1 or 2, further comprising, instead of the setting unit, a second setting unit that sets the focus detection area to a local area that is inferred and output by inputting the feature area detected by the feature area detection unit and the movement direction of the subject detected by the movement direction detection unit into the trained model.

5. The motion direction detection unit 3. The imaging apparatus according to claim 1, wherein the direction of movement of the subject is calculated from a difference between feature amount areas detected in the image data supplied sequentially.

6. The setting unit further 3. The image pickup apparatus according to claim 1, wherein the focus detection area is not set when the motion direction of the subject is not detected by the motion direction detection section.

7. 3. The imaging apparatus according to claim 1, wherein the moving direction of the subject is detected by detecting panning of the imaging apparatus itself.

8. The setting unit further 2. The image pickup apparatus according to claim 1, wherein, when a local area is detected by the local area detection section, the focus detection area is set to the detected local area.

9. 9. The imaging device according to claim 1, wherein the local region is a rectangular region in image data for a local part of a subject, including a head, an eye, and a cockpit.

10. A control method for an imaging device, comprising: an acquisition step of capturing an image of a subject and acquiring image data; a feature region detection step of detecting a feature region included in the image data acquired by the acquisition step; a local area detection step of detecting a local area for a local site of the subject included in the image data acquired by the acquisition step; a motion direction detection step of detecting a motion direction of a subject; A setting step of setting a focus detection area which is a target area for autofocus, The setting step includes: a step of setting the focus detection area in the direction of movement of the subject detected in the motion direction detection step with respect to the feature area detected in the feature area detection step when a local area is not detected in the local area detection step.

11. A program for causing a computer to execute a control method for an imaging device, The control method includes: an acquisition step of capturing an image of a subject and acquiring image data; a feature region detection step of detecting a feature region included in the image data acquired by the acquisition step; a local area detection step of detecting a local area for a local site of the subject included in the image data acquired by the acquisition step; a motion direction detection step of detecting a motion direction of a subject; A setting step of setting a focus detection area which is a target area for autofocus, The setting step includes: a step of setting the focus detection area in the direction of movement of the subject detected in the movement direction detection step for the feature area detected in the feature area detection step when a local area is not detected in the local area detection step.

Images