Imaging device, imaging control method, and imaging control program

Abstract

This invention provides a camera device, camera control method, and camera control program that can reduce the burden of manual focusing operations. The camera device includes a camera optical system, a camera element, and a processor. In a first focusing mode, the processor performs the following processing: outputting dynamic image data captured by the camera element to a display terminal via the camera optical system; detecting the focus area in the image represented by the dynamic image data; detecting the end of the focusing operation based on the time related to the focusing operation when a focusing operation has been performed; setting the subject existing in the focus area as the tracking object when the end of the focusing operation is detected; continuing tracking processing and automatic focus control; and ending the tracking processing and automatic focus control when a focusing operation is performed again after the start of tracking processing and automatic focus control.

Description

Technical Field

[0001] This invention relates to a camera device, a camera control method, and a camera control program, and more particularly to a technique for assisting manual focusing operations. Background Technology

[0002] Patent document 1 describes a technique for displaying the focus state of the main subject on a display mechanism when manually focusing.

[0003] Patent Document 2 describes a technique in which, in a camera device capable of both autofocus (AF) and manual focus (MF) based imaging, if MF operation is performed during AF imaging, the AF is stopped. Furthermore, Patent Document 2 describes a technique that restarts AF when it is stopped, triggered by a zoom operation or similar mechanism.

[0004] Patent Document 3 describes a technique that detects contact between a finger and a switch for command focusing, automatically switching between AF (Auto Focus) and MF (Multi-Focus) operations. Furthermore, Patent Document 3 describes a technique that, when switching from MF to AF operation, uses the focus area specified during MF operation as the focus area for AF processing.

[0005] Previous technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-197806

[0008] Patent Document 2: Japanese Patent Application Publication No. 2004-64713

[0009] Patent Document 3: Japanese Patent Application Publication No. 6-205261 Summary of the Invention

[0010] One embodiment of the technology disclosed herein provides a camera device, camera control method, and camera control program that can reduce the burden of manual focusing operations.

[0011] means for solving technical problems

[0012] (1) A camera device comprising a camera optical system, a camera element and a processor, wherein the processor performs the following processing in a first focus mode: outputting dynamic image data captured by the camera element to a display terminal via the camera optical system; detecting a focus area in the image represented by the dynamic image data based on the dynamic image data; detecting the end of the focus operation based on a time related to the focus operation when a focus operation has been performed; setting a subject existing in the focus area as a tracking object when the end of the focus operation has been detected; continuing to perform tracking processing and automatic focus control; and ending the tracking processing and automatic focus control when a focus operation is performed again after the start of tracking processing and automatic focus control.

[0013] (2) The camera device according to (1), wherein,

[0014] The time associated with the focusing operation is the time spent in the inactive state.

[0015] (3) The camera device according to (2), wherein,

[0016] The processor performs the following processing:

[0017] Measure the duration of the inactive state.

[0018] The detection process begins after the first time interval from the start of the no-operation state and ends after the focusing operation.

[0019] (4) The camera device according to (3), wherein,

[0020] The processor performs the following processing:

[0021] Record information about the duration of the inactive state.

[0022] The first time is set based on the recorded duration of the inactive state.

[0023] (5) The camera device according to (4), wherein,

[0024] The processor calculates and sets the first time using statistical methods based on the recorded duration of the inactive state.

[0025] (6) The camera device according to (5), wherein,

[0026] The processor calculates the average, median, or most frequent duration of the most recent specified number of inactive states as the first time.

[0027] (7) The camera device according to any one of (1) to (6), wherein,

[0028] The processor assigns information representing the focus area to the image represented by the dynamic image data and outputs the dynamic image data to the display terminal.

[0029] (8) The camera device according to (7), wherein,

[0030] The processor changes the content of the information representing the focus area based on the detection of the end of the focusing operation.

[0031] (9) The camera device according to any one of (1) to (8), wherein,

[0032] The processor performs the following processing: detects the direction of focus movement just before the focus operation is about to end, and restricts the direction of focus movement based on autofocus control to the detected direction of movement during the period from the start of tracking processing and autofocus control until the second time has elapsed.

[0033] (10) The camera device according to any one of (1) to (9), wherein,

[0034] The processor performs the following processing: acquires the posture information of the main body of the device, and after starting the tracking process and autofocus control, if the posture of the main body of the device changes, the tracking process and autofocus control are terminated.

[0035] (11) The camera device according to any one of (1) to (10), wherein,

[0036] When multiple subjects are set as tracking objects, the processor performs the following process: calculates the movement of each subject, and sets the tracking object based on the calculated movement.

[0037] (12) The camera device according to (11), wherein,

[0038] The processor sets the subject that is not moving or has the least amount of movement as the tracking object.

[0039] (13) The camera device according to (11) or (12), wherein,

[0040] The amount of movement includes the amount of movement in the optical axis direction of the camera optical system. When it is determined that all subjects are moving, the processor sets the subject with the smallest amount of movement in the optical axis direction as the tracking object.

[0041] (14) The camera device according to (13), wherein,

[0042] When the processor determines that all subjects are moving, it will release the tracking settings for all subjects if it cannot obtain information on the amount of movement in the optical axis direction.

[0043] (15) The camera device according to any one of (1) to (10), wherein,

[0044] The processor performs the following processing: based on the dynamic image data, it detects the focus state of the subject contained in the dynamic image data, and sets the tracking object based on the focus state of the subject in the focusing operation.

[0045] (16) The camera device according to (15), wherein,

[0046] The processor will set the subject that changes from focus to defocus and then returns to focus during the focusing operation as the tracking object.

[0047] (17) The camera device according to (16), wherein,

[0048] When there is no subject that changes from a focused state to a non-focused state and then returns to a focused state during the focusing operation, the processor sets the subject with the longest period of focus during the focusing operation as the tracking object.

[0049] (18) The camera device according to (15), wherein,

[0050] The processor will set the subject that is in focus for the longest period during the focusing operation as the tracking object.

[0051] (19) The camera device according to any one of (1) to (18), wherein,

[0052] The processor performs the following processing: after starting tracking processing and autofocus control, it calculates the focus evaluation value of multiple regions in the frame based on dynamic image data, extracts the region with the same level of focus evaluation value as the tracked object and maintains it for the third time, and adds and sets the subject existing in the extracted region as a new tracked object.

[0053] (20) A camera control method, which performs the following processing: detecting a focus area in an image represented by dynamic image data captured by a camera element via a camera optical system; when a focusing operation is performed, detecting the end of the focusing operation based on the time related to the focusing operation; when the end of the focusing operation is detected, setting a subject existing in the focus area as a tracking object, continuing tracking processing and automatic focus control; and when a focusing operation is performed again after the start of tracking processing and automatic focus control, ending the tracking processing and automatic focus control.

[0054] (21) A camera control program that enables a computer to perform the following functions: detecting a focus area in an image represented by dynamic image data captured by a camera element via a camera optical system; detecting the end of a focus operation based on a time related to the focus operation when a focus operation has been performed; setting a subject existing in the focus area as a tracking object and continuing tracking processing and autofocus control when the end of the focus operation is detected; and ending the tracking processing and autofocus control when a focus operation is performed again after the start of the tracking processing and autofocus control. Attached Figure Description

[0055] Figure 1 It is a diagram showing the general structure of a camera device.

[0056] Figure 2 It is a diagram showing the general structure of the lens section.

[0057] Figure 3 This is a block diagram of the functions implemented by the CPU in MF mode.

[0058] Figure 4 This is a block diagram of the functions implemented by the CPU in MF mode.

[0059] Figure 5 It is a diagram representing one frame of the moving image displayed on the display unit during the focusing operation.

[0060] Figure 6 It is a diagram representing one frame of the dynamic image displayed on the display during AF tracking.

[0061] Figure 7 This is an example diagram showing the setting of the focus area.

[0062] Figure 8 This is a flowchart showing the processing sequence of camera control in MF mode.

[0063] Figure 9 This is an example of a display showing the live view in MF mode.

[0064] Figure 10 This is a block diagram illustrating an example of the functionality implemented by the CPU when using statistical methods to set the detection time.

[0065] Figure 11 This is a block diagram illustrating an example of the functions implemented by the CPU when using image recognition to set a tracking object.

[0066] Figure 12 This is another example of a display method that shows information about the focus area.

[0067] Figure 13 This is a block diagram of the main functions implemented by the CPU when limiting the direction of movement for focusing and performing autofocus (AF).

[0068] Figure 14 This is a flowchart showing the processing sequence of camera control.

[0069] Figure 15 This is a block diagram of the main functions implemented by the CPU when the camera device's posture changes and the tracking autofocus is forcibly terminated.

[0070] Figure 16 This is a flowchart showing the processing sequence of camera control.

[0071] Figure 17 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0072] Figure 18 This is a flowchart showing the processing sequence of camera control.

[0073] Figure 19 This is a conceptual diagram of the settings for the tracking object.

[0074] Figure 20 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0075] Figure 21 This is a flowchart showing the processing sequence of camera control.

[0076] Figure 22 This is a conceptual diagram of the settings for the tracking object.

[0077] Figure 23 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0078] Figure 24 This is a flowchart showing the processing sequence of camera control.

[0079] Figure 25 This is a flowchart showing the processing sequence of camera control.

[0080] Figure 26 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0081] Figure 27 This is a flowchart showing the processing sequence of camera control.

[0082] Figure 28 This is a block diagram of the main functions implemented by the CPU when adding tracking objects for AF (Automatic Field Tracking).

[0083] Figure 29 This is a flowchart showing the processing sequence of camera control. Detailed Implementation

[0084] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0085] [First Implementation]

[0086] Focus cannot be changed through post-editing. Therefore, focus adjustment is particularly important in video recording. However, in low-budget video recording, it is common for the photographer to work alone (so-called single-person operation). In this case, the photographer needs to make various adjustments such as focus, angle of view, exposure, and white balance, which is very demanding. Although autofocus (AF) can be used for focus adjustment, the focus area needs to be set appropriately to achieve accurate focus. However, in single-person operation, other operations also need to be performed in parallel, resulting in momentary latency. The following describes a video recording device that can reduce the workload of MF operations performed by the user.

[0087] [Device Structure]

[0088] Figure 1 This is a diagram showing the schematic structure of the camera device according to this embodiment.

[0089] like Figure 1 As shown, the camera device 1 of this embodiment includes a lens section 10 and a main body section 100. The lens section 10 may be integrated with the main body section 100, or it may be a detachable (replaceable) structure.

[0090] Figure 2 It is a diagram showing the general structure of the lens section.

[0091] like Figure 2 As shown, the lens unit 10 includes a camera optical system 20, a lens drive unit 30, and a lens operation unit 40.

[0092] In this embodiment, the camera optical system 20 is composed of a zoom optical system, including a zoom lens 22, an aperture 24, and a focusing lens 26. Furthermore, as a mechanism for adjusting the amount of light, instead of the aperture 24, or in addition to the aperture 24, a neutral density filter (ND filter) or a type of neutral density filter can be included. The zoom lens 22 and the focusing lens 26 are each composed of at least one lens.

[0093] The lens driving unit 30 includes a zoom driving unit 32, an aperture driving unit 34, and a focus driving unit 36.

[0094] The zoom drive unit 32 is the drive unit for the zoom lens 22. The zoom drive unit 32 includes an actuator (not shown) for driving the zoom lens 22, a drive circuit (not shown) for the actuator, and a sensor (not shown) for detecting the position of the zoom lens 22. The actuator may be, for example, a DC motor, a linear motor, a stepper motor, or an ultrasonic motor. Driven by the zoom drive unit 32, the zoom lens 22 moves back and forth along the optical axis L. As a result, the focal distance (angle of view) changes.

[0095] The aperture drive unit 34 is the drive unit for the aperture 24. The aperture drive unit 34 includes an actuator (not shown) that drives the aperture 24 and a drive circuit (not shown) for the actuator. The actuator may be, for example, a DC motor, a linear motor, a stepper motor, or an ultrasonic motor. The aperture 24 is driven by the aperture drive unit 34, causing its opening diameter to change. This, in turn, changes the amount of light passing through the imaging optical system 20.

[0096] The focusing drive unit 36 ​​is the drive unit for the focusing lens 26. The focusing drive unit 36 ​​includes an actuator (not shown) for driving the focusing lens 26, a drive circuit (not shown) for the actuator, and a sensor (not shown) for detecting the position of the focusing lens 26. The actuator may be, for example, a DC motor, a linear motor, a stepper motor, or an ultrasonic motor. The focusing lens 26 is driven by the focusing drive unit 36 ​​to move back and forth along the optical axis L. As a result, the focusing position changes.

[0097] The lens operation unit 40 includes a zoom operation unit 42, an aperture operation unit 44, and a focus operation unit 46.

[0098] The zoom operation unit 42 includes a zoom ring (not shown) as an operation component and a sensor (not shown) for detecting the position of the zoom ring. The zoom ring is rotatably disposed on the outer periphery of the lens barrel. Information about the position of the zoom ring detected by the sensor is output to the CPU 120. The CPU 120 controls the drive of the zoom lens 22 based on the acquired position information of the zoom ring. Furthermore, in this embodiment, a structure is provided for zooming using the zoom ring, but it is also possible to provide a structure for zooming via lever operation, knob operation, or the like.

[0099] The aperture operation unit 44 includes an aperture ring (not shown) as an operation component and a sensor (not shown) for detecting the position of the aperture ring. The aperture ring is rotatably disposed on the outer periphery of the lens barrel. The position information of the aperture ring detected by the sensor is output to the CPU 120. The CPU 120 controls the driving of the aperture 24 based on the acquired position information of the aperture ring.

[0100] The focusing operation unit 46 is the focusing operation unit. The focusing operation unit 46 has a focusing ring (not shown) as an operating component and a sensor (not shown) for detecting the position of the focusing ring. The focusing ring is rotatably disposed on the outer periphery of the lens barrel. The position information of the focusing ring detected by the sensor is output to the CPU 120. In MF mode, the CPU 120 controls the driving of the focusing lens 26 based on the acquired position information of the focusing ring. MF mode is a manual focusing adjustment operation mode. There are MF mode and AF mode in the focusing adjustment operation modes (focusing modes), which are selected by the user. AF mode is an automatic focusing adjustment operation mode. The selection of the focusing mode is made by the main operating unit 118. It can also be configured to allow focusing operation via lever operation, knob operation, etc.

[0101] like Figure 1 As shown, the main body 100 includes a camera unit 110, a display unit 112, a storage unit 114, a connection unit 116, a main body operation unit 118, a CPU (Central Processing Unit) 120, a ROM (Read Only Memory) 122, and a RAM (Random Access Memory) 124.

[0102] The camera unit 110 includes an imaging element 110A that converts optical images into electrical signals (reference). Figure 2 The camera element 110A may be a CMOS image sensor (CMOS: Complementary Metal Oxide Semiconductor) or a CCD image sensor (CCD: Charge Coupled Device) with a specified color filter arrangement (e.g., Bayer arrangement).

[0103] In the imaging device 1 of this embodiment, a CMOS image sensor equipped with a driving unit, an ADC (Analog-to-Digital Converter), and a signal processing unit is used as the imaging element 110A. The imaging element 110A is driven and operated by the built-in driving unit. Furthermore, the signal of each pixel is converted into a digital signal by the built-in ADC and output. In addition, the signal of each pixel undergoes correlation double sampling, gain processing, correction processing, and other signal processing by the built-in signal processing unit before being output. The signal processing can be performed on the signal before or after conversion to a digital signal.

[0104] Furthermore, in the imaging device 1 of this embodiment, an imaging element 110A with embedded phase difference detection pixels is used as the imaging element 110A. By using the imaging element 110A with embedded phase difference detection pixels, information about the phase difference of the subject in the frame can be acquired. Furthermore, the focus shift direction and focus shift amount (defocus) of the subject in the frame can be detected from the acquired phase difference information. Since the imaging element 110A with embedded phase difference detection pixels and the phase difference detection method using this imaging element 110A are known technologies, their detailed description is omitted.

[0105] The capture of moving images is performed at a predetermined frame rate. The frame rate can be fixed or can be arbitrarily set by the user.

[0106] The display unit 112 is composed of a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode Display). The display unit 112 can also be composed of a touch panel display with a touch panel on its display surface. Furthermore, the display unit 112 may also be in the form of an EVF (Electronic View Finder). In addition to displaying live view, the display unit 112 also serves as a GUI (Graphical User Interface) for making various settings. Furthermore, when the display unit 112 is composed of a touch panel display, it also functions as an operating mechanism. Additionally, when an external display device is connected to the connection unit 116 (described later), that external display device also functions as the display unit 112.

[0107] In addition to serving as a storage area for captured video image data, the storage unit 114 also serves as a storage area for various other types of data. The storage unit 114 may be constructed from, for example, a non-volatile semiconductor memory such as EEPROM (Electrically Erasable Programmable Read-Only Memory), or an SSD (Solid State Drive) that incorporates such semiconductor memory. The storage unit 114 may be an integral part of the device body (in the form of built-in memory) or a removable structure relative to the device body (in the form of a memory card). Furthermore, it may be connected via wired or wireless communication. Moreover, when an external storage device is connected to the connection unit 116 (described later), that external storage device also functions as the storage unit 114.

[0108] The connector 116 is used for connecting external devices (e.g., external display devices, external storage devices, etc.). The connector 116 has at least one terminal. As a connection standard, for example, HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark) can be used.

[0109] The main operating unit 118 includes various operating components for operating the camera device 1. These operating components include various operation buttons such as a power button for turning the power on and off, and a recording button for starting and ending recording. Furthermore, if the display unit 112 is a touch panel display, the touch panel is also included in the main operating unit 118.

[0110] As described above, the focus mode selection is performed by the main operation unit 118. The main operation unit 118 includes a focus mode switching switch as the mechanism for selecting the focus mode. The focus mode switching switch selectively switches between MF mode and AF mode. Alternatively, the focus mode selection (setting) can also be performed using a setting screen or similar method.

[0111] CPU120 is an example of a processor. CPU120 functions as the control unit and image processing unit of the camera device 1 by executing a prescribed program (camera control program).

[0112] The control performed by the CPU 120 includes not only camera control but also display control and recording control. Camera control includes not only drive control of the lens unit 10 and the camera unit 110 but also checking, detecting, and calculating information required for control. This checking, detection, and calculation includes, for example, detecting or calculating brightness required for exposure control and detecting or calculating focus evaluation values ​​required for AF control. Furthermore, tracking processing is also included when performing tracking AF. Tracking AF is one of the functions of AF, which is the function of automatically tracking and continuously focusing even when the subject moves. In tracking AF, as a tracking process, the focus area is set according to the movement of the subject being tracked. In the camera device 1 of this embodiment, when the end of the focusing operation is detected in MF mode, the subject focused by MF is set as the tracking object, and tracking AF automatically begins. Tracking AF continues until the focusing operation is performed again. This process will be explained in more detail later. MF mode is an example of the first focusing mode.

[0113] Furthermore, the image processing performed by the CPU 120 includes not only the generation of motion picture data for recording and display (live view), but also compression processing of the recorded motion picture data. The recorded and displayed motion picture data is generated by performing prescribed image processing (so-called development processing) on ​​RAW data (unprocessed motion picture data output from the camera unit 110). This image processing includes offset processing, gamma correction processing, de-mosaic processing, RGB / YCrCb conversion processing, white balance processing, etc. Since these processes are well-known, detailed descriptions are omitted. Compression processing generates motion picture files in a prescribed compressed format. The codec used for compression can be a well-known codec. For example, codecs standardized by the MPEG (Moving Picture Experts Group) (MPEG-1, MPEG-2, MPEG-4, etc.) can be used.

[0114] Furthermore, the aforementioned processes performed by the CPU 120 can also be configured to be performed, in part or in whole, by hardware located inside the CPU 120.

[0115] ROM122 stores various data required for programs and control executed by CPU120. Furthermore, the memory constituting ROM122 includes an EEPROM containing flash memory.

[0116] RAM124 is used as a working area for CPU120 to perform various processing tasks.

[0117] [Camera control functions in MF mode]

[0118] As described above, in the camera device 1 of this embodiment, in MF mode, when the user (photographer) focuses on the desired subject and ends the focusing operation, the focused subject is set as the tracking object, and tracking AF automatically starts. Then, when the user focuses again, tracking AF ends, and the operation switches to normal MF mode.

[0119] Figure 3 and Figure 4 This is a block diagram of the functions implemented by the CPU in MF mode. Figure 3 This is a block diagram illustrating the functions primarily implemented during manual focusing. In other words, it's a block diagram illustrating the functions implemented in normal MF (Master Focus) mode. Furthermore, Figure 4 This is a block diagram showing the main functions implemented during AF tracking.

[0120] like Figure 3 and Figure 4 As shown, in MF mode, the CPU 120 functions as an image processing unit 120A for display, a display control unit 120B, a focus evaluation value calculation unit 120C, a focus area detection unit 120D, a tracking object setting unit 120E, a focus operation detection unit 120F, a focus drive control unit 120G, a tracking processing unit 120H, and an AF control unit 120I.

[0121] The display image processing unit 120A processes the dynamic image data (RAW data) output from the camera unit 110 and generates dynamic image data for display.

[0122] Figure 5 It is a diagram representing one frame of the moving image displayed on the display unit during the focusing operation. Figure 5 This is an example of an image displayed on display unit 112 when a person 300 is photographed with the camera focused on. The tree 400 in the background is photographed with the image blurred.

[0123] like Figure 5 As shown, the image IM1 displayed during the focusing operation is assigned information representing the focus area. In this embodiment, as information representing the focus area, a frame F1 of the focus area in the focusing state is displayed. More specifically, the frame F1 of the focus area in the focusing state is superimposed on the live view image. The frame F1 is displayed in a predetermined color. In this embodiment, the frame F1 of the focus area in the focusing state is displayed in red. The focus area will be described later.

[0124] During the focusing operation, the display image processing unit 120A generates dynamic image data for display based on information about the focus area detected by the focus area detection unit 120D. The detection of the focus area will be described later.

[0125] Figure 6 It is a diagram representing one frame of the dynamic image displayed on the display during AF tracking. Figure 6 This is an example of an image displayed on display unit 112 when a person 300 directly in front of you is being tracked.

[0126] In tracking AF, the subject designated as the tracking target is continuously focused. In the image IM2 displayed during tracking AF, a frame F2 representing the focus area of ​​the subject is displayed in a predetermined color. This color is different from the color (red) of the frame F1 representing the focus area during focusing operations. In this embodiment, it is displayed in green. Thus, during tracking AF, the frame representing the tracking target is displayed in a different color than during focusing operations. Therefore, it is possible to identify the tracking target from the image displayed on the display unit 112 while simultaneously recognizing that it is during tracking AF. Furthermore, since the tracking target is continuously focused during tracking AF, the focus area where the subject is located essentially becomes the focus area. Therefore, the frame F2 representing the focus area of ​​the subject is also information indicating the focus area.

[0127] The display image processing unit 120A generates display dynamic image data based on information about the focus area set by the tracking processing unit 120H during AF tracking. The focus area will be described later.

[0128] The display control unit 120B causes the display unit 112 to display dynamic image data generated by the display image processing unit 120A. The display unit 112 is an example of a display terminal. Additionally, when an external display device is connected to the connection unit 116, the dynamic image data is also displayed on the external display device. In this case, the display of the display unit 112 can also be turned off. Furthermore, it can also be configured to display other information.

[0129] The focus evaluation value calculation unit 120C calculates the focus evaluation value of the preset focus area based on the dynamic image data captured by the camera unit 110.

[0130] Figure 7 This is an example diagram showing the setting of the focus area.

[0131] like Figure 7 As shown, in the camera device of this embodiment, the focus area FA is set by dividing the screen equally in both the vertical and horizontal directions. Figure 7 In the example shown, the image is divided into 9 equal parts vertically (y direction) and 13 equal parts horizontally (x direction), thus setting 9×13 focus areas FA.

[0132] The focus evaluation value calculation unit 120C calculates the focus evaluation value for each focus area FA. The focus evaluation value is an indicator representing the focus state. In this embodiment, phase difference information or the amount of defocusing obtained from the phase difference information is calculated as the focus evaluation value. The phase difference information is calculated based on the information of the phase difference detection pixels of each focus area FA. This is a well-known technique, so a detailed description thereof is omitted.

[0133] The focus area detection unit 120D detects the focus area based on the focus evaluation value of each focus area FA calculated by the focus evaluation value calculation unit 120C. That is, it detects the area where the focus point is located. In the imaging device 1 of this embodiment, the focus area of ​​the focus point is selected as the focus area. The focus area of ​​the focus point is a focus area where the phase difference or defocus amount is below the allowable value.

[0134] The tracking object setting unit 120E sets a subject as a tracking object. The tracking object setting unit 120E sets the tracking object based on the detection of the end of the focus operation. That is, the tracking object is set when the end of the focus operation is detected by the focus operation detection unit 120F. The tracking object setting unit 120E sets a subject that exists in the focus area when the end of the focus operation is detected as the tracking object. For example, in Figure 5 When the focus operation is detected to have ended in the state shown, character 300 is set as the tracking object.

[0135] The 120F focusing operation detection unit detects the start and end of focusing operations.

[0136] The start of focusing operation is detected by a change in the position of the focusing ring. That is, the start of operation is detected by detecting a change in the position of the focusing ring. As described above, the position of the focusing ring is detected by a sensor provided in the focusing operation unit 46. Therefore, the focusing operation detection unit 120F detects the start of focusing operation based on the sensor's output.

[0137] On the other hand, the end of the focusing operation is detected based on the time associated with the focusing operation. Specifically, the end of the focusing operation is detected based on the time of the no-operation state. The no-operation state refers to a state in which no focusing operation is performed. In the imaging device 1 of this embodiment, focusing is performed by the focusing ring. Therefore, the state in which no focusing operation is performed refers to a state in which no focusing ring is operated. Here, the state in which no focusing ring is operated is a state in which there is no change in the position of the focusing ring.

[0138] The focusing operation detection unit 120F measures the duration of the inactive state (a state in which there is no change in the position of the focusing ring) based on the output of a sensor that detects the position of the focusing ring. Then, it detects if a predetermined time has elapsed since the start of the inactive state and detects the end of the focusing operation. The predetermined time is an example of the first time interval. For example, the predetermined time is set to 500 ms. At this time, the focusing operation detection unit 120F detects that 500 ms has elapsed since the start of the inactive state and detects the end of the focusing operation.

[0139] The focusing drive control unit 120G controls the driving of the focusing lens 26 via the focusing drive unit 36.

[0140] During focusing operation, the focusing drive control unit 120G controls the driving of the focusing lens 26 based on the operation information (position information of the focusing ring) from the focusing operation unit 46. On the other hand, during tracking AF, the driving of the focusing lens 26 is controlled based on control information from the AF control unit 120I.

[0141] The tracking processing unit 120H performs tracking processing. Tracking processing includes detecting the tracked object and setting the focus area based on the detection results.

[0142] The detection of the tracked object is performed based on the motion image data output from the camera unit 110. Specifically, the tracked object is detected sequentially from each frame of the motion image data. Known methods such as template matching are used for detection.

[0143] The focus area is the area where the focus is aligned. The focus area is set from the focus area. The tracking processing unit 120H sets the focus area where the tracked object is located as the focus area. The tracked object is detected sequentially for each frame, therefore the focus area is also set sequentially for each frame.

[0144] The tracking processing unit 120H continues tracking processing until the focusing operation resumes or until the tracked object disappears. Therefore, even if a tracked object is present, tracking processing can be forcibly terminated when the focusing operation resumes.

[0145] The AF control unit 120I performs AF control (autofocus control) on the focus area set by the tracking processing unit 120H. That is, it controls the focus of the subject within the set focus area. Specifically, first, it calculates the control amount (drive amount of the focus drive unit 36) required to focus the subject in the focus area based on focus evaluation value information within the focus area. Then, it provides the calculated control amount information to the focus drive control unit 120G. The focus drive control unit 120G drives the focus drive unit 36 ​​according to the provided control amount information.

[0146] Similar to the tracking processing unit 120H, the AF control unit 120I continues to perform AF control until the focusing operation is restarted, or until the tracked object disappears.

[0147] [Processing sequence of camera control in MF mode (camera control method)]

[0148] Figure 8 This is a flowchart showing the processing sequence of camera control in MF mode.

[0149] First, it is determined whether the focusing operation has started (step S1). The presence or absence of the focusing operation is determined based on the output from the focusing operation unit 46. That is, it is determined based on the output from the sensor that detects the position of the focusing ring. When there is an output from the sensor that detects the position of the focusing ring, it is determined that the focusing operation has started.

[0150] When it is determined that the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S2). The focusing mode is determined according to the setting state of the focusing mode switch.

[0151] If it is determined that the mode is set to MF (Master Focus), a focus evaluation value is calculated for each focus area (step S3). Then, the focus area is detected based on the calculation result (step S4). That is, the focus area that is in focus is detected. The detected focus area is displayed on the live view image. In this embodiment, the frame of the focus area determined to be the focus area is displayed in red. Therefore, the user (photographer) can confirm the focus area by visually recognizing the live view.

[0152] When the focus area is detected, a tracking object is set (step S5). The tracking object is set as the subject existing in the focus area. In the camera device of this embodiment, the focus area is set as the area of ​​the tracking object. That is, the focus area is set as the area where the tracking object is located.

[0153] After setting the tracking object, determine whether the focusing operation has ended (step S6). The focusing operation is considered to have ended if the inactivity continues for a specified time. That is, the end of the operation is detected.

[0154] When the focus operation is detected to have ended, tracking AF begins (step S7). That is, tracking processing and AF control are continuously performed on the subject set as the tracking object. More specifically, the focus object area is set according to the movement of the subject set as the tracking object, and AF control is continued in such a way that the focus is on the subject within the set focus object area (so-called continuous AF control).

[0155] Furthermore, when the focusing operation is detected to be complete, the display of the focus area is switched. That is, the color of the frame representing the focus area is changed. In this embodiment, the color of the frame representing the focus area changes from red to green. This allows the user to recognize that the mode has been switched to tracking AF. Simultaneously, the subject set as the tracking target can also be identified.

[0156] The tracking AF continues until the focusing operation resumes or until the tracked object disappears. Therefore, the focusing operation is constantly monitored during the tracking AF. That is, it is determined whether a focusing operation has been performed again (step S8). When it is determined that a focusing operation has resumed, the tracking AF is forcibly terminated. That is, the tracking process and the AF control based on it are terminated. After the tracking AF ends, the operation returns to the normal MF mode, and the processing after step S2 is repeated.

[0157] Figure 9 This is an example of a display showing the live view in MF mode. Figure 9 (A) represents an example of display during a focus operation. Figure 9 (B) represents an example of what is displayed immediately after the focus operation has ended. Figure 9 (C) represents an example displayed in the tracking AF.

[0158] During focusing, the frame F1 of the focusing area, which is in focus, is displayed in red as information indicating the focus area. Figure 9 (A) indicates the case where the focus is on the background tree.

[0159] When the focus operation is detected to be finished, the color of the frame F2 representing the focus area changes from red to green. Figure 9 (B) indicates that the focus operation has ended with the subject directly in front of you in focus. You can confirm switching to tracking AF by changing the color of the frame F2.

[0160] During the tracking of AF, such as Figure 9 As shown in (C), the frame F2 of the focus area where the tracked object is located is displayed in green. This allows the tracked object to be identified within the screen.

[0161] When focusing resumes during tracking AF, tracking AF ends. If tracking AF ends, operation returns to normal MF mode. Therefore, the display on display unit 112 returns to normal. Figure 9 (A) state. That is, the focus area frame F1, which is in focus mode, is displayed in red. The color of the displayed frame F1 changes from green to red, thus confirming that it has returned to normal MF mode.

[0162] Alternatively, the system can be configured to notify the user separately when the tracking AF ends or when the system returns to MF mode. For example, the display unit 112 can display information notifying the user when the tracking AF ends.

[0163] As explained above, the camera device 1 according to this embodiment maintains focus based on manual focus (MF). Therefore, the burden of focusing operations performed by the user can be greatly reduced.

[0164] Furthermore, the start of the tracking AF is determined based on the time associated with the focusing operation, especially the duration of the inactive state. Therefore, the tracking AF can be started according to the user's intention. Moreover, since no special mechanisms for switching are required, the structure can be simplified.

[0165] Furthermore, when switching to tracking AF, the display content of the switching display unit 112 is changed, so the operation status can be clearly understood. As a result, good operability can be achieved.

[0166] [Variation Example]

[0167] [A variation of the detection of the end of a focused operation]

[0168] The end of the focusing operation is detected based on the time associated with the focusing operation. The time associated with the focusing operation is preferably set to the time of the inactive state. That is, the end of the focusing operation is preferably detected based on the duration of the inactive state.

[0169] Furthermore, the duration of the inactive state used to detect the end of the focusing operation (hereinafter referred to as the detection time) can be 0. When the detection time (first time) is 0, AF tracking begins immediately after the focusing ring stops.

[0170] Furthermore, the system can be configured so that the user can arbitrarily set the detection time. This setting can be performed, for example, using the display unit 112 and the main operation unit 118. Therefore, settings can be customized according to the user's preferences.

[0171] Furthermore, the detection time can be automatically set based on the user's past operational tendencies. For example, the duration of multiple periods of inactivity can be recorded, and an optimal detection time can be calculated and set using statistical methods.

[0172] Figure 10 This is a block diagram illustrating an example of the functionality implemented by the CPU when using statistical methods to set the detection time.

[0173] like Figure 10 As shown, the CPU120 also functions as a first timing unit 130A, a measurement result recording and control unit 130B, a detection time calculation unit 130C, and a detection time setting unit 130D.

[0174] The first timing unit 130A measures the duration of inactive conditions.

[0175] The measurement result recording control unit 130B records information about the measurement results based on the duration of the inactive state of the first timing unit 130A. The measurement results are stored in the measurement result storage unit 114A. The measurement result storage unit 114A is located in a storage area of ​​the storage unit 114. The measurement result storage unit 114A stores a predetermined number of measurement results. For example, in this example, three measurement results are stored. The measurement result recording control unit 130B rewrites and records the measurement results in reverse order. Therefore, the measurement result storage unit 114A stores the most recent predetermined number of measurement results (three in this example).

[0176] The detection time calculation unit 130C calculates the detection time using a statistical method based on information about the duration of multiple inactive states stored in the measurement result storage unit 114A. In this example, the average value is calculated to determine the detection time.

[0177] The detection time setting unit 130D sets the time calculated by the detection time calculation unit 130C as the detection time.

[0178] The focusing operation detection unit 120F detects the end of the focusing operation based on the detection time set by the detection time setting unit 130D.

[0179] This also allows for the automatic setting of detection time based on the user's past operational tendencies. Consequently, it enables switching to tracking autofocus (AF) based on the user's operational tendencies.

[0180] Additionally, an upper limit can be set for the configurable detection time. This prevents the detection time from being unintentionally set too long. In this case, it is even more preferable to be able to set the upper limit value arbitrarily.

[0181] Furthermore, in the example above, the statistical method is set to calculate the average value, but the method for calculating the detection time is not limited to this. Alternatively, the method could be set to calculate the median, the most frequent value, etc., as the detection time.

[0182] [Variations on the setting of the tracking object]

[0183] Regarding the tracking target, image recognition can be used to limit it to a specific subject. For example, in videography with a person as the subject, the tracking target can be limited to the person within the focus area. Furthermore, the tracking target can be limited to the face, eyes, etc., rather than the entire person.

[0184] Figure 11This is a block diagram illustrating an example of the functions implemented by the CPU when setting a tracking object using image recognition. Here, we will explain the case where the tracking object is limited to a face.

[0185] like Figure 11 As shown, the CPU 120 also functions as an image recognition unit 140.

[0186] The image recognition unit 140 processes the dynamic image data captured by the camera unit 110 and detects faces within the images. The technique for detecting faces from images can employ known techniques. For example, techniques that use image recognition models generated through machine learning, deep learning, etc., to detect regions of a person's face can be used.

[0187] The tracking object setting unit 120E sets a tracking object based on information about the focus area detected by the focus area detection unit 120D and information about the area of ​​a person's face detected by the image recognition unit 140. Specifically, the tracking object is set in the area of ​​a person's face among the subjects present in the focus area. Therefore, even when the focus is on the entire body of the person, the tracking object is limited to the area of ​​the face and set accordingly.

[0188] This allows for the setting of a tracking object based on specific subjects, such as the camera subject. This further reduces the burden of focusing operations performed by the user.

[0189] Furthermore, while the above example illustrates identifying a person and setting a tracking target, it is also possible to further identify individuals and set tracking targets. That is, it is also possible to set a structure that uses image recognition for personal authentication, limiting the tracking target to a specific person.

[0190] [A variation of displaying information about the focus area]

[0191] In the above embodiment, the structure is set to display the frame of the focus area in a focused state as information representing the focus area. However, the way the information representing the focus area is displayed (in the form of being attached to the live view image) is not limited to this. Any form that allows the focus area to be visually identifiable on the live view image is acceptable.

[0192] Figure 12 This is another example of a display method that shows information about the focus area.

[0193] exist Figure 12 In the example shown, the focus area is displayed using a rectangular frame F3. This frame F3 is, for example, set to be the smallest frame enclosing the focus area when in focus.

[0194] The same applies to the information representing the tracked object. It simply requires that the form of the tracked object be visually identifiable in the live view image.

[0195] Furthermore, in the above embodiments, the switching of the action mode can be identified by changing the color of the frame, but the method for enabling the user to identify the switching of the action mode is not limited to this. The content representing the information of various focus areas can be changed to enable the user to identify the switching of the action mode. For example, the structure can be configured to change the shape of the frame representing the focus area, the type of lines, etc., to identify the switching of the action mode.

[0196] [Settings for turning the tracking autofocus (AF) function on and off]

[0197] In the above embodiment, the structure can be configured such that, in MF mode, when the end of the focusing operation is detected, tracking AF automatically begins; however, this function can be arbitrarily turned on and off by the user. That is, it is configured such that, in MF mode, tracking AF automatically begins only when the function is turned on, upon detection of the end of the focusing operation. This can be configured, for example, by providing a dedicated switch in the main operating unit 118. Alternatively, it can be configured using the display unit 112 and the main operating unit 118. The operating mode for tracking AF in MF mode, i.e., the operating mode when the tracking AF function is set to be on, is another example of the first focusing mode.

[0198] [Second Implementation]

[0199] When starting autofocus (AF), moving the focus in the opposite direction to the previous focus movement can create a sense of disharmony for the viewer. For example, if you move the focus from the nearest side to infinity and then point it at the subject, immediately returning the focus to the nearest side will create a sense of disharmony for the viewer.

[0200] Therefore, in the imaging device of this embodiment, after starting AF tracking, the direction of focus movement is restricted to a certain time and a certain direction. That is, the direction of focus movement based on AF is restricted to the same direction as the direction of focus movement at the end of the focusing operation. Therefore, for example, when moving the focus from the nearest side to the infinity side to align the focus with the target subject, the direction of focus movement based on AF is restricted to the infinity direction for a predetermined time. As a result, stable focus movement can be achieved in the imaging of moving images.

[0201] [Device Structure]

[0202] Here, we will only describe the structure required for the function of restricting the direction of focus movement.

[0203] Figure 13This is a block diagram of the main functions implemented by the CPU when limiting the direction of movement for focusing and performing autofocus (AF).

[0204] like Figure 13 As shown, in the camera device of this embodiment, the CPU 120 also functions as a focus movement direction detection unit 150A and a second timing unit 150B.

[0205] The focus movement direction detection unit 150A detects the focus movement direction based on the operation information of the focus ring output from the focus operation unit 46. In MF (Multi-Focus Mode), the focus moves along the operation direction of the focus ring. Therefore, by detecting the operation direction of the focus ring, the focus movement direction can be detected.

[0206] The second timing unit 150B measures the elapsed time after the start of AF tracking. AF tracking begins based on the detection of the end of the focusing operation. Therefore, the second timing unit 150B begins measuring the elapsed time based on the detection of the end of the focusing operation.

[0207] After AF control begins, the AF control unit 120I restricts the movement direction of the focus to the previous focus movement direction until a certain period of time has elapsed, and performs AF control accordingly. The previous focus movement direction is the focus movement direction just before the end of the focusing operation is detected. If the end of the focusing operation is detected, the previous focus movement direction is detected as the focus movement direction last detected by the focus movement direction detection unit 150A. The elapsed time after the start of AF tracking is measured by the second timing unit 150B. Therefore, the AF control unit 120I restricts the focus movement direction to the previous movement direction for AF control until a predetermined time has elapsed as measured by the second timing unit 150B. The predetermined time is an example of a second time. As an example, in the imaging device of this embodiment, the predetermined time is set to 3 seconds. At this time, after setting the tracking object, the focus movement direction is restricted to the previous movement direction for 3 seconds. In addition, since AF tracking starts based on the detection of the end of the focusing operation, the meaning of the elapsed time after the start of AF tracking is the same as the meaning of the elapsed time after the detection of the end of the focusing operation.

[0208] [Processing sequence for camera control]

[0209] Figure 14 This is a flowchart showing the processing sequence of camera control.

[0210] The processing until AF tracking begins (steps S11 to S16) is the same as that of the imaging device in the first embodiment described above. First, it is determined whether the focusing operation has started (step S11). If the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S12). If it is set to MF mode, the focus evaluation value of each focus area is calculated (step S13), and the focus area is detected based on the calculation result (step S14). Then, the tracking object is set based on the information of the detected focus area (step S15). That is, the subject existing in the focus area is set as the tracking object. Then, it is determined whether the focusing operation has ended (step S16).

[0211] When it is determined that the focusing operation has ended, it is determined whether the prescribed time (second time) has elapsed (step S17). That is, it is determined whether the prescribed time has elapsed since the tracking object was set.

[0212] If the predetermined time has not elapsed (if step S17 is not true), tracking AF (AF) with the focus movement direction restricted is performed (step S18). The AF control unit 120I performs AF control by restricting the focus movement direction to the direction of focus movement just before the end of the focusing operation is detected. For example, when the focusing operation ends by moving the focus from the nearest side to infinity, the focus movement direction is restricted to the infinity direction, and AF control is performed. Therefore, for example, if the tracked object moves towards the nearest side during this period, AF control is not performed.

[0213] On the other hand, when the specified time has elapsed (if step S17 is yes), normal tracking AF is performed (step S19). That is, AF control is performed without restricting the direction of focus movement.

[0214] The tracking AF continues until the focusing operation resumes or until the tracked object disappears. Therefore, the focusing operation is constantly monitored during the tracking AF. That is, it is determined whether a focusing operation has been performed again (step S20). When it is determined that a focusing operation has resumed, the tracking AF is forcibly terminated. After the tracking AF ends, the operation returns to the normal MF mode and the processing after step S12 is repeated.

[0215] As explained above, in the imaging device according to this embodiment, when AF tracking begins, the direction of focus movement is limited to a certain direction for a certain period of time. Therefore, stable focus movement can be achieved in capturing moving images.

[0216] Furthermore, this function can be turned on or off at will according to the user's choice. Also, the specified time can be configured so that the user can arbitrarily change the settings.

[0217] [Third Implementation]

[0218] Typically, in filming moving images, the output format of television, film screens, etc., is determined during recording. Therefore, when the posture (or orientation) of the camera device changes (e.g., from vertical to horizontal shooting), it is considered a performance, scene change, etc. If tracking autofocus continues even when the posture of the camera device changes, it is possible to continue tracking unwanted subjects.

[0219] Therefore, in the camera device of this embodiment, when the posture of the camera device changes during tracking AF, the tracking AF is forcibly terminated and the device switches to normal MF mode. This enables stable focus movement during the recording of moving images.

[0220] [Device Structure]

[0221] Here, we will only describe the structure required for the function of forcibly ending the tracking autofocus when the camera device's posture changes.

[0222] Figure 15 This is a block diagram of the main functions implemented by the CPU when the camera device's posture changes and the tracking autofocus is forcibly terminated.

[0223] like Figure 15 As shown, in the camera device of this embodiment, the CPU 120 also functions as a posture change detection unit 160. Furthermore, the camera device of this embodiment includes a posture detection sensor 50 for detecting the posture of the device body. The posture detection sensor 50 is composed of a known posture detection sensor (e.g., an accelerometer, a gyroscope, or a combination thereof) and is built into the device body.

[0224] The posture change detection unit 160 detects posture changes that are above a threshold relative to a reference posture based on information about the posture of the device body detected by the posture detection sensor 50. In particular, in this embodiment, it detects posture changes around the optical axis. The reference posture is, for example, a posture where the long side of the imaging element 110A is horizontal or vertical. A posture where the long side of the imaging element 110A is horizontal is called a horizontal shooting posture. On the other hand, a posture where the long side of the imaging element 110A is vertical is called a vertical shooting posture. The reference posture is preset by the user. Alternatively, the posture used when capturing moving images is set as the reference posture. In this case, the posture of the device body is detected when capturing moving images and set as the reference posture.

[0225] When the posture change detection unit 160 detects a posture change above a threshold during the tracking process, the tracking processing unit 120H terminates the tracking process.

[0226] Furthermore, when the posture change detection unit 160 detects a posture change exceeding a threshold during AF control, the AF control unit 120I terminates AF control.

[0227] [Processing sequence for camera control]

[0228] Figure 16 This is a flowchart showing the processing sequence of camera control.

[0229] The processing until AF tracking begins (steps S31 to S37) is the same as that of the imaging device in the first embodiment described above. First, it is determined whether the focusing operation has started (step S31). If the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S32). If it is set to MF mode, the focus evaluation value of each focus area is calculated (step S33), and the focus area is detected based on the calculation result (step S34). Then, the tracking object is set based on the information of the detected focus area (step S35). That is, the subject existing in the focus area is set as the tracking object. Then, it is determined whether the focusing operation has ended (step S36). When it is determined that the focusing operation has ended, AF tracking begins (step S37). That is, tracking processing and AF control are performed on the subject set as the tracking object.

[0230] When AF tracking begins, first, it is determined whether a focusing operation has been performed again (step S38). If it is determined that a focusing operation has been restarted, AF tracking is forcibly terminated. Then, the operation returns to the normal MF mode and the processing after step S32 is repeated.

[0231] On the other hand, in step S38, if it is determined that the focusing operation has not been restarted, the posture of the main body of the device is detected (step S39). Then, based on the detection result, it is determined whether there is a change in posture (step S40). That is, it is determined whether the posture has changed by more than a threshold relative to the reference posture.

[0232] If a change in pose is detected, the tracking autofocus (AF) is forcibly terminated. Then, the operation returns to the normal forward MF mode, and the processing after step S32 is repeated.

[0233] On the other hand, if it is determined that there is no change in pose, then the tracking AF continues. That is, the processing after step S37 is repeated.

[0234] As explained above, according to the imaging device of this embodiment, if the posture of the imaging device changes during tracking AF, the tracking AF automatically ends and switches to normal MF mode. Therefore, stable focus movement can be achieved in capturing moving images.

[0235] Furthermore, this function can be turned on or off at will according to the user's selection. Also, the threshold for detecting posture changes can be set by the user, and the type of posture detected can also be set by the user.

[0236] Furthermore, in the above embodiment, a structure is designed to detect changes in the posture of the main body of the device using a dedicated sensor. However, it is also possible to design a structure that analyzes captured dynamic image data to detect changes in the posture of the main body of the device. Alternatively, a structure can be designed to detect changes in the posture of the main body of the device using a sensor that detects hand tremors.

[0237] [Fourth Implementation]

[0238] In the imaging device of the first embodiment described above, if there are multiple subjects in the focus area, all of them are set as tracking objects. However, if multiple subjects are set as tracking objects, subsequent tracking AF cannot be performed if each subject moves differently. Therefore, in the imaging device of this embodiment, when multiple subjects are set as tracking objects, tracking AF is preferentially performed on a specific subject. Specifically, tracking AF is preferentially performed on subjects that do not move or subjects with the smallest amount of movement.

[0239] [Device Structure]

[0240] This section only describes the functions required for performing AF tracking.

[0241] Figure 17 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0242] like Figure 17 As shown, in the camera device of this embodiment, the CPU 120 also functions as the first motion calculation unit 170.

[0243] The first motion calculation unit 170 processes the moving image data acquired through shooting and calculates the amount of movement of the subject within the image represented by the moving image data. In this embodiment, the motion amount is calculated as a motion vector. That is, the first motion calculation unit 170 calculates the motion vector within the image and the motion vector of the subject within the image. The calculation of the motion vector uses known techniques. For example, a method of dividing the screen into M×N blocks (M and N are integers of 2 or more) and calculating the motion vector for each block is used.

[0244] When multiple subjects are designated as tracking targets, the tracking processing unit 120H designates tracking targets based on the movement amount information of each subject calculated by the first movement amount calculation unit 170, and performs tracking processing. Specifically, subjects that do not move or have the smallest movement amount are preferentially designated as tracking targets for tracking processing. Therefore, when multiple subjects are designated as tracking targets, the tracking processing unit 120H excludes subjects whose movement amount is above a threshold from the tracking targets. Thus, ultimately, subjects that do not move or have the smallest movement amount are designated as tracking targets for tracking processing.

[0245] [Processing sequence for camera control]

[0246] Figure 18 This is a flowchart showing the processing sequence of camera control.

[0247] The processing until AF tracking begins (steps S51 to S56) is the same as that of the imaging device in the first embodiment described above. First, it is determined whether the focusing operation has started (step S51). If the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S52). If it is set to MF mode, the focus evaluation value of each focus area is calculated (step S53), and the focus area is detected based on the calculation result (step S54). Then, the tracking object is set based on the information of the detected focus area (step S55). That is, the subject existing in the focus area is set as the tracking object. Then, it is determined whether the focusing operation has ended (step S56).

[0248] If it is determined that the focusing operation has ended, then it is determined whether to set multiple subjects as tracking objects (step S57). If it is determined that multiple subjects have not been set as tracking objects, then tracking AF is performed on the subjects set as tracking objects (step S61).

[0249] On the other hand, if it is determined that multiple subjects are set as tracking objects, the amount of movement of each subject within the image is calculated (step S58). Then, based on the calculation result, it is determined whether there is a moving subject (step S59). This determination is performed on the subjects set as tracking objects. That is, it is determined whether there is a moving subject (moving object) among the subjects set as tracking objects.

[0250] If it is determined that there is a moving subject among the subjects set as the tracking objects, then that subject is excluded from the tracking objects (step S60). Here, subjects with a detected movement amount exceeding a threshold are excluded from the tracking objects.

[0251] Additionally, when all subjects are moving, subjects with the greatest movement are excluded first. In this case, they are excluded in descending order of movement. As a result, the subject with the least movement is selected as the tracking target.

[0252] Then, AF tracking is performed on the subject set as the tracking target (step S61). Then, it is determined whether to perform a refocusing operation again (step S62). If it is determined that the refocusing operation should be restarted, the AF tracking ends. Then, the operation returns to the normal MF mode and the processing after step S52 is repeated.

[0253] Figure 19 This is a conceptual diagram of the settings for the tracking object.

[0254] Figure 19 (A) indicates the initial setting state of the tracked objects. In this example, it shows a scenario where three people 300A, 300B, and 300C are present in the focus area at the point when the end of the focusing operation is detected. At this time, all people 300A, 300B, and 300C are set as tracked objects.

[0255] Figure 19 (B) indicates the set state of the tracked object after a specified time. In this example, it shows the left and right figures 300A and 300C moving. At this time, the left and right figures 300A and 300C, which are moving, are excluded from the tracked objects. Therefore, only the central figure 300B is selected as the tracked object.

[0256] As explained above, in the camera device according to this embodiment, when multiple subjects are set as tracking targets, the tracking target is set based on the subsequent movement of each subject, and tracking AF is performed. This allows for continued focusing on the subject desired by the user. Furthermore, it reduces the user's focus adjustment workload and enables stable focus movement.

[0257] [Fifth Implementation]

[0258] In this embodiment, when multiple subjects are designated as tracking targets, the camera device uses phase difference information to determine the tracking target and performs tracking autofocus (AF). Specifically, using the phase difference information, the amount of movement along the optical axis of each subject is calculated, and the subject with the smallest amount of movement along the optical axis is preferentially designated as the tracking target for tracking AF. The optical axis direction is the direction along the optical axis of the camera optical system 20. In this embodiment, the optical axis direction of the camera optical system 20 is consistent with the direction orthogonal to the light-receiving surface of the imaging element.

[0259] [Device Structure]

[0260] This section only describes the functions required for performing AF tracking.

[0261] Figure 20 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0262] like Figure 20 As shown, in the camera device of this embodiment, the CPU 120 also functions as a second motion calculation unit 180.

[0263] The second motion calculation unit 180 processes the dynamic image data acquired through shooting and calculates the motion amount in the optical axis direction of each subject. Specifically, based on the pixel signal of the phase difference detection pixel, it calculates the phase difference of each focusing area, calculates the change amount, and calculates the motion amount in the optical axis direction of each subject. Therefore, in the imaging device of this embodiment, the change amount of phase difference is set as the motion amount in the optical axis direction.

[0264] When multiple subjects are set as tracking targets, the tracking processing unit 120H sets the tracking targets based on the information of the amount of movement of each subject in the optical axis direction calculated by the second movement amount calculation unit 180, and performs tracking processing. Specifically, subjects that move in the optical axis direction are excluded from the tracking targets, thereby performing tracking processing. Subjects that move in the optical axis direction are those whose amount of movement in the optical axis direction is above a threshold. If all subjects move in the optical axis direction, they are excluded in descending order of the amount of movement in the optical axis direction. As a result, the subject with the smallest amount of movement in the optical axis direction is set as the tracking target, and tracking autofocus (AF) is performed.

[0265] [Processing sequence for camera control]

[0266] Figure 21 This is a flowchart showing the processing sequence of camera control.

[0267] The processing until AF tracking begins (steps S71 to S76) is the same as that of the imaging device in the first embodiment described above. First, it is determined whether the focusing operation has started (step S71). If the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S72). If it is set to MF mode, the focus evaluation value of each focus area is calculated (step S73), and the focus area is detected based on the calculation result (step S74). Then, the tracking object is set based on the information of the detected focus area (step S75). That is, the subject existing in the focus area is set as the tracking object. Then, it is determined whether the focusing operation has ended (step S76).

[0268] If it is determined that the focusing operation has ended, then it is determined whether to set multiple subjects as tracking objects (step S77). If it is determined that multiple subjects have not been set as tracking objects, then tracking AF is performed on the subjects set as tracking objects (step S82).

[0269] On the other hand, if it is determined that multiple subjects are set as tracking objects, it is then determined whether the amount of movement in the optical axis direction can be calculated (step S78). As described above, in the imaging device of this embodiment, the amount of movement in the optical axis direction is calculated based on the phase difference. Therefore, if the phase difference cannot be calculated, the amount of movement in the optical axis direction also cannot be calculated. Therefore, if the phase difference cannot be calculated, it is determined that the amount of movement in the optical axis direction cannot be calculated. For example, if a subject exists in an area where no phase difference detection pixel is configured, it is determined that the phase difference cannot be calculated.

[0270] If it is determined that the amount of movement in the optical axis direction cannot be calculated, the tracking object setting is deactivated (step S84), and the process ends. After the process is completed, it operates in normal MF mode.

[0271] On the other hand, if it is determined that the amount of movement in the optical axis direction can be calculated, then the amount of movement in the optical axis direction of each subject is calculated (step S79). Then, based on the calculation result, it is determined whether there is a subject that has moved in the optical axis direction (step S80). This determination is performed on the subject that is set as the tracking object. That is, it is determined whether there is a subject that has moved in the optical axis direction among the subjects that are set as the tracking object.

[0272] If it is determined that there is a subject that is moving in the optical axis direction among the subjects set as tracking objects, then the subject is excluded from the tracking objects (step S81).

[0273] Furthermore, when all subjects exhibit movement along the optical axis, subjects with the largest amount of movement along the optical axis are prioritized for exclusion. In this case, subjects are excluded in descending order of their movement along the optical axis. As a result, the subject with the smallest amount of movement along the optical axis is designated as the tracking target.

[0274] Then, AF tracking is performed on the subject set as the tracking target (step S82). Then, it is determined whether to perform a refocusing operation (step S83). If it is determined that the refocusing operation should be restarted, the AF tracking ends. Then, the operation returns to the normal MF mode and the processing after step S52 is repeated.

[0275] Figure 22 This is a conceptual diagram of the settings for the tracking object.

[0276] Figure 22 (A) indicates the initial setting state of the tracked objects. In this example, it shows a scenario where three people 300A, 300B, and 300C are present in the focus area at the point when the end of the focusing operation is detected. At this time, all people 300A, 300B, and 300C are set as tracked objects.

[0277] Figure 22(B) indicates the set state of the tracking object after a specified time. In this example, it shows an example where the left and right figures 300A and 300C move forward (directly in front in the depth direction). That is, it shows an example where they move in the direction of the optical axis of the camera optical system 20. At this time, the left and right figures 300A and 300C, which are moving in the direction of the optical axis, are excluded from the tracking object. Therefore, only the central figure 300B is selected as the tracking object.

[0278] As explained above, in the imaging device according to this embodiment, when multiple subjects are set as tracking targets, the tracking target is set based on the subsequent movement of each subject along its optical axis, and tracking autofocus (AF) is performed. This allows for continued focusing on the subject desired by the user. Furthermore, it reduces the user's focus adjustment workload and enables stable focus movement.

[0279] Furthermore, in the above embodiment, the structure is designed to detect the change in phase difference and the amount of movement in the optical axis direction of each subject. However, the method for detecting the amount of movement in the optical axis direction of each subject is not limited to this. Alternatively, the structure can be designed to perform detection using methods such as Time of Flight (ToF) or Structured Light.

[0280] Furthermore, in the above embodiment, a structure is provided that uses an imaging element with a phase difference detection pixel to detect the phase difference, but the method for detecting the phase difference is not limited to this.

[0281] Furthermore, if the movement of the subject can be calculated over the entire area of ​​the image, the processing in step S78 can be omitted. For example, if a focus area is set over the entire area of ​​the image and the phase difference can be detected in each focus area, the processing in step S78 can be omitted.

[0282] [Sixth Implementation]

[0283] When multiple subjects are designated as tracking targets, the imaging device of this embodiment determines the tracking target and performs tracking autofocus (AF) by utilizing information on the amount of movement of each subject within the image and information on the amount of movement of each subject along the optical axis. Specifically, subjects that do not move are preferentially designated as tracking targets, while when all subjects move, the tracking target is determined based on the amount of movement along the optical axis, and tracking AF is performed. That is, the subject with the smallest amount of movement along the optical axis is preferentially designated as the tracking target, and tracking AF is performed.

[0284] [Device Structure]

[0285] This section only describes the functions required for performing AF tracking.

[0286] Figure 23 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0287] like Figure 23 As shown, in the camera device of this embodiment, the CPU 120 also functions as a first motion calculation unit 170 and a second motion calculation unit 180.

[0288] The function of the first motion calculation unit 170 is the same as that of the first motion calculation unit 170 provided in the camera device of the fourth embodiment described above. That is, it processes the moving image data obtained by shooting and calculates the motion amount (motion vector) of the subject within the image represented by the moving image data.

[0289] The function of the second motion calculation unit 180 is the same as that of the second motion calculation unit 180 provided in the camera device of the fifth embodiment described above. That is, it processes the dynamic image data obtained by shooting and calculates the amount of movement (change in phase difference) of each subject in the optical axis direction.

[0290] When multiple subjects are set as tracking objects, the tracking processing unit 120H sets the tracking objects based on the calculation results of the first motion calculation unit 170 and the second motion calculation unit 180, and performs tracking processing.

[0291] [Processing sequence for camera control]

[0292] Figure 24 and Figure 25 This is a flowchart showing the processing sequence of camera control.

[0293] The processing until AF tracking begins (steps S91 to S96) is the same as that of the imaging device in the first embodiment described above. First, it is determined whether the focusing operation has started (step S91). If the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S92). If it is set to MF mode, the focus evaluation value of each focus area is calculated (step S93), and the focus area is detected based on the calculation result (step S94). Then, the tracking object is set based on the information of the detected focus area (step S95). That is, the subject existing in the focus area is set as the tracking object. Then, it is determined whether the focusing operation has ended (step S96).

[0294] If it is determined that the focusing operation has ended, then it is determined whether to set multiple subjects as tracking objects (step S97). If it is determined that multiple subjects have not been set as tracking objects, then tracking AF is performed on the subjects set as tracking objects (step S101).

[0295] On the other hand, if it is determined that multiple subjects are set as tracking objects, the amount of movement of each subject within the image is calculated (step S98). Then, based on the calculation result, it is determined whether all subjects are moving objects (step S99). That is, it is determined whether all subjects set as tracking objects are moving subjects.

[0296] If the subjects set as the tracking objects are not all moving objects, that is, if they include subjects that do not move (subjects whose movement is below the threshold), then subjects that are moving are excluded from the tracking objects (step S100). That is, subjects whose movement is detected to be above the threshold are excluded from the tracking objects.

[0297] Then, AF tracking is performed on the subject set as the tracking target (step S101). Then, it is determined whether to perform a refocusing operation again (step S102). If it is determined that the refocusing operation should be restarted, the AF tracking ends. Then, the operation returns to the normal MF mode and the processing after step S92 is repeated.

[0298] On the other hand, in step S99 above, if it is determined that all the subjects set as tracking objects are moving objects, then the process is transferred to... Figure 25 The process is as follows: Specifically, it determines whether the movement along the optical axis can be calculated (step S103). The determination of whether the movement along the optical axis can be calculated is based on whether the phase difference can be calculated for each subject. If there are subjects for which the phase difference cannot be calculated, it is determined that the movement along the optical axis cannot be calculated.

[0299] If it is determined that the amount of movement in the optical axis direction cannot be calculated, the tracking object setting is deactivated (step S106), and the process ends. After the process is completed, the system operates in normal MF mode.

[0300] On the other hand, if it is determined that the amount of movement in the optical axis direction can be calculated, then the amount of movement in the optical axis direction of each subject is calculated (step S104). Then, based on the calculation result, subjects that have movement in the optical axis direction are excluded from the tracking objects in descending order of the amount of movement (step S105). As a result, the subject with the smallest amount of movement in the optical axis direction is set as the tracking object.

[0301] Then, as Figure 24 As shown, tracking AF is performed on the subject set as the tracking target (step S101). Then, it is determined whether to perform a refocusing operation (step S102). If it is determined that the refocusing operation should be restarted, the tracking AF ends. Then, the operation returns to the normal MF mode and the processing after step S92 is repeated.

[0302] As explained above, the camera device according to this embodiment sets the tracking target and performs tracking autofocus (AF) based on information about the amount of movement of each subject within the image and information about the amount of movement of each subject in the optical axis direction. This allows for continued focusing on the subject desired by the user. Furthermore, it reduces the burden of focus adjustment for the user. Moreover, it enables stable focusing; that is, it allows for achieving the most accurate focus possible.

[0303] [Seventh Implementation]

[0304] The imaging device of this embodiment sets a tracking object based on the focus state of each subject during the focusing operation. Specifically, if there is a subject that changes from a focus state to a defocus state and then returns to a focus state during the focusing operation, that subject is set as the tracking object. The operation of changing from a focus state to a defocus state and then returning to a focus state is called a shift-back operation. A shift-back operation is an operation that immediately returns to focus after blurring during the focusing operation. It is assumed that the subject that has been focused through the shift-back operation when the user performs a shift-back operation is the main subject. Therefore, this subject is set as the tracking object and tracking AF is performed. When there is no subject that changes from a focus state to a defocus state and then returns to a focus state during the focusing operation, the subject that has been in focus for the longest time during the focusing operation is set as the tracking object.

[0305] [Device Structure]

[0306] This section only describes the functions required for setting up the tracking object.

[0307] Figure 26 This is a block diagram of the main functions implemented by the CPU during AF tracing.

[0308] like Figure 26 As shown, in the camera device of this embodiment, the CPU 120 also functions as the third timing unit 190 and the specific operation detection unit 192.

[0309] The third timing unit 190 measures the period during which each focus area is in focus (focusing period) during the focusing operation. The third timing unit 190 measures the focusing period of each focus area based on the information of the focus area detected in real time by the focus area detection unit 120D.

[0310] The specific operation detection unit 192 detects specific operations based on information from real-time calculated focus evaluation values. Specifically, it detects operations that return to focus by shifting. The specific operation detection unit 192 also detects operations that return to focus within a specified time after changing from a focus state to a defocus state, and detects operations that return to focus by shifting.

[0311] The tracking target setting unit 120E sets the tracking target based on the detection results of the focus area detection unit 120D, the measurement results of the third timing unit 190, and the detection results of the specific operation detection unit 192. Specifically, when there is an odd number of subjects in the focus area, the tracking target setting unit 120E sets the subjects in the focus area as tracking targets. On the other hand, when there are multiple subjects in the focus area, the tracking target setting unit 120E prioritizes setting the subject that has been focused through the shift return operation as the tracking target. If there is no subject that has been focused through the shift return operation, the subject with the longest focus state period is set as the tracking target.

[0312] [Processing sequence for camera control]

[0313] Figure 27 This is a flowchart showing the processing sequence of camera control.

[0314] First, it is determined whether a focusing operation has started (step S111). If it is determined that a focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S112). If it is determined that the focusing mode is set to MF mode, the focus evaluation value of each focus area is calculated (step S113). Then, the focus area is detected based on the calculation result (step S114). If a focus area is detected, the focusing period of each focus area is measured based on the detection result (step S115). Furthermore, a specific operation, namely, the operation of returning to the focus point by shifting, is detected based on the calculation result of each focus evaluation value. Then, it is determined whether a specific operation has occurred based on the detection result (step S117).

[0315] When a specific operation is performed, the subject that is aligned to the focus point through that specific operation is set as the tracking object (step S118). That is, the subject that is aligned to the focus point by shifting is set as the tracking object.

[0316] On the other hand, without performing a specific operation, among the subjects existing in the focus area, the subject with the longest focus period is designated, and the designated subject is set as the tracking object (step S119).

[0317] After setting the tracking object, it is determined whether the focusing operation has ended (step S120). Then, when it is determined that the focusing operation has ended, AF tracking begins (step S121). That is, the subject set as the tracking object is continuously tracked and AF controlled.

[0318] When AF tracking begins, it is determined whether a focusing operation has been performed again (step S122). Then, if it is determined that a focusing operation has been restarted, AF tracking ends. After AF tracking ends, the process returns to normal MF mode and repeats the processing after step S2.

[0319] As explained above, the camera device according to this embodiment can automatically set the subject assumed to be the main subject as the tracking object and perform tracking AF even when there are multiple subjects in the focus area.

[0320] Furthermore, while the above embodiment is designed to set the tracking object based on detecting a specific operation, it is also possible to designate the tracking object without detecting a specific operation. In this case, the subject with the longest focusing period is set as the tracking object.

[0321] Similarly, it is also possible to configure the structure to set the tracking object without measuring the focusing period. That is, even if no specific operation is detected, the tracking object can be set without being limited to the subject with the longest focusing period. In addition, if there are multiple subjects in the focusing area, they are all set as tracking objects. However, when multiple subjects are set as tracking objects and AF tracking is started, the tracking objects can be limited by the methods described in embodiments 4 to 6 above.

[0322] [Eighth Implementation]

[0323] During tracking autofocus (AF), the focus remains on the subject being tracked. Conversely, if other subjects enter the depth of field, the focus also remains on those subjects. The subject in focus, other than the tracked subject, can be a passing subject or the main subject.

[0324] Therefore, in the camera device of this embodiment, when a subject that can become the main subject is detected during tracking AF, the subject is added to the tracking target and tracking AF continues. Specifically, if the focus evaluation value is maintained at the same level as that of the subject currently being tracked for a predetermined time, a new subject is also set as the tracking target and tracking AF is performed.

[0325] [Device Structure]

[0326] This section only describes the functions required for adding settings to the tracked objects.

[0327] Figure 28 This is a block diagram of the main functions implemented by the CPU when adding tracking objects for AF (Automatic Field Tracking).

[0328] like Figure 28 As shown, in the camera device of this embodiment, the CPU 120 also functions as a tracking candidate detection unit 200.

[0329] The tracking candidate detection unit 200 detects tracking candidates based on information about the focus area detected by the focus area detection unit 120D during tracking AF. Tracking candidates are subjects that can become the main subject, other than the currently designated tracking subject. The tracking candidate detection unit 200 extracts subjects whose focus evaluation value is similar to that of the currently designated tracking subject for a sustained period of time or more as tracking candidates. "Similarity" means the difference is below a threshold. Therefore, the tracking candidate detection unit 200 extracts subjects whose focus evaluation value difference from that of the currently designated tracking subject is below a threshold for a sustained period of time and detects tracking candidates. Specifically, it extracts focus areas whose focus evaluation value is below the threshold for a sustained period of time and detects subjects existing within those focus areas as tracking candidates. The sustained period used for extraction is an example of a third time.

[0330] Furthermore, during AF tracking, the subject being tracked remains in focus for approximately the entire duration (including states that are substantially considered to be in focus). Therefore, a subject in focus can be considered as having a focus evaluation value similar to that of the subject being tracked. Consequently, the tracking candidate detection unit 200 can also be configured to extract subjects in focus for a sustained period of time or longer as tracking candidates. In this case, the tracking candidate detection unit 200 extracts areas outside the area where the tracking subject is located that remain in focus for a sustained period of time or longer, and detects subjects present in the extracted areas as tracking candidates. Specifically, it extracts focused areas that remain in focus for a sustained period of time or longer, and detects subjects present in the extracted focused areas as tracking candidates.

[0331] If a tracking candidate is detected by the tracking candidate detection unit 200, the tracking processing unit 120H adds the detected tracking candidate to the tracking object for tracking processing.

[0332] [Processing sequence for camera control]

[0333] Figure 29 This is a flowchart showing the processing sequence of camera control.

[0334] The processing until AF tracking begins (steps S131 to S137) is the same as that of the camera device in the first embodiment described above.

[0335] First, it is determined whether the focusing operation has started (step S131). If it is determined that the focusing operation has started, it is determined whether the focusing mode is set to MF mode (step S132). If it is determined that it is set to MF mode, the focusing evaluation value of each focusing area is calculated (step S133), and the focusing area is detected based on the calculation result (step S134). Then, the tracking object is set based on the information of the detected focusing area (step S135). If the tracking object is set, it is determined whether the focusing operation has ended (step S136). If it is determined that the focusing operation has ended, AF tracking begins (step S137).

[0336] If AF tracking begins, a focus evaluation value is calculated (step S138). Focus evaluation values ​​are detected across the entire frame. Then, tracking candidates are detected based on the detected focus evaluation values ​​across the entire frame (step S139).

[0337] The detection of tracking candidates is performed in the following order. First, subjects with a focus evaluation value that is the same as the subject currently designated as the tracking target are extracted. Specifically, focus regions whose focus evaluation value differs from that of the subject currently designated as the tracking target by a value below a threshold are extracted. If a focus region with a difference below the threshold is extracted, the duration of the state with the difference below the threshold is measured. If the state with a difference below the threshold persists for a specified time or longer, the subject present in the extracted focus region is detected as a tracking candidate.

[0338] When a tracking candidate is detected, the tracking object is reset (step S142). That is, the detected tracking candidate is added to the tracking object and configured.

[0339] Then, it is determined whether a focusing operation has been performed again (step S143). If it is determined that a focusing operation has been restarted, the tracking AF is forcibly terminated. After the tracking AF ends, the operation returns to the normal MF mode and the processing after step S132 is repeated.

[0340] On the other hand, if it is determined that the focusing operation has not restarted, tracking AF continues. That is, the processing after step S137 is repeated. At this time, when a new tracking object is added, tracking AF is performed including the newly added tracking object.

[0341] As explained above, the camera device according to this embodiment automatically adds a subject that can become the main subject to the tracking AF when it detects such a subject during tracking AF, and performs tracking AF. This significantly reduces the burden of focusing operations performed by the user. Furthermore, stable focus movement can be achieved in capturing moving images.

[0342] [Other Implementation Methods]

[0343] [Example of a camera device]

[0344] In the camera devices that apply the present invention, in addition to so-called video cameras, movie cameras and other dedicated video devices for animation, camera devices embedded in smartphones, personal computers and the like are also included.

[0345] [Hardware Structure of the Processing Unit]

[0346] In this invention, the hardware structure of the processing units performing various processes is implemented by various processors. These processors include: general-purpose processors that execute programs and function as various processing units, such as CPUs and / or GPUs (Graphics Processing Units); processors with circuitry that can be modified after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs); and processors with circuitry specifically designed for performing specific processes, such as ASICs (Application Specific Integrated Circuits), which are dedicated circuits. The term "program" has the same meaning as "software."

[0347] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be constructed using multiple FPGAs or a combination of a CPU and an FPGA. Furthermore, multiple processing units can also be constructed using a single processor. As examples of multiple processing units constructed using a single processor, firstly, as exemplified by computers used in client and server applications, there exists a method where a single processor is composed of a combination of one or more CPUs and software, functioning as multiple processing units. Secondly, as exemplified by System-on-Chip (SoC), there is a method where a processor is used to implement the overall system functionality containing multiple processing units using a single IC (Integrated Circuit) chip. Thus, various processing units are constructed as hardware structures using one or more of the aforementioned processors.

[0348] Symbol Explanation

[0349] 1-Camera device, 10-Lens unit, 20-Camera optical system, 22-Zoom lens, 24-Aperture, 26-Focusing lens, 30-Lens drive unit, 32-Zoom drive unit, 34-Aperture drive unit, 36-Focusing drive unit, 40-Lens operation unit, 42-Zoom operation unit, 44-Aperture operation unit, 46-Focusing operation unit, 50-Attitude detection sensor, 100-Main body, 110-Camera unit, 110A-Camera element, 112-Display unit, 114-Storage unit, 116-Connection unit, 118-Main body operation unit, 120-CP U, 120A - Image processing unit for display, 120B - Display control unit, 120C - Focus evaluation value calculation unit, 120D - Focus area detection unit, 120E - Tracking object setting unit, 120F - Focus operation detection unit, 120G - Focus drive control unit, 120H - Tracking processing unit, 120I - AF control unit, 122 - ROM, 124 - RAM, 114A - Measurement result storage unit, 130A - First timing unit, 130B - Measurement result recording control unit, 130C - Detection time calculation unit, 130D - Detection time setting unit 140 - Image Recognition Unit, 150A - Focus Movement Direction Detection Unit, 150B - Second Timing Unit, 160 - Pose Change Detection Unit, 170 - First Movement Calculation Unit, 180 - Second Movement Calculation Unit, 190 - Third Timing Unit, 192 - Specific Operation Detection Unit, 200 - Tracking Candidate Detection Unit, 300 - Person, 300A - Person, 300B - Person, 300C - Person, 400 - Tree, F1 - Frame of the Focused Area in Focused State, F2 - Frame of the Focused Area in Focused State, F3 - Focused Area in Focused State Frame, FA - Focusing area, IM1 - Image displayed during focusing operation, IM2 - Image displayed during tracking AF, L - Optical axis, S1~S8 - Processing sequence of camera control, S11~S20 - Processing sequence of camera control, S31~S40 - Processing sequence of camera control, S51~S62 - Processing sequence of camera control, S71~S83 - Processing sequence of camera control, S91~S106 - Processing sequence of camera control, S111~S122 - Processing sequence of camera control, S131~S142 - Processing sequence of camera control.

Claims

1. A camera device comprising a camera optical system, a camera element, and a processor. The processor performs the following processing in the first focus mode: The camera optical system outputs the dynamic image data captured by the camera element to the display terminal. Detect the focus area in the image represented by the dynamic image data. If a focusing operation has been performed, the end of the focusing operation is detected based on the time associated with the focusing operation. Detect the direction of focus movement just before the end of the focusing operation is detected. If the end of the focusing operation is detected, the subject present in the focus area is set as the tracking object, and tracking processing and autofocus control continue. Furthermore, during the period from the start of the tracking processing and autofocus control until the second time interval has elapsed, the direction of focus movement based on the autofocus control is restricted to the direction of focus movement before the end of the focusing operation is detected. If the focusing operation is performed again after the tracking process and the autofocus control have started, the tracking process and the autofocus control shall be terminated.

2. The camera device according to claim 1, wherein, The time associated with the focusing operation is the time of the inactive state.

3. The camera device according to claim 2, wherein, The processor performs the following processing: Measure the duration of the inactive state. The detection period is 1 time elapsed from the start of the no-operation state, and the focus operation is also detected as ending.

4. The camera device according to claim 3, wherein, The processor performs the following processing: Record information about the duration of the inactive state. The first time is set based on the recorded duration of the inactive state.

5. The camera device according to claim 4, wherein, The processor calculates and sets the first time based on the recorded duration of the inactive state using a statistical method.

6. The camera device according to claim 5, wherein, The processor calculates the average, median, or most frequent duration of the most recent specified number of inactive states as the first time.

7. The camera device according to any one of claims 1 to 6, wherein, The processor assigns information representing the focus area to the image represented by the dynamic image data and outputs the dynamic image data to the display terminal.

8. The camera device according to claim 7, wherein, The processor changes the content of the information representing the focus area based on the detection of the end of the focusing operation.

9. The camera device according to any one of claims 1 to 6, wherein, The processor performs the following processing: Obtain information about the posture of the main body of the device. If the posture of the main body of the device changes after the tracking process and the autofocus control are initiated, the tracking process and the autofocus control are terminated.

10. The camera device according to any one of claims 1 to 6, wherein, When multiple subjects are set as the tracking objects The processor performs the following processing: Calculate the amount of movement of each of the subjects. The tracking object is set based on the calculated amount of movement.

11. The camera device according to claim 10, wherein, The processor sets the subject that is not moving or the subject with the least amount of movement as the tracking object.

12. The camera device according to claim 10, wherein, The amount of movement includes the amount of movement along the optical axis of the imaging optical system. When it is determined that all the subjects are moving, the processor sets the subject with the smallest amount of movement in the optical axis direction as the tracking object.

13. The camera device according to claim 12, wherein, When the processor determines that all the subjects are moving, and cannot obtain information on the amount of movement in the optical axis direction, it releases the tracking target setting for all the subjects.

14. The camera device according to any one of claims 1 to 6, wherein, The processor performs the following processing: After the tracking process and the autofocus control are initiated, focus evaluation values ​​for multiple regions within the image are calculated based on the dynamic image data. The region with a focus evaluation value of the same degree as the tracked object is extracted and maintained for a third time, and the subject present in the extracted region is newly added and set as the tracked object.

15. A camera device comprising a camera optical system, a camera element, and a processor. The processor performs the following processing in the first focus mode: The camera optical system outputs the dynamic image data captured by the camera element to the display terminal. Detect the focus area in the image represented by the dynamic image data. The focus state of the subject contained in the dynamic image data is detected based on the dynamic image data. If a focusing operation has been performed, the end of the focusing operation is detected based on the time associated with the focusing operation. If the focus operation is detected to have ended, the subject existing in the focus area, i.e., the subject that changed from a focus state to a defocus state and then returned to a focus state during the focus operation, is set as the tracking object, and tracking processing and autofocus control continue. If the focusing operation is performed again after the tracking process and the autofocus control have started, the tracking process and the autofocus control shall be terminated.

16. The camera device according to claim 15, wherein, When there is no subject that changes from a focus state to a defocus state and then returns to a focus state during the focusing operation, the processor sets the subject with the longest focus state period during the focusing operation as the tracking object.

17. The camera device according to claim 15 or 16, wherein, The time associated with the focusing operation is the time of the inactive state.

18. The camera device according to claim 15 or 16, wherein, The processor assigns information representing the focus area to the image represented by the dynamic image data and outputs the dynamic image data to the display terminal.

19. The camera device according to claim 15 or 16, wherein, The processor performs the following processing: Obtain information about the posture of the main body of the device. If the posture of the main body of the device changes after the tracking process and the autofocus control are initiated, the tracking process and the autofocus control are terminated.

20. The camera device according to claim 15 or 16, wherein, When multiple subjects are set as the tracking objects The processor performs the following processing: Calculate the amount of movement of each of the subjects. The tracking object is set based on the calculated amount of movement.

21. The camera device according to claim 15 or 16, wherein, The processor performs the following processing: After the tracking process and the autofocus control are initiated, focus evaluation values ​​for multiple regions within the image are calculated based on the dynamic image data. The region with a focus evaluation value of the same degree as the tracked object is extracted and maintained for a third time, and the subject present in the extracted region is newly added and set as the tracked object.

22. A camera control method, which performs the following processing: The focus area in the image represented by the dynamic image data is detected based on the dynamic image data captured by the camera element via the camera optical system. If a focusing operation has been performed, the end of the focusing operation is detected based on the time associated with the focusing operation. Detect the direction of focus movement just before the end of the focusing operation is detected. If the end of the focusing operation is detected, the subject present in the focus area is set as the tracking object, and tracking processing and autofocus control continue. Furthermore, during the period from the start of the tracking processing and autofocus control until the second time interval has elapsed, the direction of focus movement based on the autofocus control is restricted to the direction of focus movement before the end of the focusing operation is detected. If the focusing operation is performed again after the tracking process and the autofocus control have started, the tracking process and the autofocus control shall be terminated.

23. A camera control method, which performs the following processing: The focus area in the image represented by the dynamic image data is detected based on the dynamic image data captured by the camera element via the camera optical system. Based on the dynamic image data, the focus state of the subject contained in the dynamic image data is detected. If a focusing operation has been performed, the end of the focusing operation is detected based on the time associated with the focusing operation. If the focus operation is detected to have ended, the subject existing in the focus area, i.e., the subject that changed from a focus state to a defocus state and then returned to a focus state during the focus operation, is set as the tracking object, and tracking processing and autofocus control continue. If the focusing operation is performed again after the tracking process and the autofocus control have started, the tracking process and the autofocus control shall be terminated.

24. A recording medium, which is non-transitory and computer-readable, having recorded a program that causes a computer to execute the camera control method of claim 22.

Images