Imaging apparatus, focus position setting method, and program

Abstract

To perform focus control intended by a user in autofocus control.SOLUTION: An imaging apparatus according to the present technique comprises: a user operation detection unit that detects a user's operation for focus control; a focus position movement detection unit that detects the movement of a focus position based on the user's operation detected by the user operation detection unit; a depth information detection unit that detects depth information in a focus control target region; and a focus control unit that sets a target focus position on the basis of the movement of the focus position and the depth information. Consequently, a technique is proposed with which focus control further in line with the user's intention is executed.SELECTED DRAWING: Figure 23

Description

[Technical Field]

[0001] The present technology relates to an imaging device, a focus position setting method, and a program for automatic focus control and manual focus control. [Background technology]

[0002] Some imaging devices are equipped with both automatic focus control (autofocus control) and manual focus control (manual focus control). In such imaging devices, there is a demand for linking the autofocus control and manual focus control to perform appropriate focus control for a subject intended by the user. For example, Patent Document 1 proposes a method for determining a main subject with high accuracy in order to keep the subject that the user intends to focus on. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Publication No. 2018-125536 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the method described in Patent Document 1, the determination of the main subject is performed automatically, and therefore focus control is not necessarily performed as intended by the user. Therefore, the present disclosure proposes a technique for executing focus control that is more in line with the user's intentions. [Means for solving the problem]

[0005] The imaging device according to the present technology includes a user operation detection unit that detects a user operation for focus control, a focus position movement detection unit that detects movement of the focus position based on the user operation detected by the user operation detection unit, a depth information detection unit that detects depth information within a focus control target area, and a focus control unit that sets a target focus position based on the movement of the focus position and the depth information. That is, a change in the focus position based on a user operation is detected, and after the user operation is completed, a target focus position is set and focus control is performed that reflects the user operation. In the present disclosure, the "focus position" refers to the position in the optical axis direction that is in focus from the imaging device, and if there is a subject that is in focus, it refers to the position of the subject relative to the imaging device. The "target focus position" is a target position for changing the focus position, and in many cases refers to the position of the subject in the depth direction that is the target of focus control. The depth direction refers to the direction from the imaging device to the subject.

[0006] The focus position movement detection unit in the imaging device may detect a movement direction of the focus position. By detecting the movement direction of the focus position, the user's intention can be estimated more accurately.

[0007] The focus position movement detection unit in the imaging device may detect the movement direction at predetermined time intervals. By detecting the movement direction at predetermined time intervals, the operation direction of the user can be detected appropriately.

[0008] The depth information detection unit in the above-mentioned imaging device may detect depth information of the entire focus control target area, and the focus control unit may set the target focus position based on the depth information of the entire focus control target area, the movement direction of the focus position, and an end focus position when it is detected that the user's operation for moving the focus position has ended. By setting the target focus position based on the end focus position, the user's intention is reflected in the focus control.

[0009] The imaging device described above may include an in-screen target position setting unit that sets an in-screen target position based on the depth information, and the focus control unit may set a focus position corresponding to the in-screen target position as the target focus position. The target position within the screen is a region set on the captured image, which is a region consisting of a group of pixels with similar depth information. For example, a pixel region containing a certain subject is set as the target position within the screen because the depth information for each pixel is similar.

[0010] In the above-described imaging device, when the in-screen target position setting unit sets a plurality of in-screen target positions, the focus control unit may set, as the target focus position, a focus position that is closest to the end focus position among the focus positions corresponding to the in-screen target positions. As a result, when the user moves the focus position to the vicinity of the target subject, focus control is performed on the target subject.

[0011] The imaging device described above may include a subject recognition unit that recognizes a subject, and when the focus control unit determines that the target position within the screen coincides with the subject position within the screen, which is the position of the subject within the screen recognized by the subject recognition unit, the focus control unit may set a focus position corresponding to the subject position within the screen as the target focus position. As a result, for example, an image recognition process is used to select a subject to be subjected to focus control.

[0012] In the above-described imaging device, when the subject recognition unit recognizes a plurality of subjects, the focus control unit may set, as the target focus position, a focus position that is closest to the end focus position among focus positions corresponding to the subject positions within the screen. This allows an appropriate subject to be selected as a focus control target, and also allows a subject that reflects the user's intention to be selected as a focus control target.

[0013] The focus control unit in the imaging device described above may perform tracking focus control on the subject focused in accordance with the target focus position that has been set after the movement of the focus position has been completed. This eliminates the need for the user to continue to focus on a subject that has already been manually focused on.

[0014] The imaging device may further include a region setting section that sets the focus control target region in response to a user operation. This makes it possible to select a focus control target from among the subjects in a manner that better reflects the user's intentions.

[0015] The imaging device described above may be provided with a display control unit that performs a process of superimposing a first frame image on a subject that is focused by the focus control unit among the subjects recognized by the subject recognition unit, and a process of superimposing a second frame image on other subjects. This allows the user to check the focus control target by checking the display unit.

[0016] The focus control unit in the imaging device may perform focus control in accordance with a set focus movement speed. By simply changing the operation mode, the speed (time) until focus is achieved during focus control can be specified.

[0017] In the imaging device described above, the user operation detection unit may detect a user operation on a focus ring, and the focus position movement detection unit may detect the movement direction of the focus position based on the operation direction of the focus ring. In addition, the user operation detection unit in the above-mentioned imaging device may detect a user's focus operation by receiving operation information transmitted from an external operation device, and the focus position movement detection unit may detect the movement direction of the focus position based on the operation direction of the focus operation. This allows the movement direction of the focus position to be detected appropriately.

[0018] The focus control unit in the above-mentioned imaging device may be capable of switching between a first mode that performs autofocus control on a subject and a second mode that performs manual focus control, and may switch to the second mode when the user operation detection unit detects an operation on the focus ring in the first mode. This allows the user to simply operate the focus ring to switch to the second mode.

[0019] The user operation detection unit in the above-mentioned imaging device may determine that the end of movement of the focus position has been detected when a state in which an operation on the focus ring is detected as the user's operation changes to a state in which an operation on the focus ring is not detected. This makes it possible to detect the end of a user operation on the focus ring using, for example, a sensor or the like.

[0020] The focus control unit in the above-described imaging device may be capable of executing a first control of setting, as the target focus position, a focus position that is closest to the end focus position among focus positions corresponding to the target position within the screen, and a second control of setting the target focus position in accordance with the focus position at the end of the movement and the movement direction, and may switch from the first control to the second control when a deviation between the closest focus position and the end focus position is equal to or greater than a predetermined threshold. This allows an appropriate selection to be made according to the user's operation mode and the position of the subject.

[0021] The focus position setting method according to the present technology detects a user's operation for focus control, detects movement of the focus position based on the detected user operation, detects depth information within a focus control target area, and sets a target focus position based on the movement of the focus position and the depth information.

[0022] The program related to the present technology is a program that causes an imaging device to execute the following processes: detecting a user's operation for focus control; detecting a movement of the focus position based on the detected user's operation; detecting depth information within a focus control target area; and setting a target focus position based on the movement of the focus position and the depth information. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a perspective view of an imaging device according to an embodiment of the present technology; [Figure 2] FIG. [Figure 3] FIG. 1 is a block diagram of an imaging device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus. [Figure 5] FIG. 10 is a diagram showing an example of a target position within a screen. [Figure 6] 10A and 10B are diagrams showing examples of a first frame image and a second frame image. [Figure 7] FIG. 10 is a diagram showing an example of a through image displayed on a rear monitor. [Figure 8] 1A and 1B are diagrams for explaining the positional relationship between an imaging device and a subject, and an example of movement of a focus position by a user operation. [Figure 9] FIG. 10 is a diagram illustrating an example of a small region for which depth information is calculated. [Figure 10] FIG. 10 is a diagram illustrating a change in a focus control target by a user operation. [Figure 11] 10A and 10B are diagrams for explaining another example of the positional relationship between the imaging device and the subject and movement of the focus position by a user operation. [Figure 12]FIG. 10 is a diagram illustrating another example in which the focus control target is changed by a user operation. [Figure 13] 10 is a diagram showing an example in which target positions within a screen are set inside and outside a focus control target area. FIG. [Figure 14] 10A and 10B are diagrams for explaining yet another example of the positional relationship between the imaging device and the subject and movement of the focus position by a user operation. [Figure 15] 10A and 10B are diagrams for explaining another example of the positional relationship between the imaging device and the subject and the movement of the focus position by a user operation. [Figure 16] 17 and 18 are diagrams showing an example in which the focus control target is changed for a moving subject, and this diagram shows the state before the focus control target is changed. [Figure 17] FIG. 10 is a diagram showing a state after the focus control target has been changed. [Figure 18] FIG. 10 is a diagram showing a state in which a subject selected as a focus control target moves toward the imaging device. [Figure 19] 20, 21, and 22, and shows another example of the display mode for the subject selected as the focus control target, and this figure shows the state before the focus control target is changed. [Figure 20] FIG. 10 is a diagram illustrating a state during operation of a focus ring. [Figure 21] FIG. 10 is a diagram showing a state after the focus control target has been changed. [Figure 22] FIG. 10 is a diagram showing a state in which a subject selected as a focus control target moves toward the imaging device. [Figure 23] 10 is a flowchart illustrating an example of processing executed by a camera control unit. [Figure 24] 10 is a flowchart showing a first example of a process for acquiring information within a focus control target area. [Figure 25] 10 is a flowchart illustrating a second example of a process for acquiring information within a focus control target area. [Figure 26]10 is a flowchart of a first example of a process for selecting a new focus control target. [Figure 27] 10 is a flowchart of a second example of a process for selecting a new focus control target. [Figure 28] 10 is a flowchart of a fourth example of a process for selecting a new focus control target. [Figure 29] 10 is a flowchart for changing the focusing speed for a subject that is a focus control target in accordance with the user's operation mode. [Figure 30] FIG. 1 is a diagram illustrating an example of a schematic configuration of an endoscope system. [Figure 31] 31 is a block diagram showing an example of the functional configuration of the camera head and the CCU shown in FIG. 30. FIG. [Figure 32] FIG. 1 is a diagram illustrating an example of a schematic configuration of a microsurgery system. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments will be described in the following order with reference to the accompanying drawings. <1. Configuration of imaging device> 2. Functional configuration of the imaging device <3. Example of selecting focus control target> <3-1. First selection example> <3-2. Second selection example> <3-3. Third selection example> <3-4. Other selection examples> <4. Examples of display modes on the display unit> <5. Processing example> <5-1. Overall flow> <5-2. First example of focus control target selection process> <5-3. Second example of focus control target selection process> <5-4. Third example of focus control target selection process> <5-5. Fourth Example of Focus Control Target Selection Process> <6. Focus movement speed control> <7. Application Examples> <8. Summary> <9. This Technology>

[0025] <1. Configuration of imaging device> The appearance of an imaging device 1 according to this embodiment is shown in FIGS. In the following examples, the proximity or distance of a subject or a focus control target may be described, with "close" indicating closer to the imaging device 1 and "far" indicating farther from the imaging device 1. In other words, a "close subject" indicates a subject closer to the imaging device 1. Furthermore, although the imaging device 1 is an example equipped with an interchangeable lens, the invention is not limited to this and can be widely applied to, for example, still cameras, video cameras, and various imaging devices incorporated into other devices.

[0026] The imaging device 1 is configured to include a camera housing 2 in which required components are arranged inside and outside, and a lens barrel 3 attached to a front portion 2a of the camera housing 2. A rear monitor 4 is disposed on the rear surface 2b of the camera housing 2. The rear monitor 4 displays a through image, a recorded image, and the like.

[0027] The rear monitor 4 is a display device such as a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

[0028] An EVF (Electric View Finder) 5 is disposed on the top surface 2c of the camera housing 2. The EVF 5 includes an EVF monitor 5a and a frame-shaped enclosure 5b that protrudes rearward so as to surround the upper and left and right sides of the EVF monitor 5a.

[0029] The EVF monitor 5a is formed using an LCD, an organic EL display, etc. Note that an optical viewfinder (OVF) may be provided instead of the EVF monitor 5a.

[0030] The rear surface 2b and the top surface 2c are provided with various controls 6. The controls 6 include, for example, a playback menu start button, a decision button, a cross key, a cancel button, a zoom key, a slide key, and a shutter button 6S (release button).

[0031] The various types of controls 6 include various types of controls such as buttons, dials, composite controls that can be pressed and rotated, etc. The various types of controls 6 enable, for example, menu operation, playback operation, mode selection / switching operation, focus operation, zoom operation, and parameter selection / setting such as shutter speed and F-number.

[0032] The lens barrel 3 has various lenses arranged therein, and is configured with a ring-shaped focus ring 7 and a ring-shaped zoom ring 8.

[0033] The focus ring 7 is rotatable in the circumferential direction, and various lenses move in the optical axis direction depending on the rotation direction, thereby moving the focus position in the optical axis direction. The "focus position" is the position in the optical axis direction at which the image is in focus from the imaging device 1. For example, if there is a subject in focus, this is the position of the subject relative to the imaging device 1. The focus position changes due to focus control. By rotating the focus ring 7, it is possible to move the focus position closer or farther away from the imaging device 1. Furthermore, by rotating the focus ring 7, manual focus control can be achieved to manually adjust the in-focus state.

[0034] The zoom ring 8 is rotatable in the circumferential direction, and various lenses move in the optical axis direction depending on the direction of rotation, thereby enabling manual zooming control.

[0035] FIG. 3 is a block diagram of the imaging device 1. As shown in FIG. Inside and outside the camera housing 2 and lens barrel 3 of the imaging device 1, there are provided a lens system 9, an imaging element section 10, a signal processing section 11, a recording control section 12, a display section 13, an output section 14, an operation section 15, a camera control section 16, a memory section 17, a driver section 18, a sensor section 19, etc. In addition to these, a power supply section and the like are also provided as appropriate.

[0036] The lens system 9 is composed of various lenses such as an entrance end lens, a zoom lens, a focus lens, and a condenser lens, an aperture mechanism that controls exposure by adjusting the opening size of the lens or iris (aperture) so that sensing is performed while the signal charge is not saturated and is within the dynamic range, and a shutter unit such as a focal plane shutter. It should be noted that some of the optical components such as the lens system 9 may be provided in the camera housing 2 .

[0037] The image sensor unit 10 is, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal-Oxide Semiconductor) type, and is configured with a sensing element in which multiple pixels are arranged two-dimensionally, thereby performing exposure control for light from a subject that is incident through a lens system 9.

[0038] The image sensor unit 10 includes a processing unit that performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and A / D (Analog / Digital) conversion processing on the electrical signals photoelectrically converted by the pixels. Therefore, the image sensor unit 10 outputs captured image signals as digital data to the signal processing unit 11 and the camera control unit 16.

[0039] The signal processing unit 11 is configured by, for example, a microprocessor specialized for digital signal processing, such as a DSP (Digital Signal Processor), a microcomputer, or the like.

[0040] The signal processing unit 11 includes various units for performing various types of signal processing on the digital signal (captured image signal) sent from the imaging element unit 10.

[0041] Specifically, correction processing between the R, G, and B color channels, white balance correction, aberration correction, shading correction, and the like are performed. In addition, the signal processing unit 11 performs various processes such as YC generation processing to generate (separate) a luminance (Y) signal and a color (C) signal from the R, G, and B image data, processing to adjust the luminance and color, knee correction, and gamma correction.

[0042] Furthermore, the signal processing unit 11 performs conversion to a final output format by performing resolution conversion processing and codec processing for encoding for recording or communication. The image data converted to a final output format is stored in the memory unit 17. The image data is output to the display unit 13, whereby an image is displayed on the rear monitor 4 or the EVF monitor 5a. The image data is also output from an external output terminal, whereby it is displayed on a device such as a monitor provided outside the imaging device 1.

[0043] The recording control unit 12 is made up of, for example, a nonvolatile memory, and functions as a storage means for storing image files (content files) such as still image data and video data, attribute information of the image files, thumbnail images, and the like. Image files are stored in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format). There are various possible actual forms for the recording control unit 12. For example, the recording control unit 12 may be configured as a flash memory built into the imaging device 1, or may be configured as a memory card (for example, a portable flash memory) that can be attached to or detached from the imaging device 1 and an access unit that accesses the memory card for storage and reading. The recording control unit 12 may also be realized as an HDD (Hard Disk Drive) built into the imaging device 1.

[0044] The display unit 13 executes processing for providing various displays to the photographer. The display unit 13 is, for example, the rear monitor 4 or the EVF monitor 5a. The display unit 13 executes processing for displaying image data that has been converted to an appropriate resolution and input from the signal processing unit 11. This allows the display of a so-called through image, which is an image captured during release standby. Furthermore, the display unit 13 realizes the display of various operation menus, icons, messages, etc. as a GUI (Graphical User Interface) on the screen based on instructions from the camera control unit 16. The display unit 13 is also capable of displaying a reproduced image of the image data read from the recording medium by the recording control unit 12.

[0045] In this example, both the EVF monitor 5a and the rear monitor 4 are provided, but the implementation of the present technology is not limited to this configuration, and only one of the EVF monitor 5a and the rear monitor 4 may be provided, or either or both of the EVF monitor 5a and the rear monitor 4 may be configured to be detachable.

[0046] The output unit 14 performs wired or wireless data communication and network communication with external devices, for example, transmitting captured image data (still image files and video files) to external display devices, recording devices, playback devices, etc. The output unit 14 may also function as a network communication unit. For example, the output unit 14 may communicate via various networks such as the Internet, a home network, or a LAN (Local Area Network) and transmit and receive various data to and from servers, terminals, and the like on the network.

[0047] The operation unit 15 includes not only the various operators 6 described above, but also a rear monitor 4 that employs a touch panel system, and outputs operation information to the camera control unit 16 in response to various operations such as tapping and swiping performed by the photographer. The operation unit 15 may function as a receiving unit for an external operation device such as a remote controller separate from the imaging device 1. Possible external operation devices include a smartphone, a tablet, a Bluetooth (registered trademark) remote controller, a wired remote controller, and a wireless operation device for focus operation.

[0048] The focus ring 7 that detects an operation for manual focus control and the zoom ring 8 that detects an operation for zooming control are considered to be examples of the operation unit 15.

[0049] The camera control unit 16 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit), and performs overall control of the imaging device 1. For example, it controls the shutter speed in response to the operation of the photographer, gives instructions regarding various signal processes in the signal processing unit 11, performs imaging operations and recording operations in response to the user's operation, and plays back recorded image files. The camera control unit 16 switches between various shooting modes, for example, a still image shooting mode, a video shooting mode, and a continuous shooting mode in which still images are continuously captured.

[0050] The camera control unit 16 performs user interface control to enable the user to operate these functions. The UI (User Interface) control includes processing to detect operations on the controls 6 provided on the imaging device 1, display processing on the rear monitor 4, and operation detection processing.

[0051] Furthermore, the camera control unit 16 issues instructions to a driver unit 18 to control the various lenses included in the lens system 9 . For example, it performs processing to specify an aperture value to ensure the amount of light required for AF (Auto Focus) control, and instructs the operation of the aperture mechanism according to the aperture value.

[0052] The memory unit 17 stores information and the like used in the processing executed by the camera control unit 16. The illustrated memory unit 17 comprehensively represents, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like. The memory unit 17 may be a memory area built into the microcomputer chip that serves as the camera control unit 16, or may be configured as a separate memory chip.

[0053] The ROM, flash memory, etc. of the memory unit 17 stores programs and the like used by the camera control unit 16. The ROM, flash memory, etc. stores an OS (Operating System) for the CPU to control each unit, content files such as image files, as well as application programs and firmware for various operations. The camera control unit 16 executes the program to control the entire imaging device 1 including the lens barrel 3.

[0054] The RAM of the memory unit 17 is used as a work area for the camera control unit 16 by temporarily storing data, programs, etc. used when the CPU of the camera control unit 16 executes various data processing.

[0055] The driver unit 18 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, an aperture mechanism driver for a motor that drives an aperture mechanism, and the like. Each driver supplies a drive current to the corresponding drive motor in response to an instruction from the camera control unit 16 .

[0056] The sensor unit 19 collectively refers to various sensors mounted on the imaging device 1. The sensor unit 19 includes, for example, a position information sensor, an illuminance sensor, an acceleration sensor, and the like. A sensor provided on the focus ring 7 or the zoom ring 8 for detecting the rotation direction or the operation amount of the focus ring 7 or the zoom ring 8 is one example of the sensor unit 19 .

[0057] 2. Functional configuration of the imaging device The camera control unit 16 executes various functions by executing programs stored in the memory unit 17 . Each function of the camera control unit 16 will be described with reference to Fig. 4. Note that some of the functions may be included in the signal processing unit 11. Furthermore, some of the functions may be realized by cooperation between the camera control unit 16 and the signal processing unit 11.

[0058] The camera control unit 16 has the functions of a user operation detection unit 31, a focus position movement detection unit 32, a depth information detection unit 33, an in-screen target position setting unit 34, a subject recognition unit 35, an area setting unit 36, a mode switching unit 37, a focus control unit 38, and a display control unit 39.

[0059] The user operation detection unit 31 performs a process of detecting a user operation on the operation unit 15. Specifically, the process of detecting an operation of pressing the shutter button 6S and an operation of rotating the focus ring 7 is performed. Detecting manipulation and a process for detecting an operation of rotating the zoom ring 8.

[0060] The user operation detection unit 31 also detects the amount and type of operation, for example, by distinguishing between half-pressing the shutter button 6S and fully pressing the shutter button 6S. Furthermore, when the user operation detection unit 31 detects an operation to rotate the focus ring 7 or the zoom ring 8, it detects the direction of rotation and the amount of rotation.

[0061] The user operation detection unit 31 detects the start and end of a user operation to rotate the focus ring 7. There are various possible ways to detect the start of a user operation, such as detecting that the user has touched the focus ring 7 using a sensor or the like provided on the focus ring 7, thereby determining that this is the start of a user operation. The end of the user operation can be detected in various ways, such as determining that the user operation has ended when a state changes from one in which an operation on the focus ring 7 is detected to one in which it is not detected. Specifically, the end of the user operation may be detected based on the output of a sensor that detects contact of a person's finger with the focus ring 7, or may be detected when no operation is performed for a certain period of time.

[0062] The user operation detection unit 31 may detect an operation performed by an external operating device. For example, when the operation unit 15 receives an operation signal from an external operation device such as a smartphone, a tablet, a Bluetooth remote controller, a wired remote controller, or a wireless operation device for focus operation, the user operation detection unit 31 may recognize the operation as a focus operation. In the following description, focus operation using the focus ring 7 will be taken as an example, but the present technology can also be applied to focus operations performed by remote control.

[0063] The focus position movement detection unit 32 detects (calculates) the movement direction (movement direction of the focus position) and movement distance (movement amount of the focus position) of the focus position based on the operation direction and operation amount of the focus ring 7 detected by the user operation detection unit 31. There are various possible ways to detect the movement direction of the focus position. As an example, the minute operation amount (or the difference in rotation position) of the focus ring 7 detected for each frame (e.g., 1 / 60 sec or 1 / 30 sec) is acquired, and the operation direction of the user with respect to the focus ring 7, i.e., the rotation direction of the focus ring 7, is determined based on the multiple minute operation amounts. Furthermore, the movement distance of the focus position can be obtained by integrating the minute amount of operation. Alternatively, the amount of operation may be acquired at predetermined time intervals rather than for each frame.

[0064] The depth information detection unit 33 detects, for example, one piece of depth information for each pixel region consisting of multiple pixels. The depth information is taken as depth information of the subject, and may be detected, for example, based on a phase difference signal output from an image plane phase difference pixel included in the image sensor unit 10, or may be detected by receiving reflected light of a reference light such as near-infrared light using a depth sensor provided separately from the image sensor unit 10, or by acquiring distortion or intensity of the distribution of the reflected light. Note that depth information may be detected for each pixel. When a focus control target area FA, which will be described later, is set, depth information may be detected only for the focus control target area FA.

[0065] The in-screen target position setting unit 34 sets an in-screen target position PS based on the depth information. The in-screen target position PS is an area set on the captured image, and is an area consisting of a group of pixels with similar depth information. In other words, the in-screen target position PS set by the in-screen target position setting unit 34 is an area that can be set for each subject, and is an area in which a subject that is a candidate for focus control is captured.

[0066] An example of an in-screen target position PS is shown in Fig. 5. In this example, an in-screen target position PS1 with a depth value DP1 and an in-screen target position PS2 with a depth value DP2 are set on a captured image.

[0067] The subject recognition unit 35 performs image processing on the captured image to recognize the subject in the captured image. The subject recognition unit 35 may be capable of executing processing to determine the type of subject. This makes it possible to perform appropriate focus control that reflects the user's desire to perform focus control on a person or a cat.

[0068] Furthermore, the subject recognition unit 35 sets the position of the recognized subject on the captured image as an in-screen subject position PO. The in-screen subject position PO differs from the in-screen target position PS, which is set based on depth information, and is set to indicate the display position of the subject recognized by image recognition.

[0069] The area setting unit 36 sets a focus control target area FA. The focus control target area FA is an area where autofocus control is performed. For example, if the center of the captured image is selected as the focus control target area FA, a focus control target is selected from among the subjects captured in the center. The focus control target area FA can be set to the entire captured image or to a part of the captured image. The focus control target area FA may be set in response to a user operation, or may be set automatically by automatically determining the situation through image recognition. Furthermore, a plurality of focus control target areas FA may be set, and priorities may be assigned to the plurality of focus control target areas FA.

[0070] The mode switching unit 37 switches between various modes. For example, the mode switching unit 37 switches between an AF mode for performing autofocus control and an MF (Manual Focus) mode for performing manual focus control. The switching process is executed, for example, by the user operation detection unit 31 detecting that the user has operated the operator 6 for switching between the AF mode and the MF mode. Alternatively, the switching process may be executed by detecting a menu operation by the user.

[0071] The mode switching unit 37 switches between multiple AF modes. The AF modes include a tracking focus mode in which focus control is performed according to the movement of a subject set as a focus control target, and a single focus mode in which focus control is performed only once when the shutter button 6S is half-pressed. These modes are switched in response to a menu operation by the user.

[0072] In particular, in this embodiment, the mode switching unit 37 detects a specific user operation while the AF mode is being executed and temporarily switches to the MF mode. Furthermore, it detects the end of the specific user operation in the temporary MF mode and switches back to the AF mode. That is, when the user performs a specific operation in the AF mode, the mode transitions to the MF mode, and when the specific operation is ended, the mode transitions back to the AF mode.

[0073] The specific user operation is, for example, a rotation operation of the focus ring 7. Rotating the focus ring 7 in the circumferential direction during AF mode temporarily transitions to MF mode.

[0074] The focus control unit 38 performs autofocus control according to each mode. For example, in single focus mode, the focus control unit 38 performs lens drive control to focus on a subject within a focus control target area. In tracking focus mode, the focus control unit 38 performs lens drive control to keep the focus on the focus control target according to the movement of the focus control target (particularly the movement in the optical axis direction). Furthermore, when the AF mode is switched to the temporary MF mode by the specific user operation described above, the focus control unit 38 performs a process of estimating the focus control target (subject) that has been changed in accordance with the manner in which the user manually operates the focus ring 7. Then, when the specific user operation ends and the AF mode is selected, the focus control unit 38 performs control to focus on the estimated new focus control target. For example, if tracking focus control was performed before the temporary MF mode, tracking focus control is performed on the new focus control target when the mode returns from the temporary MF mode to the AF mode. The process of estimating the new focus control target after the change may be executed after returning from the temporary MF mode to the AF mode.

[0075] To perform such focus control, the focus control unit 38 sets a target focus position. In the tracking focus mode, the target focus position is set by performing a process of fine-tuning the target focus position in accordance with the movement of the focus control target, thereby maintaining a focused state for the focus control target. The fine-tuning of the target focus position is performed based on, for example, depth information.

[0076] Furthermore, when the camera temporarily transitions to MF mode, the focus control unit 38 performs focus control based on the direction of operation (rotation direction of the focus ring 7) and amount of operation of the specific user operation detected by the user operation detection unit 31. Specifically, when the camera returns to AF mode, the focus control unit 38 calculates the direction and distance of movement of the focus position based on the direction and amount of operation of the specific operation, and estimates a new focus control target. The estimated focus control target is then selected to reset the target focus position. Specific examples of the process of estimating and selecting a new focus control target will be described later.

[0077] The display control unit 39 performs display processing such as displaying a menu screen to the user using the display unit 13, display processing for displaying captured images, and display processing for displaying warnings and the like. The display control unit 39 executes a process to display a through image. When displaying the through image, the display control unit 39 superimposes a frame image on the subject recognized by the subject recognition unit 35. The frame image may be changed depending on the focus state of the subject. For example, the frame image superimposed on a subject selected as a focus control target is a first frame image FP1 framed by a solid line, and the frame image superimposed on a subject not selected as a focus control target is a second frame image FP2 framed by a dashed line.

[0078] Specifically, an example of the first frame image FP1 and the second frame image FP2 is shown in FIG. 6 shows an image in which two people are recognized as subjects by the subject recognition unit 35. Of the two people, one is recognized as a close subject and the other is recognized as a distant subject. Also, the focus control unit 38 is in a state where a nearby subject is selected as the focus control target.

[0079] The focus control unit 38 performs focus control on the focus control target, so that a nearby subject is in focus, and a distant subject is left out of focus.

[0080] The display control unit 39 displays the first frame image FP1 superimposed on the subject selected by the focus control unit as the focus control target. Furthermore, the display control unit 39 displays the other subjects with a second frame image FP2 superimposed thereon.

[0081] Various frame images can be considered for the display control unit 39 to superimpose on the image on the display unit 13. For example, a circular or elliptical frame may be superimposed instead of a rectangular frame. Alternatively, the area in which the subject is displayed may be indicated by superimposing a rectangular frame on each of the four corners of the subject being recognized. Furthermore, the inside of the frame of the first frame image FP1 or the second frame image FP2 may be displayed with hatching. Also, in Figure 6, an example is shown in which the display mode of the subject as the focus control target and other subjects is changed depending on the line type, but they may also be distinguished by line color or line thickness, or by a combination of these. Alternatively, the display of the subject to focus control and other subjects may be separated without using a frame. For example, a depth map display may be performed in which the subject is displayed by changing the display color according to the depth amount, or a peaking display may be performed in which the subject is displayed by changing the display color according to the contrast amount within the screen.

[0082] <3. Example of selecting focus control target> As described above, in the imaging device 1, a new focus control target is selected when the imaging device 1 transitions back to the AF mode based on a user operation when the imaging device 1 temporarily transitions from the AF mode to the MF mode. Here, some examples of selection of a new focus control target will be described.

[0083] <3-1. First selection example> The first selection example is an example in which, when the user rotates the focus ring 7 upon transitioning to temporary MF mode, a new focus control target is selected depending on the rotational position of the focus ring 7 at the end of the user operation.

[0084] 7 shows an example of a through image displayed on the rear monitor 4 serving as the display unit 13. The focus control target area FA is displayed by a thick frame on the display unit 13. The display unit 13 also shows frame images of the in-screen target positions PS3, PS4, and PS5 for each of the three subjects recognized by the in-screen target position setting unit 34 based on the depth information within the frame of the focus control target area FA.

[0085] The subject located at target position PS3 on the screen is designated as subject A, which is a smiling face, the subject located at target position PS4 on the screen is designated as subject B, which is a soccer ball, and the subject located at target position PS5 on the screen is designated as subject C, which is a plant.

[0086] Here, subjects A, B, and C are located at different positions in the depth direction, that is, the distances from the imaging device 1 to each subject are different as shown in FIG. As shown in FIG. 7, the depth value of the target position PS3 within the screen is set to depth value DP3, the depth value of the target position PS4 within the screen is set to depth value DP4, and the depth value of the target position PS5 within the screen is set to depth value DP5. The magnitude relationship between depth values DP3, DP4, and DP5 is depth value DP3<depth value DP5<depth value DP4. That is, subject A is closest to image capture device 1, followed by subject C, and subject B is the farthest from image capture device 1 (see FIG. 8). As shown in FIG. 9, the depth information is calculated for each small region 41 made up of a plurality of pixels.

[0087] 7, subject A is selected as the focus control target, and a first frame image FP1, which is a solid-line frame image, is superimposed on the subject A. This indicates the area of the on-screen target position PS3 as the focus control target on the screen. Subject B is not selected as a focus control target, and a second frame image FP2, which is a dashed frame image, is superimposed on the subject B. This indicates the area of the on-screen target position PS4 as a subject that is not a focus control target on the screen. Subject C is not selected as a focus control target, and a second frame image FP2, which is a dashed frame image, is superimposed on the position of the subject C. This indicates the area of the in-screen target position PS5 as a subject that is not a focus control target on the screen.

[0088] In this way, it is assumed that subject A is selected as the focus control target before the user operates the focus ring 7 (see FIGS. 7 and 8). That is, the focus position before the user operates the focus ring 7 is the position of subject A.

[0089] Here, when the user operates the focus ring 7, the imaging device 1 transitions from the tracking focus mode to a temporary MF mode. Then, suppose that the user operates the focus ring 7 to move the focal position to a position between subject C and subject B and closer to subject C (see FIG. 8).

[0090] When the user finishes operating the focus ring 7 in this state, the imaging device 1 estimates the user's intention and selects a subject as a focus control target.

[0091] In the first selection example, subject C, which is the subject closest to the focus position after the user operation, is selected as the focus control target. As a result, on the display unit 13, the in-screen target position PS5 for subject C is indicated by a first frame image FP1, which is a frame image drawn with solid lines, and the in-screen target positions PS3 and PS4 for subjects A and B are indicated by a second frame image FP2, which is a frame image drawn with dashed lines (see FIG. 10).

[0092] In this example, the in-screen target position setting unit 34 sets an in-screen target position PS for each subject detected based on depth information, and selects a focus control target from among the in-screen target positions PS, but other examples are also possible. For example, when an in-screen target position PS set for each subject detected by the in-screen target position setting unit 34 based on the depth information matches an in-screen subject position PO set for each subject recognized by the subject recognition unit 35 through image processing, the subject may be displayed as a candidate for focus control on the display unit 13. Alternatively, the candidate for focus control may be displayed on the display unit 13 based only on the in-screen subject position PO without using the in-screen target position PS. According to this example, the subject closest to the focus position after user operation is selected as the subject to be subjected to focus control from among the subjects detected as subjects in both the detection process using depth information and the detection process using image recognition, thereby preventing focus control from being performed on an inappropriate subject.

[0093] <3-2. Second selection example> In the second selection example, a new focus control target is selected taking into consideration the movement direction of the focus position, that is, the rotation direction of the focus ring 7. Fig. 11 is a diagram showing the positional relationship between the imaging device 1 and subjects A, B, and C. Fig. 11 also shows the focus position before and after a user operation.

[0094] As shown in the figure, the second selection example, like the first selection example, shows a state in which the user operates the focus ring 7 to cause the imaging device 1 to transition from the tracking focus mode to the temporary MF mode, and the user operates the focus ring 7 to move the focus position to a position between subject C and subject B and close to subject C. In other words, the focus position at the end of the user operation (end focus position) is set to a position between subject C and subject B.

[0095] Here, in the second selection example, the subject closest to the focus position after the user operation is subject C as the focus control subject, but rather than simply selecting subject C as the focus control subject, the focus control subject is selected taking into account the rotation direction of the focus ring 7. This is synonymous with selecting the focus control subject taking into account the movement direction of the focus position.

[0096] Specifically, the imaging device 1 determines whether the difference between the focus position after the user's operation and the position of subject C is equal to or greater than a predetermined threshold. If the imaging device 1 determines that the difference is equal to or greater than the predetermined threshold, it presumes that the target of focus control intended by the user is not subject C but subject B, which is located ahead in the direction of movement of the focus position.

[0097] 12, the imaging device 1 selects the subject B as the focus control target, so that the subject B is displayed on the display unit 13 by a first frame image FP1.

[0098] As in the first selection example, the focus control target may be selected by further using the subject position PO within the screen for each subject recognized by the subject recognition unit 35 through image processing.

[0099] <3-3. Third selection example> In the third selection example, a new focus control target is selected taking into consideration the movement direction of the focus position and the focus control target area FA.

[0100] 13 shows the relationship between the focus control target area FA set in the tracking focus mode and each subject. That is, subjects A and B are located within the focus control target area FA, and subject C is located outside the focus control target area FA. Also, subject A is selected as the subject to focus control.

[0101] In this state, the user rotates the focus ring 7 to change the focus position and the state in which the focus position is changed are shown in Figure 14. As shown in the figure, subjects C and B are located ahead of the focus position at the end in the direction of movement of the focus position. Subject C is closer to the focus position than subject B.

[0102] In the state shown in FIG. 14, subject C is located ahead in the movement direction, but this subject C is positioned outside the focus control target area FA as shown in FIG. In the first and second selection examples described above, the subject is selected as the focus control subject without considering the focus control target area FA. Therefore, depending on the conditions, a subject outside the focus control target area FA may be selected as the focus control subject. Specifically, when the user rotates the focus ring 7 as shown in Figures 13 and 14, in the first and second selection examples, the subject C located outside the focus control target area FA is selected as the focus control target. In contrast to this, in the third selection example, the focus control target is selected in consideration of the focus control target area FA, resulting in the following.

[0103] That is, although subject C is located ahead in the movement direction of the focus position, it is a subject located outside the focus control target area FA, so subject B is selected as the subject for focus control.

[0104] In addition, Figure 13 shows an example in which an in-screen target position PS5 is set for a subject C outside the focus control target area FA, but the subjects for which an in-screen target position PS is set may be limited to subjects within the focus control target area FA.

[0105] As in the first selection example, the focus control target may be selected by further using the subject position PO within the screen for each subject recognized by the subject recognition unit 35 through image processing.

[0106] <3-4. Other selection examples> In another example, when a specific user operation is detected in MF mode, the camera transitions to MF mode (the temporary MF mode described above) as a preparatory step before switching to AF mode, and then transitions to AF mode upon detecting the end of the specific operation. Then, in the AF mode, focus control is performed on the new focus control target estimated in the MF mode as a preparatory step.

[0107] A specific description will be given with reference to FIG. Before a specific operation by the user is detected, the focus position is set to a position between subject A and subject C.

[0108] In this state, it is assumed that the user performs a specific operation (rotating the focus ring 7) to move the focus position to a position between subject C and subject B.

[0109] When the user has finished the specific operation, the camera shifts to AF mode, where the focus control target is determined according to the specific operation performed by the user. For example, as in the first selection example, subject C, which is closest to the final focus position, may be selected as the subject to focus control.

[0110] Alternatively, instead of selecting subject C, which is closest to the final focus position, as the focus control target as in the second selection example, subject B may be selected as the focus control target taking into account the rotation direction of the focus ring 7.

[0111] Alternatively, as in the third selection example, taking into consideration the final focus position, the rotation direction of the focus ring 7, and the focus control target area FA, a subject within the focus control target area FA and located ahead in the direction of movement of the focus position may be selected as the focus control target.

[0112] After a new focus control target is selected, tracking focus control is performed in accordance with the movement of the selected subject.

[0113] Although the example described here transitions to the tracking focus mode after the user completes a specific operation, it is also possible to automatically select a subject as the focus control target, focus on the subject, and then transition back to the MF mode. In this case, a so-called "fixed focus" state is established. In other words, the subject remains in focus as long as it does not move.

[0114] <4. Examples of display modes on the display unit> Here, an example of the display mode on the display unit 13 will be described.

[0115] 16 shows a scene in which tracking focus is being performed in AF mode. As shown in the figure, on the display unit 13, for a through image showing a subject H1 in the foreground and a subject H2 in the background, a target position PS6 within the screen corresponding to subject H1 is shown by a first frame image FP1, and a target position PS7 within the screen corresponding to subject H2 is shown by a second frame image FP2.

[0116] In this state, it is assumed that the user rotates the focus ring 7 to move the focal position to the back side (toward the subject) and then ends the operation of the focus ring 7.

[0117] At this time, the imaging device 1 temporarily transitions to MF mode while the focus ring 7 is being operated, and then returns to AF mode (in this case, tracking focus mode) again.

[0118] After returning to the tracking focus mode, the imaging device 1 selects the subject H2 as a new focus control target, depending on the relationship between the focus position and the position of the subject at the end of the rotation operation of the focus ring 7, the rotation direction of the focus ring 7, etc. As a result, as shown in Fig. 17, in the through image displayed on the display unit 13, the in-screen target position PS7 for the subject H2 selected as the new focus control target is indicated by a first frame image FP1 in solid lines, and the in-screen target position PS6 for the subject H1, which is not the focus control target, is indicated by a second frame image FP2 in dashed lines.

[0119] Thereafter, if the subject H2 moves toward the imaging device 1, focus control in the tracking focus mode is executed, and the focus position is changed in accordance with the change in the position of the subject H2. That is, even if the subject moves, the imaging device 1 continues to focus on the subject to be tracked (see FIG. 18). By operating the focus ring 7 by the user during the tracking focus mode, the subject to be tracked can be changed to a subject intended by the user while the subject remains in focus.

[0120] Other examples of the display mode of the focus state on the display unit 13 will be described with reference to FIGS.

[0121] 19 shows a through image in the tracking focus mode. The focus control target is subject H1, and a third frame image FP3 consisting of four frame images is displayed superimposed on the pupil area, indicating that the pupil of subject H1 is in focus. At this time, no frame image is superimposed on the subject H2, but the imaging device 1 has already recognized the subjects H1 and H2 based on the depth information and the results of image processing.

[0122] When the user rotates the focus ring 7, the imaging device 1 transitions from the tracking focus mode to the temporary MF mode. Figure 20 shows the state in which the focus position has moved to the back side due to the user's operation, i.e., the state in which the focus position is located between subjects H1 and H2.

[0123] As shown in the figure, the focus control target is subject H1, but the focus position is not aligned with the pupil position, so the first frame image FP1 is superimposed on subject H1. That is, the change in the display mode of the frame image allows the user to understand that the focus on the pupil is not being maintained.

[0124] When the user stops rotating the focus ring 7 in the state shown in Fig. 20, the imaging device 1 temporarily transitions from the MF mode back to the tracking focus mode. At this time, a subject to be a new focus control subject (tracking subject) is selected. In this case, subject H2 is selected as the new focus control subject, and this state is shown in Fig. 21.

[0125] As shown in the figure, a first frame image FP1 is superimposed on subject H2, which is the focus control target. This state indicates that subject H2 is the focus control target, and also indicates that the pupil position is not recognized and focus control on the pupil is not being performed.

[0126] Next, when the subject H2 moves closer to the imaging device 1, a through image as shown in FIG. In this state, the subject H2 has approached the imaging device 1, so that the pupil position of the subject H2 can be recognized, and the pupil of the subject H2 is appropriately focused. Therefore, a third frame image FP3 made up of four frame images is displayed superimposed on the pupil portion of the subject H2. Although no frame image is superimposed on the subject H1, the imaging device 1 has already recognized the subject H1 based on the depth information and the results of image processing.

[0127] <5. Processing example> An example of the flow of processing executed by the camera control unit 16 of the imaging device 1 to realize the various functions described above will be described below.

[0128] <5-1. Overall flow> The camera control unit 16 determines whether or not a specific operation has been detected in step S101 of Fig. 23. The specific operation is, for example, the above-mentioned rotation operation of the focus ring 7, which is an operation that is performed when manually changing the focus control target in AF mode, when changing the focus control target in MF mode, or when smoothly transitioning to AF mode while changing the focus control target in MF mode.

[0129] If it is determined that a specific operation has not been detected, the camera control unit 16 proceeds to step S102 and executes control according to the current mode. For example, if the current mode is a tracking focus mode, the focus position is moved according to the movement of the subject being focused on, and the subject is kept in focus. Also, if the current mode is an eye AF mode that automatically focuses on the subject's eyes, the camera control unit 16 performs control to focus on the subject's eyes by detecting the subject's eyes.

[0130] On the other hand, if it is determined that a specific operation has been detected, the camera control unit 16 proceeds to step S103 and temporarily changes the mode to MF mode. Furthermore, in step S104, the camera control unit 16 acquires the movement direction of the focus position (the rotation direction of the focus ring 7) and the movement distance of the focus position (the operation amount of the focus ring 7).

[0131] In step S105, the camera control unit 16 determines whether or not the end of the specific operation has been detected. The camera control unit 16 repeats the process of step S104 until the end of the specific operation is detected. Although not shown, until the end of the specific operation of the user is detected, the imaging device 1 drives the lens system 9 in accordance with the specific operation of the user, thereby realizing manual adjustment of the focus position by the user.

[0132] On the other hand, if it is determined that the end of the specific operation has been detected, the camera control unit 16 proceeds to step S106 and performs processing to transition to the AF mode before transitioning to the MF mode. The case where the end of the specific operation has been detected is, for example, when the state where the touch of the user's finger on the focus ring 7 is detected changes to the state where it is not detected.

[0133] In step S107, the camera control unit 16 acquires information within the focus control target area FA. Two examples of the information acquisition process within the focus control target area FA will be described. The first example is shown in Fig. 24. In step S151, the camera control unit 16 acquires depth information within the focus control target area FA. Next, in step S152, the camera control unit 16 identifies subjects that are candidates for focus control targets based on the acquired depth information.

[0134] A second example is shown in Fig. 25. The camera control unit 16 acquires depth information in step S151. Then, in step S153, the camera control unit 16 acquires information about the subject recognized by image processing. Next, in step S154, the camera control unit 16 selects the subject recognized in step S153 from among the subjects recognized based on the depth information as a candidate for focus control. That is, the example shown in Fig. 25 is an example in which a subject is selected as a candidate for focus control when the in-screen target position PS set for each subject detected by the in-screen target position setting unit 34 based on the depth information matches the in-screen subject position PO set for each subject recognized by the subject recognition unit 35 through image processing.

[0135] Returning to the explanation of FIG. After selecting or identifying a candidate focus control target, in step S108, the camera control unit 16 sets a new focus control target based on the movement direction and movement distance of the focus position obtained in step S104 during MF mode and the depth information within the focus control target area FA obtained in step S107. Note that, based on this information, it may be determined not to change the focus control target, for example, if no other suitable subjects are detected.

[0136] After executing the processing of step S108, the camera control unit 16 returns to step S101 and determines again whether or not a specific operation has been detected, and if not, control according to the current mode is executed in step S102. That is, when a new focus control object is selected in step S108, tracking focus control is performed by setting the target object distance according to the subject selected as the new focus control object in step S102 until the next specific operation is detected.

[0137] <5-2. First example of focus control target selection process> A first example of the process of selecting a new focus control target shown in step S108 of FIG. 23 will be described with reference to FIG. The first example corresponds to the first selection example of the focus control target described above.

[0138] In step S201, the camera control unit 16 identifies the subject that is closest to the focus position after the user's operation, that is, the end-time focus position, from among the subjects extracted as candidates.

[0139] Next, in step S202, the camera control unit 16 selects the identified subject as a focus control target. This realizes the selection mode described in the first selection example.

[0140] <5-3. Second example of focus control target selection process> A second example of the process of selecting a new focus control target will be described with reference to FIG. The second example corresponds to the second selection example of the focus control target described above.

[0141] In step S211, the camera control unit 16 identifies the closest subject from among subjects positioned ahead of the focus position at the end in the direction of movement of the focus position.

[0142] Next, in step S202, the camera control unit 16 selects the identified subject as a focus control target. This realizes the selection mode described in the second selection example.

[0143] <5-4. Third example of focus control target selection process> A third example of the process of selecting a new focus control target will be described with reference to FIG. The third example corresponds to the third selection example of the focus control target described above.

[0144] 23, the camera control unit 16 detects a specific operation by the user, performs a process of transitioning to a temporary MF mode, and then detects the end of the specific operation and performs a process of transitioning back to the original mode. Each process is the same as each process with the same reference numerals as described in the first example, and therefore detailed description will be omitted.

[0145] Next, instead of step S107, the camera control unit 16 acquires information about the entire area regardless of the focus control target area FA. Specifically, the camera control unit 16 identifies the subject based on depth information and / or by image processing for the entire area.

[0146] Next, in step S108, the camera control unit 16 performs a process of selecting a new focus control target from among the subjects identified in the focus control target area FA. However, if there is no candidate subject within the focus control target area FA, a new focus control target is selected from subjects outside the focus control target area FA. This can be realized by acquiring information for all areas in step S107.

[0147] For example, if the imaging device 1 cannot be moved in accordance with the movement of the subject, the subject may temporarily be positioned outside the focus control target area FA. Even in such a case, by acquiring information about the area outside the focus control target area FA, it becomes possible to select an appropriate subject as the focus control target.

[0148] Note that information about the focus control target area FA may be acquired in step S107 of Fig. 23. In this case, too, focusing is not performed on subject C (see Fig. 13) located outside the focus control target area FA, and the focus is appropriately focused on subject B within the focus control target area FA. Furthermore, by limiting the information to be acquired within the focus control target area FA, the processing load on the imaging device 1 can be reduced.

[0149] <5-5. Fourth Example of Focus Control Target Selection Process> A fourth example of the process of selecting a new focus control target, which is executed in step S108 of FIG. 23, will be described with reference to FIG. The fourth example is an example in which the first and second examples are used depending on the conditions.

[0150] In order to select a new focus control target, the camera control unit 16 identifies the subject that is closest to the focus position after the user operation in step S201. The identified subject is a subject that can be selected as the focus control target depending on the conditions.

[0151] Next, in step S222, the camera control unit 16 determines whether or not the distance (deviation) between the focus position after the user's operation and the position of the identified subject is equal to or greater than a predetermined threshold. If it is determined that the difference is less than the predetermined threshold, that is, if it is determined that the focus position after the user operation and the position of the subject are close to each other, the camera control unit 16 selects the identified subject as a new focus control target in step S202.

[0152] On the other hand, if it is determined that the distance between the focus position after the user operation and the position of the identified subject is equal to or greater than a predetermined threshold, the camera control unit 16 identifies, in step S211, the closest subject from among the subjects located ahead of the focus position at the end in the direction of movement of the focus position. Then, in step S202, the camera control unit 16 selects the identified subject as a new focus control target.

[0153] By performing such processing, it becomes possible for the user to select an intended subject as the subject for focus control.

[0154] <6. Focus movement speed control> In each of the above-described examples, it is conceivable to adjust the time and speed (focus movement speed) from when a subject is selected as a focus control target until it is actually brought into focus to match the user's operation mode.

[0155] For example, consider a case where a user slowly moves the focus ring 7 to gradually move the focus position from subject H1 to subject H2, and then ends the user operation before the focus position reaches the position of subject H2. In this case, the imaging device 1 can estimate that subject H2 is an appropriate subject for focus control, and then immediately focus on subject H2.

[0156] However, taking into consideration the manner in which the user moves the focus ring 7 slowly to move the focus position, the lens may be driven slowly to focus on the subject H2. This allows the user to specify the speed (time) until focus is achieved in focus control simply by changing the operation mode, allowing the user to achieve preferred autofocus control with simple operations, thereby improving user convenience.

[0157] An example of processing executed by the camera control unit 16 to adjust the time and speed until focusing (focus movement speed) to match the user's operation mode is shown in Fig. 29. Note that the same processes as those in Fig. 23 are denoted by the same reference numerals and descriptions thereof will be omitted as appropriate.

[0158] In step S101, the camera control unit 16 determines whether or not a specific operation has been detected. If it is determined that a specific operation has been detected, the camera control unit 16 performs a process of transitioning to temporary MF mode in step S103, and acquires information about the user's specific operation in step S111.

[0159] The information acquired in step S111 includes not only the movement direction and movement distance of the focus position but also the movement speed of the focus position in response to the user's specific operation of moving the focus position in the optical axis direction.

[0160] In step S105, the camera control unit 16 determines whether or not the end of the specific operation has been detected. The camera control unit 16 repeats the process of step S111 until the end of the specific operation has been detected.

[0161] On the other hand, if it is determined that the end of the specific operation has been detected, the camera control unit 16 proceeds to step S106 and performs processing to transition to the mode prior to transition to the MF mode.

[0162] In step S107, the camera control unit 16 acquires information within the focus control target area FA, and in step S108, selects a new focus control target.

[0163] Then, the camera control unit 16 returns to step S101 and determines again whether or not a specific operation has been detected, and if not, control according to the current mode is executed in step S112. At this time, if the movement speed of the focus position has been acquired in step S111, the camera control unit 16 performs focus control according to the mode while taking the movement speed into consideration in step S112. As a result, control is performed to bring the subject, which is the new focus control target selected in step S108, into focus at a speed according to the user's operation mode.

[0164] The focus movement speed can be set in advance by a menu operation, in addition to being changed in response to a specific user operation as described above. For example, the focus movement speed may be configured to be selectable between a setting that speeds up the focus speed, a setting that slows the focus speed, and a setting that varies in response to a specific user operation.

[0165] <7. Application Examples> The technology disclosed herein can be applied to a medical imaging system, which is a medical system that uses imaging technology, such as an endoscope system or a microscope system.

[0166] [Endoscope system] An example of an endoscopic system will be described with reference to FIGS. 30 and 31. FIG. 30 is a diagram illustrating an example of a schematic configuration of an endoscopic system 5000 to which the technology according to the present disclosure can be applied. FIG. 31 is a diagram illustrating an example of the configuration of an endoscope 5001 and a CCU (Camera Control Unit) 5039. FIG. 30 illustrates a state in which an operator (e.g., a doctor) 5067, who is a surgical participant, is performing surgery on a patient 5071 on a patient bed 5069 using the endoscopic system 5000. As shown in FIG. 30, the endoscopic system 5000 includes an endoscope 5001, which is a medical imaging device, a CCU 5039, a light source device 5043, a recording device 5053, an output device 5055, and a support device 5027 that supports the endoscope 5001.

[0167] In endoscopic surgery, an insertion aid called a trocar 5025 is inserted into a patient 5071. Then, a scope 5003 and surgical tools 5021 connected to an endoscope 5001 are inserted into the body of the patient 5071 via the trocar 5025. The surgical tools 5021 are, for example, energy devices such as an electric scalpel, forceps, etc.

[0168] A surgical image, which is a medical image showing the inside of the body of a patient 5071 photographed by an endoscope 5001, is displayed on a display device 5041. An operator 5067 performs a procedure on the surgical target using a surgical tool 5021 while viewing the surgical image displayed on the display device 5041. Note that the medical image is not limited to a surgical image, and may be a diagnostic image photographed during a diagnosis.

[0169] [Endoscopy] The endoscope 5001 is a camera that captures images of the inside of the body of a patient 5071. For example, as shown in FIG. 31 , the endoscope 5001 is a camera head including a focusing optical system 50051 that focuses incident light, a zoom optical system 50052 that changes the focal length of the camera to enable optical zoom, a focus optical system 50053 that changes the focal length of the camera to enable focus adjustment, and a light-receiving element 50054. The endoscope 5001 generates pixel signals by focusing light onto the light-receiving element 50054 via a connected scope 5003, and outputs the pixel signals to the CCU 5039 via a transmission system. The scope 5003 has an objective lens at its tip and is an insertion section that guides light from a connected light source device 5043 into the body of the patient 5071. The scope 5003 is, for example, a rigid scope in the case of a rigid endoscope, or a flexible scope in the case of a flexible endoscope. The pixel signals may be signals based on signals output from pixels, such as RAW signals or image signals. Furthermore, a configuration may be adopted in which a memory is installed in the transmission system connecting the endoscope 5001 and the CCU 5039, and parameters related to the endoscope 5001 and the CCU 5039 are stored in the memory. The memory may be disposed, for example, at a connection portion of the transmission system or on a cable. For example, parameters at the time of shipment of the endoscope 5001 and parameters that change when power is applied may be stored in the memory of the transmission system, and the operation of the endoscope may be changed based on parameters read from the memory. Furthermore, the endoscope and the transmission system may be collectively referred to as an endoscope. The light receiving element 50054 is a sensor that converts received light into pixel signals, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type imaging element. It is preferable that the light receiving element 50054 be an imaging element having a Bayer array and capable of color imaging. Furthermore, the light receiving element 50054 is preferably an imaging element having a number of pixels corresponding to a resolution of, for example, 4K (3840 horizontal pixels × 2160 vertical pixels), 8K (7680 horizontal pixels × 4320 vertical pixels), or square 4K (3840 or more horizontal pixels × 3840 or more vertical pixels). The light receiving element 50054 may be a single sensor chip or multiple sensor chips. For example, a prism may be provided to separate incident light into predetermined wavelength bands, and each wavelength band may be imaged by a different light receiving element.Furthermore, multiple light-receiving elements may be provided for stereoscopic vision. The light-receiving element 50054 may be a sensor including an arithmetic processing circuit for image processing within its chip structure, or may be a ToF (Time of Flight) sensor. The transmission system may be, for example, an optical fiber cable or wireless transmission. Wireless transmission may be performed via any means capable of transmitting pixel signals generated by the endoscope 5001. For example, the endoscope 5001 and the CCU 5039 may be connected wirelessly, or the endoscope 5001 and the CCU 5039 may be connected via a base station in the operating room. In this case, the endoscope 5001 may simultaneously transmit not only the pixel signals but also information related to the pixel signals (e.g., the processing priority of the pixel signals, a synchronization signal, etc.). The endoscope may be configured such that the scope and camera head are integrated, or a light-receiving element is provided at the tip of the scope.

[0170] [CCU (Camera Control Unit)] The CCU 5039 is a control device that comprehensively controls the connected endoscope 5001 and light source device 5043, and is, for example, an information processing device having an FPGA 50391, a CPU 50392, a RAM 50393, a ROM 50394, a GPU 50395, and an I / F 50396, as shown in FIG. 31 . The CCU 5039 may also comprehensively control the connected display device 5041, recording device 5053, and output device 5055. For example, the CCU 5039 controls the irradiation timing, irradiation intensity, and type of irradiation light source of the light source device 5043. The CCU 5039 also performs image processing such as development processing (e.g., demosaic processing) and correction processing on pixel signals output from the endoscope 5001, and outputs the processed pixel signals (e.g., images) to an external device such as the display device 5041. The CCU 5039 also transmits control signals to the endoscope 5001 to control the driving of the endoscope 5001. The control signal is, for example, information regarding imaging conditions such as the magnification and focal length of the camera. The CCU 5039 may have an image down-conversion function and may be configured to be able to simultaneously output a high-resolution (e.g., 4K) image to the display device 5041 and a low-resolution (e.g., HD) image to the recording device 5053.

[0171] The CCU 5039 may also be connected to an external device via an IP converter that converts signals into a predetermined communication protocol (e.g., IP (Internet Protocol)). The connection between the IP converter and the external device may be configured as a wired network, or a part or all of the network may be configured as a wireless network. For example, the IP converter on the CCU 5039 side may have a wireless communication function, and may transmit the received video to an IP switcher or an output-side IP converter via a wireless communication network such as a fifth-generation mobile communication system (5G) or a sixth-generation mobile communication system (6G).

[0172] [Light source device] The light source device 5043 is a device capable of emitting light in a predetermined wavelength band and includes, for example, multiple light sources and a light source optical system that guides the light from the multiple light sources. The light sources are, for example, a xenon lamp, an LED light source, or an LD light source. The light source device 5043 has, for example, LED light sources corresponding to the three primary colors R, G, and B, and emits white light by controlling the output intensity and output timing of each light source. The light source device 5043 may also include a light source that can emit special light used for special light observation, in addition to a light source that emits normal light used for normal light observation. The special light is light in a predetermined wavelength band different from the normal light used for normal light observation, such as near-infrared light (light with a wavelength of 760 nm or more), infrared light, blue light, or ultraviolet light. The normal light is, for example, white light or green light. Narrowband light observation, which is a type of special light observation, alternately emits blue light and green light, allowing high-contrast imaging of specific tissues, such as blood vessels on the surface of mucous membranes, by utilizing the wavelength-dependence of light absorption in body tissue. Furthermore, in fluorescence observation, which is a type of special light observation, excitation light that excites a reagent injected into body tissue is irradiated, and fluorescence emitted by the body tissue or the reagent is received to obtain a fluorescence image, thereby making it easier for the surgeon to visualize body tissue that is difficult for the surgeon to visualize under normal light. For example, in infrared observation using infrared light, near-infrared light is irradiated as excitation light that excites a reagent such as indocyanine green (ICG) injected into body tissue, making it easier to visualize structures deep within the body tissue. In addition, in fluorescence observation, a reagent (e.g., PDD or 5-ALA) that is excited by special light in the blue wavelength band and emits fluorescence in the red wavelength band may be used. The type of irradiated light for the light source device 5043 is set under the control of the CCU 5039. The CCU 5039 may have a mode in which normal light observation and special light observation are alternately performed by controlling the light source device 5043 and the endoscope 5001. In this case, it is preferable that information based on pixel signals obtained under special light observation be superimposed on pixel signals obtained under normal light observation.

[0173] [Recording Device] The recording device 5053 is a device, such as a recorder, that records pixels acquired from the CCU 5039. The recording device 5053 records images acquired from the CCU 5039 on a HDD, an SDD, or an optical disk. The recording device 5053 may be connected to a network within the hospital so as to be accessible from devices outside the operating room. The storage device 5053 may also have an image down-conversion or up-conversion function.

[0174] [Display device] The display device 5041 is a device capable of displaying an image, such as a display monitor. Under the control of the CCU 5039, the display device 5041 displays an image based on pixel signals that have been subjected to image processing by the CCU 5039. The display device 5041 may also function as an input device that enables gaze recognition, voice recognition, and instruction input using gestures by including a camera and a microphone.

[0175] [Output device] The output device 5055 is a device, such as a printer, that outputs information acquired from the CCU 5039. The output device 5055 prints, for example, a print image based on the pixel signal acquired from the CCU 5039 onto paper.

[0176] [Support device] The support device 5027 is an articulated arm including a base 5029 having an arm control device 5045, an arm 5031 extending from the base 5029, and a holder 5032 attached to the tip of the arm 5031. The arm control device 5045 is configured with a processor such as a CPU and controls the drive of the arm 5031 by operating according to a predetermined program. The support device 5027 controls the position and posture of the endoscope 5001 held by the holder 5032, for example, by controlling parameters such as the length of each link 5035 constituting the arm 5031 and the rotation angle and torque of each joint 5033 using the arm control device 5045. This allows the endoscope 5001 to be changed to a desired position or posture, allowing the scope 5003 to be inserted into the patient 5071 and the observation area within the body to be changed. The support device 5027 functions as an endoscope support arm that supports the endoscope 5001 during surgery. This allows the support device 5027 to take the place of an assistant scopist who holds the endoscope 5001. The support device 5027 may also be a device that supports a microscope device 5301, which will be described later, and may also be called a medical support arm. The control of the support device 5027 may be an autonomous control method by the arm control device 5045, or a control method in which the arm control device 5045 controls the support device 5027 based on user input. For example, the control method may be a master-slave method in which the support device 5027 as a slave device is controlled based on the movement of a master device in the user's hand. The support device 5027 may also be remotely controlled from outside the operating room.

[0177] The above describes an example of an endoscope system 5000 to which the technology according to the present disclosure can be applied. For example, the technology according to the present disclosure may be applied to a microscope system.

[0178] [Microscope system] 32 is a diagram showing an example of a schematic configuration of a microsurgical system to which the technology according to the present disclosure can be applied. In the following description, components similar to those in the endoscope system 5000 are assigned the same reference numerals, and redundant description thereof will be omitted.

[0179] 32 shows a schematic diagram of an operator 5067 performing surgery on a patient 5071 on a patient bed 5069 using a microsurgical system 5300. For simplicity, the illustration of the cart 5037 of the microsurgical system 5300 is omitted, and a microscope device 5301 that replaces the endoscope 5001 is simplified. However, the microscope device 5301 in this description may refer to the microscope unit 5303 provided at the tip of the link 5035, or may refer to the entire configuration including the microscope unit 5303 and the support device 5027.

[0180] 32, during surgery, a microsurgery system 5300 is used, and an image of the surgical site captured by a microscope device 5301 is enlarged and displayed on a display device 5041 installed in an operating room. The display device 5041 is installed in a position facing the surgeon 5067, and the surgeon 5067 performs various procedures on the surgical site, such as resecting the affected area, while observing the state of the surgical site using the image displayed on the display device 5041.

[0181] The above describes examples of an endoscopic system 5000 and a microsurgery system 5300 to which the technology of the present disclosure can be applied. Note that the systems to which the technology of the present disclosure can be applied are not limited to these examples. For example, the support device 5027 may support another observation device or another surgical tool at its tip instead of the endoscope 5001 or the microscope unit 5303. Examples of such other observation devices include forceps, a surgeon, an insufflation tube for insufflation, or an energy treatment tool for incising tissue or sealing blood vessels by cauterization. Supporting these observation devices and surgical tools with a support device allows them to be more stably fixed in position than when medical staff support them manually, and also reduces the burden on medical staff. The technology of the present disclosure may be applied to support devices that support components other than the microscope unit.

[0182] The technology according to the present disclosure can be suitably applied to the endoscope system 5000, the microscope device 5301, and the like, among the configurations described above. Specifically, the subject intended by the surgeon is estimated based on the moving direction and moving distance of the focus position and selected as the focus control target. This allows the focus position to be adjusted more quickly and accurately than when the focus position is manually adjusted to the focus control target, thereby shortening the surgery time and improving efficiency, and reducing the burden on the patient.

[0183] <8. Summary> As described above, the imaging device 1 includes a user operation detection unit 31 that detects a user operation for focus control (e.g., a rotation operation on the focus ring 7), a focus position movement detection unit 32 that detects a movement (change) of the focus position (the position in the optical axis direction of the subject in focus) based on the user operation detected by the user operation detection unit 31, a depth information detection unit 33 that detects depth information within the focus control target area FA, and a focus control unit 38 that sets a target focus position based on the movement of the focus position and the depth information. That is, a change in the focus position due to a user operation is detected, and after the user operation, a target focus position is set and focus control is performed that reflects the user operation. For example, if a user operates the camera in AF mode, the camera temporarily switches to MF mode to detect the movement of the focus position by the user. This detection process is, so to speak, a process for estimating the focus control target (subject) intended by the user, and also a process for collecting information for selecting the subject as the focus control target in subsequent AF control. The camera control unit 16 of the imaging device 1 detects manual operation of the focus ring 7 by the user, and by detecting the direction of operation (direction of movement of the focus position) and amount of operation (amount of movement of the focus position), it infers the user's intention, and after switching to AF mode, performs focus control based on this inference, thereby enabling optimal focus control using MF mode and AF mode. Furthermore, when using only AF mode, there are many situations where the focus is set on a subject other than the one intended by the user. By using the imaging device 1 having this configuration, the user can specify the intended subject via temporary MF mode while the AF mode is in operation, thereby achieving AF control as desired. In particular, the ability to seamlessly switch between AF mode and MF mode, and the use of information acquired in one mode in the transition to the other mode, allow for such highly convenient control to be achieved. In the above examples, the focus operation has been described with a focus on rotating the focus ring 7. However, as described above, the same processing can be performed even when the focus operation is performed by remote control using an external operating device, and as a result, the same effect can be obtained. In this case, usability in remote control can be improved, and it is particularly suitable for a case where a user remotely controls a fixedly placed imaging device 1 to focus on an intended subject, and since there is no need to touch the imaging device 1, it is possible to eliminate blurring during focus operation. In other words, the configuration of such an imaging device 1 includes a user operation detection unit 31 that detects user operations for focus control, a focus position movement detection unit 32 that detects movement of the focus position due to the user operations detected by the user operation detection unit 31, a depth information detection unit 33 that detects depth information within the focus control target area FA for focus control, and a focus control unit 38 that sets a target focus position based on the depth information when the user operation detection unit 31 detects the end of the movement of the focus position.

[0184] As explained in the sections on the functional configuration of the imaging device 1 (FIGS. 4, 8, 11, and 14), the focus position movement detection unit 32 in the imaging device 1 may detect the movement direction of the focus position. By detecting the direction of movement of the focus position, the user's intention can be estimated more accurately, thereby realizing easy-to-use autofocus control.

[0185] As explained in the section on the functional configuration of the imaging device 1 (FIG. 4), the focus position movement detection unit 32 of the imaging device 1 may detect the movement direction every predetermined time (for example, every frame). By detecting the movement direction at predetermined time intervals, the operation direction of the user can be detected appropriately. Therefore, it is possible to appropriately estimate the focus control target intended by the user.

[0186] As explained in the sections on the functional configuration of the imaging device 1 (Figures 4, 8, and 9), the depth information detection unit 33 of the imaging device 1 detects depth information of the entire focus control target area FA, and the focus control unit 38 may set the target focus position based on the depth information of the entire focus control target area FA, the direction of movement of the focus position, and the end focus position (focus position at the end of movement) when it is detected that the user's operation to move the focus position has ended. By setting the target focus position based on the end focus position, the user's intention is reflected in the focus control. Furthermore, even if a mode change is made such as transitioning from the tracking focus mode to a temporary MF mode and then returning to the tracking focus mode, the information on the user's operation acquired in the MF mode is utilized in the tracking focus mode, thereby enabling tracking focus control in line with the user's intentions.

[0187] As explained in the section on the functional configuration of the imaging device 1 (Figure 5), the imaging device 1 is provided with an in-screen target position setting unit 34 that sets an in-screen target position PS based on depth information, and the focus control unit 38 may set a focus position corresponding to the in-screen target position PS as the target focus position. The in-screen target position PS is a region set on the captured image, which is a region consisting of a group of pixels with similar depth information. For example, a pixel region containing a certain subject is set as the in-screen target position PS because the depth information for each pixel is similar. This allows appropriate focus control to be performed on the subject recognized based on the depth information. In other words, the configuration of such an imaging device 1 includes an area setting unit (in-screen target position setting unit 34) that sets an area in an image where a focus candidate object is located as a focus candidate area (in-screen target position) based on depth information, and a focus control unit 38 that sets a target focus position so as to focus on the focus candidate object in the focus candidate area set by the area setting unit.

[0188] As explained in the first selection example (FIGS. 7 to 10, FIG. 26), when the in-screen target position setting unit 34 sets multiple in-screen target positions PS, the focus control unit 38 of the imaging device 1 may set the focus position closest to the end focus position among the focus positions corresponding to the in-screen target positions PS as the target focus position. As a result, when the user moves the focus position to the vicinity of the target subject, focus control is performed on the target subject. In other words, the user can perform autofocus control for the intended subject without having to move the focus ring 7 to accurately focus on the target subject, thereby improving convenience. In other words, the configuration of the imaging device 1 is such that, when the area setting unit (target position setting unit 34 within the screen) sets multiple focus candidate areas (target positions within the screen), the focus control unit 38 sets the target focus position so as to focus on the focus candidate object that is closest to the end focus position in the optical axis direction.

[0189] As explained in the first selection example and the second example of the information acquisition process within the focus control target area FA (Figure 25), the imaging device 1 is provided with a subject recognition unit 35 that recognizes the subject, and when the focus control unit 38 determines that the target position PS within the screen matches the subject position within the screen, which is the position within the screen of the subject recognized by the subject recognition unit 35, the focus control unit 38 may set the focus position corresponding to the subject position within the screen as the target focus position. As a result, for example, an image recognition process is used to select a subject to be subjected to focus control. Therefore, it is possible to increase the possibility that an appropriate subject will be selected as the focus control target, and it is possible to perform appropriate focus control. In particular, when a category such as a person, cat, or car is specified as the subject to be detected, appropriate subjects belonging to the specified subject category are selected as candidates for focus control, thereby reducing the possibility of focusing on an unintended subject. In other words, the configuration of such an imaging device 1 includes a subject recognition unit 35 that recognizes the subject based on the captured image, and a focus control unit 38 that sets a target focus position so as to focus on a focus candidate object in a focus candidate area (target position within the screen) that coincides with the position of the subject in the image.

[0190] As explained in the first selection example, when the subject recognition unit 35 recognizes multiple subjects, the focus control unit 38 of the imaging device 1 may set the focus position closest to the end focus position among the focus positions corresponding to the subject positions within the screen as the target focus position. This allows an appropriate subject to be selected as a focus control target, and also allows a subject that reflects the user's intention to be selected as a focus control target. In other words, when the subject recognition unit 35 recognizes multiple subjects, the focus control unit 38 sets the target focus position so as to focus on the subject that is closest to the final focus position in the optical axis direction.

[0191] As described in the section on the functional configuration of the imaging device 1 (FIG. 4) and in FIGS. 20 to 22, the focus control unit 38 of the imaging device 1 may perform tracking focus control on the subject that has been focused according to the target focus position that has been set after the movement of the focus position has been completed. This eliminates the need for the user to continue focusing on a subject that has already been manually focused on by selecting the focus control target in the tracking focus mode in consideration of the user's operation of the focus ring 7. By performing such control, convenience can be improved.

[0192] As explained in the sections on the functional configuration of the imaging device 1 (Figures 4, 7, 12, and 13), the imaging device 1 may be provided with an area setting unit 36 that sets the focus control target area FA in accordance with user operation. This makes it possible to select a focus control target from among the subjects in a manner that better reflects the user's intentions. Therefore, it is possible to perform focus control that is appropriate for the user.

[0193] As explained in the sections on the functional configuration of the imaging device 1 (Figures 4, 6, 7, 10, 12, 13, 16 to 18), the imaging device 1 may be provided with a display control unit 39 that performs a process of superimposing a first frame image FP1 on a subject that is focused by the focus control unit 38 among the subjects recognized by the subject recognition unit 35, and a process of superimposing a second frame image FP2 on other subjects. This allows the user to check the display unit 13 to confirm the focus control target. Therefore, convenience can be improved.

[0194] As described in the section on focus movement speed control (FIG. 29), the focus control unit 38 of the imaging device 1 may perform focus control in accordance with the set focus movement speed. This makes it possible to specify the speed (time) required for focus control to be achieved simply by changing the manner of operation. Therefore, preferred autofocus control can be achieved with a simple operation, improving user convenience.

[0195] As explained in the section on the configuration of the imaging device 1 (Figure 3), the user operation detection unit 31 of the imaging device 1 may detect the user's operation on the focus ring 7, and the focus position movement detection unit 32 may detect the movement direction of the focus position based on the operation direction of the focus ring 7. This allows the movement direction of the focus position to be detected appropriately. Therefore, it is possible to perform appropriate focus control based on the user's operation on the focus ring 7.

[0196] As explained in the section on the functional configuration of the imaging device 1 (FIG. 4), the focus control unit 38 of the imaging device 1 is capable of switching between a first mode (AF mode) that performs autofocus control on a subject and a second mode (MF mode) that performs manual focus control, and may switch to the second mode when the user operation detection unit 31 detects an operation on the focus ring 7 in the first mode. This allows the user to simply operate the focus ring 7 when switching to MF mode. That is, mode switching can be performed with a simple operation, thereby improving convenience.

[0197] As explained in the section on the functional configuration of the imaging device 1 (Figure 4), the user operation detection unit 31 of the imaging device 1 may determine that the end of movement of the focus position has been detected when a change occurs from a state in which an operation on the focus ring 7 is detected as a user operation to a state in which an operation on the focus ring 7 is not detected. This makes it possible to detect the end of the user's operation on the focus ring 7 using, for example, a sensor or the like. Therefore, it is possible to appropriately determine the time when the user's operation has ended, and to appropriately perform subsequent focus control.

[0198] As explained in the fourth example of the selection process for the focus control target (FIG. 28), the focus control unit 38 of the imaging device 1 is capable of executing a first control (selection mode shown in the first selection example) that sets the focus position closest to the end focus position among the focus positions corresponding to the target position PS within the screen as the target focus position, and a second control (selection mode shown in the second selection example) that sets the target focus position depending on the focus position at the end of the movement and the movement direction, and may switch from the first control to the second control when the deviation between the closest focus position and the end focus position is equal to or greater than a predetermined threshold. This allows the subject to be selected for focus control according to the user's operation mode and the position of the subject. Therefore, focus control can be performed on the subject that is the focus control target, appropriately reflecting the user's intention. In other words, the configuration of such an imaging device 1 includes an area setting unit (in-screen target position setting unit 34) that sets an area in an image where a focus candidate object is located as a focus candidate area (in-screen target position) based on depth information, a focus position movement detection unit 32 that detects the movement direction of the focus position, and a focus control unit 38 that is capable of executing a first control that sets a target focus position so as to focus on a focus candidate object that is closest in the optical axis direction to the focus position at the end of the movement (end focus position), and a second control that sets the target focus position according to the focus position at the end of the movement and the movement direction, and switches from the first control to the second control when the deviation between the target focus position in the first control and the focus position at the end of the movement is equal to or greater than a predetermined threshold.

[0199] The program executed by the imaging device 1 is a program that causes, for example, a CPU, a DSP, or a device including these to execute the processes shown in each of FIGS. In other words, this program causes the imaging device 1 or the like to execute the following processes: detecting a user operation for focus control; detecting movement of the focus position based on the detected user operation; detecting depth information within the focus control target area FA; and setting a target focus position based on the movement of the focus position and the depth information. The imaging device 1 described above can be realized by such a program.

[0200] A program for realizing such an imaging device 1 can be recorded in advance on a HDD serving as a recording medium built into a device such as the imaging device 1, or on a ROM or the like in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magnet Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0201] Furthermore, such a program is suitable for widespread provision of the imaging device 1 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet equipped with a camera function, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the imaging device 1 of the present disclosure.

[0202] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0203] <9. This Technology> The present technology can also be configured as follows. (1) a user operation detection unit that detects a user operation for focus control; a focus position movement detection unit that detects movement of a focus position based on the user operation detected by the user operation detection unit; a depth information detection unit that detects depth information within a focus control target area; a focus control unit that sets a target focus position based on the movement of the focus position and the depth information. Imaging device. (2) The focus position movement detection unit detects the movement direction of the focus position. The imaging device according to (1) above. (3) The focus position movement detection unit detects the movement direction at predetermined time intervals. The imaging device according to (2) above. (4) the depth information detection unit detects depth information of the entire focus control target area; The focus control unit sets the target focus position based on depth information of the entire focus control target area, the movement direction of the focus position, and an end focus position when it is detected that the user's operation for moving the focus position has ended. The imaging device according to any one of (2) to (3) above. (5) an in-screen target position setting unit that sets an in-screen target position based on the depth information; The focus control unit sets a focus position corresponding to the target position within the screen as the target focus position. The imaging device according to (4) above. (6) When the in-screen target position setting unit sets a plurality of the in-screen target positions, the focus control unit sets, as the target focus position, a focus position that is closest to the end focus position among the focus positions corresponding to the in-screen target positions. The imaging device according to (5) above. (7) An object recognition unit is provided to recognize an object, When it is determined that the target position within the screen coincides with an object position within the screen, which is the position of the object recognized by the object recognition unit within the screen, the focus control unit sets a focus position corresponding to the object position within the screen as the target focus position. The imaging device according to any one of (5) to (6) above. (8) When the subject recognition unit recognizes a plurality of the subjects, the focus control unit sets, as the target focus position, a focus position that is closest to the end focus position among focus positions corresponding to the subject positions within the screen. The imaging device according to (7) above. (9) The focus control unit performs tracking focus control on the subject focused according to the target focus position set after the movement of the focus position is completed. The imaging device according to any one of (7) to (8) above. (10) and a region setting unit that sets the focus control target region in response to a user operation. The imaging device according to any one of (1) to (9) above. (11) a display control unit that performs a process of superimposing a first frame image on a subject focused by the focus control unit among the subjects recognized by the subject recognition unit, and a process of superimposing a second frame image on other subjects; The imaging device according to any one of (7) to (9) above. (12) The focus control unit performs focus control in accordance with a set focus movement speed. The imaging device according to any one of (1) to (11) above. (13) the user operation detection unit detects a user operation on a focus ring, The focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus ring. The imaging device according to any one of (2) to (9) above. (14) the user operation detection unit detects a user's focus operation by receiving operation information transmitted from an external operation device; The focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus operation. The imaging device according to any one of (2) to (9) above. (15) The focus control unit The camera is capable of switching between a first mode that performs autofocus control on a subject and a second mode that performs manual focus control, When the user operation detection unit detects an operation on the focus ring in the first mode, the mode is switched to the second mode. The imaging device according to any one of (13) to (14) above. (16) The user operation detection unit determines that the end of movement of the focus position has been detected when a state in which an operation on the focus ring is detected as the user operation changes from a state in which an operation on the focus ring is detected to a state in which an operation on the focus ring is not detected. The imaging device according to any one of (13) to (15) above. (17) The focus control unit a first control for setting a focus position closest to the end focus position among focus positions corresponding to the target position within the screen as the target focus position, and a second control for setting the target focus position in accordance with the focus position at the end of the movement and the movement direction, When the deviation between the closest focus position and the end focus position is equal to or greater than a predetermined threshold, the first control is switched to the second control. The imaging device according to any one of (5) to (9) above. (18) Detecting a user operation on a focus control; Detecting a movement of the focus position based on the detected user operation; Detecting depth information within a focus control target area; A target focus position is set based on the movement of the focus position and the depth information. How to set the focus position. (19) detecting a user operation on a focus control; a process of detecting movement of a focus position based on the detected user operation; A process of detecting depth information within a focus control target area; A process of setting a target focus position based on the movement of the focus position and the depth information. A program executed by the imaging device. [Explanation of symbols]

[0204] 1. Imaging device 7 Focus ring 31 User operation detection unit 32 Focus position movement detection unit 33 Depth information detection unit 34 In-screen target position setting section 35 Subject recognition section 36 Area setting section 38 Focus control section PS, PS1, PS2, PS3, PS4, PS5, PS6, PS7 Target position on screen PO Subject position in screen FP1 1st frame image FP2 2nd frame image

Claims

1. a user operation detection unit that detects a user operation for focus control; a focus position movement detection unit that detects movement of a focus position based on the user operation detected by the user operation detection unit; a depth information detection unit that detects depth information within a focus control target area; a focus control unit that sets a target focus position based on the movement of the focus position and the depth information. Imaging device.

2. The focus position movement detection unit detects the movement direction of the focus position. The imaging device according to claim 1 .

3. The focus position movement detection unit detects the movement direction at predetermined time intervals. The imaging device according to claim 2 .

4. the depth information detection unit detects depth information of the entire focus control target area; The focus control unit sets the target focus position based on depth information of the entire focus control target area, the movement direction of the focus position, and an end focus position when it is detected that the user's operation for moving the focus position has ended. The imaging device according to claim 2 .

5. an in-screen target position setting unit that sets an in-screen target position based on the depth information; The focus control unit sets a focus position corresponding to the target position within the screen as the target focus position. The imaging device according to claim 4 .

6. When the in-screen target position setting unit sets a plurality of the in-screen target positions, the focus control unit sets, as the target focus position, a focus position that is closest to the end focus position among the focus positions corresponding to the in-screen target positions. The imaging device according to claim 5 .

7. An object recognition unit is provided to recognize an object, When it is determined that the target position within the screen coincides with an object position within the screen, which is the position of the object recognized by the object recognition unit within the screen, the focus control unit sets a focus position corresponding to the object position within the screen as the target focus position. The imaging device according to claim 5 .

8. When the subject recognition unit recognizes a plurality of the subjects, the focus control unit sets, as the target focus position, a focus position that is closest to the end focus position among focus positions corresponding to the subject positions within the screen. The imaging device according to claim 7 .

9. The focus control unit performs tracking focus control on the subject focused according to the target focus position set after the movement of the focus position is completed. The imaging device according to claim 7 .

10. and a region setting unit that sets the focus control target region in response to a user operation. The imaging device according to claim 1 .

11. a display control unit that performs a process of superimposing a first frame image on a subject focused by the focus control unit among the subjects recognized by the subject recognition unit, and a process of superimposing a second frame image on other subjects; The imaging device according to claim 7 .

12. The focus control unit performs focus control in accordance with a set focus movement speed. The imaging device according to claim 1 .

13. the user operation detection unit detects a user operation on a focus ring, The focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus ring. The imaging device according to claim 2 .

14. the user operation detection unit detects a user's focus operation by receiving operation information transmitted from an external operation device; The focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus operation. The imaging device according to claim 2 .

15. The focus control unit The camera is capable of switching between a first mode for performing autofocus control on a subject and a second mode for performing manual focus control, When the user operation detection unit detects an operation on the focus ring in the first mode, the mode is switched to the second mode. The imaging device according to claim 13.

16. The user operation detection unit determines that the end of movement of the focus position has been detected when a state in which an operation on the focus ring is detected as the user operation changes from a state in which an operation on the focus ring is detected to a state in which an operation on the focus ring is not detected. The imaging device according to claim 13.

17. The focus control unit a first control for setting a focus position closest to the end focus position among focus positions corresponding to the target position within the screen as the target focus position, and a second control for setting the target focus position in accordance with the focus position at the end of the movement and the movement direction, When the deviation between the closest focus position and the end focus position is equal to or greater than a predetermined threshold, the first control is switched to the second control. The imaging device according to claim 5 .

18. Detecting a user operation on a focus control; Detecting a movement of the focus position based on the detected user operation; Detecting depth information within a focus control target area; A target focus position is set based on the movement of the focus position and the depth information. How to set the focus position.

19. detecting a user operation on a focus control; a process of detecting movement of a focus position based on the detected user operation; A process of detecting depth information within a focus control target area; A process of setting a target focus position based on the movement of the focus position and the depth information. A program executed by the imaging device.

Images