Focus control device, focus control method, and imaging device

Abstract

According to the present invention, reduction of cost and power consumption and miniaturization can be achieved, and focus control can be carried out. In the present invention, machine learning is executed, and a depth map is generated. A control unit 134b carries out focus control by using the depth map generated by a depth calculation unit 134, and switches control so as to be different between a tracking mode for tracking a prescribed object and an untracking mode for not tracking the object. For example, in the tracking mode, the focus control focusing on the prescribed object searched for on the basis of distance information of the object generated by using the depth map is carried out. In the untracking mode, the focus control focusing on an object at a nearest position indicated by the distance information of the object generated by using the depth map is carried out. Even without providing a ranging sensor, the focus control can be carried out in accordance with the tracking mode or the untracking mode by using the depth map generated on the basis of a captured image.

Description

[Technical Field]

[0001] This technology relates to a focus control device, a focus control method, and an imaging device, and enables focus control to be performed at low cost. [Background technology]

[0002] Various focus control techniques have been proposed to automatically focus on a subject. For example, Patent Document 1 discloses a technique for performing focus control using both the distance measurement result of a distance measurement sensor and an evaluation value calculated at a predetermined interval using high-frequency components in a specific region of an image signal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-009341 Summary of the Invention [Problem to be solved by the invention]

[0004] However, using a distance measuring sensor increases costs and power consumption, and also requires space to install the distance measuring sensor, making it difficult to miniaturize the device.

[0005] Therefore, an object of this technology is to provide a focus control device, a focus control method, a program, and an imaging device that can reduce costs, power consumption, and size. [Means for solving the problem]

[0006] The first aspect of this technology is A control unit that performs focus control using a depth map generated from a captured image including a predetermined subject, and switches the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked. The focus control device includes:

[0007] In this technology, the control unit performs focus control using a depth map generated by machine learning using a captured image including a predetermined subject. When in a tracking mode in which the subject is tracked, the control unit performs focus control to focus on the predetermined subject searched for based on subject distance information generated using the depth map. The subject distance information is generated using depth values ​​indicated in the depth map and the depth values ​​of the predetermined subject. The control unit may also generate the subject distance information using an adjustment coefficient that adjusts the distance relationship between the depth values ​​indicated in the depth map and the depth values ​​of the predetermined subject, specifically, an adjustment coefficient that adjusts the distance difference corresponding to the depth difference between the depth values ​​indicated in the depth map and the depth values ​​of the subject. The adjustment coefficient is set, for example, according to the composition of the captured image. Furthermore, the control unit may perform filtering in the time direction to generate the subject distance information. In the filtering in the time direction, filtering is performed on the depth map, for example.

[0008] In addition, when the control unit is in a non-tracking mode in which a specified subject is not tracked, it performs focus control to focus on the subject at the closest position indicated by the subject distance information generated using the depth map, for example, the subject at the closest position within a search range set in the captured image.

[0009] The second aspect of this technology is A control unit performs focus control by using a depth map generated from a captured image including a predetermined object, and switches the focus control between a tracking mode in which the object is tracked and a non-tracking mode in which the object is not tracked. The focus control method includes:

[0010] The third aspect of this technology is A program for causing a computer to execute focus control of an imaging device, A step of performing focus control using a depth map generated from a captured image including a predetermined subject; a step of switching the focus control to different controls between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked; The program is executed by the computer.

[0011] The program of the present technology is, for example, a program that can be provided in a computer-readable format to a general-purpose computer capable of executing various program codes via a storage medium or communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer.

[0012] The fourth aspect of this technology is an imaging unit that generates a captured image; a control unit that generates a depth map from an image captured by the imaging unit, the image capturing unit includes a predetermined subject, performs focus control using the depth map, and switches the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked; and The imaging device includes: [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an imaging device. [Figure 2] FIG. 2 is a diagram illustrating a configuration of an image processing unit. [Figure 3] 10 is a flowchart illustrating a focus control operation in the imaging apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a captured image obtained by an imaging device. [Figure 5] FIG. 10 is a diagram illustrating an example of a depth map. [Figure 6] FIG. 10 is a diagram illustrating a focus operation. [Figure 7] FIG. 10 is a diagram showing subject distance information when the adjustment coefficient c is set to "1." [Figure 8]FIG. 10 is a diagram showing subject distance information when the adjustment coefficient c is set to "2." [Figure 9] FIG. 10 is a diagram showing subject distance information when the adjustment coefficient c is set to "0.5." [Figure 10] FIG. 10 is a diagram showing subject distance information when IIR filtering is performed. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present technology will be described. The explanation will be given in the following order. 1. Configuration of the imaging device 2. Operation of the imaging device 2-1. Operation of the imaging device 2-2. Example of operation 2-3.Other actions 3. Application Examples

[0015] <1. Configuration of the imaging device> 1 illustrates an example of the configuration of an imaging device using the focus control device of the present technology. The imaging device 10 includes an imaging optical system 11, an imaging unit 12, an image processing unit 13, a display unit 14, a recording unit 15, an operation unit 16, a system control unit 17, and a bus 18.

[0016] The imaging optical system 11 is configured using a focus lens, a zoom lens, etc. The imaging optical system 11 drives the focus lens, the zoom lens, etc. based on control signals from the image processing unit 13 and the system control unit 17, and forms an optical image of a subject on the imaging surface of the imaging unit 12. The imaging optical system 11 may also be provided with an iris (aperture) mechanism, a shutter mechanism, etc.

[0017] The imaging unit 12 is configured using imaging elements such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The imaging unit 12 performs photoelectric conversion to generate an analog imaging signal corresponding to an optical image of a subject. The imaging unit 12 also performs CDS (Correlated Double Sampling) processing, AGC (Auto Gain Control) processing, and A / D conversion processing on the analog imaging signal, and outputs the result to the image processing unit 13 as a digital image signal.

[0018] 2 shows an example of the configuration of the image processing unit 13. The image processing unit 13 includes a development processing unit 131, a post-processing unit 132, a detection processing unit 133, and a focus control unit .

[0019] The development processing unit 131 performs clamping, defect correction, demosaic processing, etc. on the digital image signal supplied from the imaging unit 12. The development processing unit 131 performs clamping to set the signal level of a pixel in a black state where no light is present to a predetermined level in the image signal generated by the imaging unit 12. The development processing unit 131 also performs defect correction to correct the pixel signal of a defective pixel using, for example, pixel signals of surrounding pixels. Furthermore, the development processing unit 131 performs demosaic processing to generate image signals in which one pixel represents each color component (for example, a red component, a green component, and a blue component) from the image signal generated by the imaging unit 12 in which one pixel represents one color component. The development processing unit 131 outputs the image signal after development processing to the post-processing unit 132, the detection processing unit 133, and the focus control unit 134.

[0020] The post-processing unit 132 performs noise removal, white balance adjustment, gamma adjustment, distortion correction, camera shake correction, high-quality image processing, image enlargement / reduction processing, encoding processing, etc. according to user settings or as needed, and generates an image signal suitable for image display, recording, etc. Note that the white balance adjustment is performed using a white balance coefficient calculated by the detection processing unit 133, for example. The post-processing unit 132 also outputs an image signal used for image display to the display unit 14, and outputs an image signal used for image recording to the recording unit 15.

[0021] The detection processing unit 133 calculates a white balance coefficient based on the luminance value and integral value of each color component using the image signal of the detection area set automatically or by the user, and outputs the white balance coefficient to the post-processing unit 132. The detection processing unit 133 may also determine the exposure state based on the image signal of the detection area, generate an exposure control signal based on the determination result, and output the signal to the imaging optical system 11, thereby driving an aperture or the like so that the subject in the detection area has appropriate brightness.

[0022] The focus control unit 134 generates a control signal to focus on a predetermined subject based on the image signal supplied from the development processing unit 131, and outputs the control signal to the imaging optical system 11. The focus control unit 134 has a depth calculation unit 134a that calculates the depth using the image signal, and a control unit 134b that performs focus control using the depth value calculated by the depth calculation unit 134a.

[0023] The depth calculation unit 134a generates a depth map of the imaging area by machine learning, for example, deep learning such as DNN (Deep Neural Network), using the image signal supplied from the development processing unit 131. The depth calculation unit 134a also calculates a depth value of a predetermined subject by machine learning using the image signal. The predetermined subject may be set based on the depth map, or may be set based on the subject recognition result or a user instruction.

[0024] The control unit 134b performs focus control using a depth map generated from a captured image including a predetermined subject, and switches the focus control between a tracking mode in which the predetermined subject is tracked and continuously focused, and a non-tracking mode in which tracking is not performed. For example, in the tracking mode, the control unit 134b performs focus control based on subject distance information generated using the depth values ​​indicated in the depth map and the depth values ​​of the predetermined subject. In addition, in the non-tracking mode, the control unit 134b performs focus control using the depth map as subject distance information. The control unit 134b generates a control signal for focusing on the predetermined subject based on the subject distance information, and outputs the control signal to the imaging optical system 11.

[0025] The display unit 14 is a display device configured by, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, etc. The display unit 14 displays the user interface of the imaging device 10, a menu screen, a through image during imaging, still images and videos recorded in the recording unit 15, etc.

[0026] The recording unit 15 is configured using a recording medium such as a hard disk or a memory card. The recording medium may be fixed to the imaging device 10 or may be detachable. The recording unit 15 records images generated by the post-processing unit 132 of the image processing unit 13 on the recording medium. For example, the recording unit 15 records image signals of still images in a compressed state based on a predetermined standard (e.g., JPEG (Joint Photographic Experts Group)). The recording unit 15 also records information about the recorded images (e.g., EXIF ​​(Exchangeable Image File Format) data including additional information such as the date and time of capture) in association with the images. The recording unit 15 also records image signals of moving images in a compressed state based on a predetermined standard (e.g., MPEG2 (Moving Picture Experts Group 2) or MPEG4). The image signals may be compressed or decompressed by the recording unit 15 or by the post-processing unit 132 of the image processing unit 13.

[0027] The operation unit 16 includes, for example, a power button for switching the power on / off, a release button for issuing an instruction to start recording a captured image, a control for adjusting the zoom, and a touch screen formed integrally with the display unit 14. The operation unit 16 generates operation signals in response to user operations and outputs the signals to the system control unit 17.

[0028] The system control unit 17 is composed of a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), etc. The ROM stores programs that are read and run by the CPU. The RAM is used as a work memory for the CPU. The CPU executes various processes in accordance with the programs stored in the ROM and issues commands to control the operation of each unit so that the imaging device 10 operates in response to user operations. The programs of the system control unit 17 do not need to be pre-installed, but may be updated by download or using a distributed recording medium. The system control unit 17 may also be provided with some or all of the functions of the focus control unit 134.

[0029] The bus 18 electrically connects the above-mentioned components to enable transmission and reception of image signals, control signals, etc. The bus 18 may include multiple types of buses.

[0030] <2. Operation of the imaging device> <2-1. Operation of the imaging device> 3 illustrates a flowchart of a focus control operation in an imaging device. In step ST1, the image processing unit 13 generates a depth map. The focus control unit 134 of the image processing unit 13 calculates depth values ​​by machine learning using image signals generated in the development process, generates a depth map of the imaging area, and then proceeds to step ST2.

[0031] In step ST2, the image processing unit 13 determines whether it is in a tracking mode. If the focus control unit 134 of the image processing unit 13 is set to a tracking mode in which a predetermined subject is tracked and kept in focus, the process proceeds to step ST4, and if the focus control unit 134 is set to a non-tracking mode in which a predetermined subject is not tracked, unlike the tracking mode, the process proceeds to step ST3.

[0032] In step ST3, the image processing unit 13 performs focus control at a fixed position. The focus control unit 134 of the image processing unit 13 performs focus control by fixing the position of the subject to be focused. For example, the focus control unit 134 uses a depth map as subject distance information and performs focus control to focus on the closest subject indicated by the subject distance information. When performing focus control to focus on the closest subject, setting the detection range of the closest subject in the captured image can prevent focus control from being performed to focus on a subject located in front of the subject of interest. Furthermore, the focus control unit 134 may perform focus control to focus on a subject at a position indicated by the subject distance information, using information indicating the image position of a subject specified by the user in the captured image or a subject set based on the recognition result of subject recognition or the like for the captured image as subject distance information. The focus control unit 134 performs focus control at a fixed position and proceeds to step ST7.

[0033] In step ST4, the image processing unit 13 performs a process of generating subject distance information. The focus control unit 134 of the image processing unit 13 uses the depth map as subject distance information. The depth value of a predetermined subject and the depth value indicated by the depth map are information indicating distance, not distance. Therefore, the focus control unit 134 may use the distance to the predetermined subject as a reference and calculate an estimated distance to another subject based on the depth difference between the depth value indicated by the depth map and the depth value of the predetermined subject, thereby generating subject distance information indicating the distance to the predetermined subject and the estimated distance to the other subject.

[0034] When generating subject distance information indicating distance information to a predetermined subject and estimated distance information to other subjects, the focus control unit 134 calculates a depth value of the predetermined subject by machine learning using the captured image. The focus control unit 134 also calculates the estimated distance using the difference between the depth value of the predetermined subject and the depth value of the depth map generated in step ST1. Equation (1) illustrates an example of a calculation formula for the estimated distance DP. DP = (1 + (ax) × c) × d (1)

[0035] In equation (1), the variable "a" represents the depth value of a specific subject (e.g., a value within a range of 0 (far) to 1 (near)), and the variable "x" represents the depth value (e.g., a value within a range of 0 (far) to 1 (near)) indicated by a two-dimensional depth map based on a captured image. The variable "d" represents the absolute distance obtained from the focus lens position when the specific subject is in focus. The absolute distance is, for example, the distance to the subject that can be calculated using the focal length of the imaging optical system 11 and the distance from the lens position to the focal position when the subject is in focus, or the distance from the lens position to the focal position when the subject is in focus. The coefficient "c" is an adjustment coefficient that adjusts the distance relationship between the depth value indicated in the depth map and the depth value of the specific subject. In equation (1), the coefficient "c" adjusts the distance difference corresponding to the depth difference between the depth value indicated in the depth map and the depth value of the subject. In equation (1), as the value of the adjustment coefficient "c" increases, the calculated estimated distance to the specific subject increases. Therefore, the focus control unit 134 can calculate the estimated distance by placing emphasis on the distinguishability between the predetermined subject and other subjects with different depth values. Furthermore, as the value of the adjustment coefficient "c" becomes smaller, the calculated estimated distance has a smaller distance difference from the predetermined subject. Therefore, the focus control unit 134 can calculate a stable estimated distance with little fluctuation even if the depth value fluctuates. The focus control unit 134 generates subject distance information and proceeds to step ST5.

[0036] In step ST5, the image processing unit 13 searches for a target to be tracked. The focus control unit 134 of the image processing unit 13 searches for a predetermined target to be tracked based on the target distance information generated in step ST4, and the process proceeds to step ST6.

[0037] In step ST6, the image processing unit 13 performs focus control based on the search result. The focus control unit 134 of the image processing unit 13 performs focus control so as to focus on the tracking target searched for in step ST5, and the process proceeds to step ST7.

[0038] In step ST7, the image processing unit 13 determines whether the operation has ended. If the focus operation is in autofocus mode, the focus control unit 134 of the image processing unit 13 returns to step ST1, and ends the operation if the focus operation is switched to manual focus mode or an operation to end the imaging operation is performed.

[0039] <2-2. Example of operation> Next, an example of operation in the tracking mode will be described with reference to Fig. 4 to Fig. 9. Fig. 4 illustrates an example of an image captured by an imaging device, showing a case in which a desired subject, for example, a person OB holding a dog, moves away and then returns. Note that Fig. 4(a) shows an image captured at time t1, Fig. 4(b) shows an image captured at time t2, Fig. 4(c) shows an image captured at time t3, Fig. 4(d) shows an image captured at time t4, Fig. 4(e) shows an image captured at time t5, and Fig. 4(f) shows an image captured at time t6.

[0040] FIG. 5 illustrates depth maps obtained by performing machine learning using captured images acquired by an imaging device. (a) of FIG. 5 shows a depth map at time t1, (b) of FIG. 5 shows a depth map at time t2, (c) of FIG. 5 shows a depth map at time t3, (d) of FIG. 5 shows a depth map at time t4, (e) of FIG. 5 shows a depth map at time t5, and (f) of FIG. 5 shows a depth map at time t6. In the depth map shown in FIG. 5, closer positions have higher brightness and farther positions have lower brightness. By using the depth map obtained by performing machine learning using captured images, the focus control unit 134 can perform autofocus without using a distance measurement sensor.

[0041] FIG. 6 illustrates a focus operation. (a) to (f) in FIG. 6 show a conventional operation, while (g) to (l) in FIG. 6 show an operation using subject distance information (depth map). FIG. 6 also shows a rectangular tracking frame MA and a focus point MP within the tracking frame. When using a conventional method of tracking based on pattern and color information, if the pattern and color of the background and the subject being tracked are similar, the focus point MP moves from the subject to the background at time t4, as shown in (d) in FIG. 6. Therefore, at time t5 and time t6, focus control is performed to focus on the focus point located in the background, and the specified subject OB cannot be tracked and kept in focus. However, by using a depth map obtained by machine learning using captured images, as in the present technology, the depth values ​​of the background and the subject OB are different, and the subject OB can be searched for using the depth values. Therefore, as shown in (g) to (l) in FIG. 6, the subject OB can be tracked and kept in focus from time t1 to time t6.

[0042] However, the depth values ​​shown in the depth map may fluctuate in the time direction. For example, the position of the subject OB at time t3 is farther away than its position at time t2, but the depth value of the subject OB in (c) of Figure 5 indicates a closer distance than the depth value of the subject OB in (b) of Figure 5, resulting in fluctuation in the time direction. Therefore, if the fluctuation becomes large, there is a risk that the focus point (tracking frame) will move from the subject OB to another subject.

[0043] Therefore, the focus control unit 134 may generate subject distance information indicating distance information to the predetermined subject and estimated distance information to other subjects, and search for the predetermined subject based on the generated subject distance information. In this case, the generation of the subject distance information may use an adjustment coefficient that adjusts the distance difference corresponding to the depth difference between the depth value indicated in the depth map and the depth value of the predetermined subject, as described above.

[0044] Fig. 7 shows subject distance information when adjustment coefficient c is set to 1. Note that Fig. 7(a) shows subject distance information at time t1, Fig. 7(b) shows subject distance information at time t2, Fig. 7(c) shows subject distance information at time t3, Fig. 7(d) shows subject distance information at time t4, Fig. 7(e) shows subject distance information at time t5, and Fig. 7(f) shows subject distance information at time t6.

[0045] FIG. 8 shows subject distance information when the adjustment coefficient c is set to "2." Note that (a) in FIG. 8 shows subject distance information at time t1, (b) in FIG. 8 shows subject distance information at time t2, (c) in FIG. 8 shows subject distance information at time t3, (d) in FIG. 8 shows subject distance information at time t4, (e) in FIG. 8 shows subject distance information at time t5, and (f) in FIG. 8 shows subject distance information at time t6. The subject distance information when the adjustment coefficient c is set to "2" has a larger distance difference between the subject OB and the background than when the adjustment coefficient c is set to "1." Therefore, focus control that emphasizes discrimination can be performed.

[0046] FIG. 9 shows subject distance information when the adjustment coefficient c is set to "0.5." Note that (a) in FIG. 9 shows subject distance information at time t1, (b) in FIG. 9 shows subject distance information at time t2, (c) in FIG. 9 shows subject distance information at time t3, (d) in FIG. 9 shows subject distance information at time t4, (e) in FIG. 9 shows subject distance information at time t5, and (f) in FIG. 9 shows subject distance information at time t6. The subject distance information when the adjustment coefficient c is set to "0.5" has a smaller distance difference between the subject OB and the background than when the adjustment coefficient c is set to "1." Therefore, focus control can be performed with emphasis on stability.

[0047] The adjustment coefficient may be set in advance or may be set by the user. The adjustment coefficient may also be set according to the composition of the captured image. For example, when the composition of the captured image is a portrait in which a specific person is the main subject, tracking the main subject is easy. Therefore, the value of the adjustment coefficient c is set to be larger than usual to emphasize distinctiveness, thereby preventing the focus point from moving to the foreground or background. Furthermore, in a scene in which multiple people are subjects, tracking the specific subject is difficult. Therefore, the value of the adjustment coefficient c is set to be smaller than usual to emphasize stability.

[0048] Furthermore, in the non-tracking mode, the focus control unit 134 performs focus control to focus on the subject at the closest position indicated by the subject distance information generated using the depth map. For example, if the subject OB shown in FIG. 4 does not move from its closest position at time t1, focus control can be performed to focus on the subject OB at the closest position in the depth map generated from the captured image including the subject OB, thereby enabling focus on the subject OB at the closest position. Also, if the subject OB is located at time t5 shown in FIG. 4 (5), for example, performing focus control to focus on the closest position in the depth map generated from the captured image including the subject OB may result in focusing on the foreground. However, if a search range is set in the center of the captured image and focus control is performed to focus on the subject at the closest position within the search range, it becomes possible to focus on the subject OB.

[0049] <2-3. Other actions> In the above-described operation, the focus control is performed by changing the adjustment coefficient, but the focus control unit 134 may perform time-direction filtering on the depth map to reduce fluctuations in the depth value in the time direction. The focus control unit 134 may perform IIR (Infinite Impulse Response) filtering as the time-direction filtering. Alternatively, the focus control unit 134 may perform FIR (Finite Impulse Response) filtering as the time-direction filtering. Note that the time-direction filtering is not limited to the depth map, and may instead be filtering of an estimated distance.

[0050] FIG. 10 illustrates an example of subject distance information when IIR filtering is performed. Note that FIG. 10 illustrates an example where the adjustment coefficient is "c=1" and the IIR feedback coefficient is "0.5." Note that (a) of FIG. 10 illustrates subject distance information at time t1, (b) of FIG. 10 illustrates subject distance information at time t2, (c) of FIG. 10 illustrates subject distance information at time t3, (d) of FIG. 10 illustrates subject distance information at time t4, (e) of FIG. 10 illustrates subject distance information at time t5, and (f) of FIG. 10 illustrates subject distance information at time t6. In this case, even if fluctuations occur in the depth map as shown in FIG. 5 and the depth value of the subject OB at time t3 is closer than the depth value of the subject OB at time t2, the fluctuations are reduced by filtering in the time direction. Therefore, as shown in (c) of Figure 10, the position difference between the position of the subject OB indicated by the subject distance information at time t2 and the position of the subject OB indicated by the subject distance information at time t3 can be made smaller than before filtering.

[0051] In this way, the focus control unit 134 performs filtering in the time direction, reduces the influence of fluctuations in depth values ​​in the time direction, and performs focus control so as to track and keep in focus a predetermined subject.

[0052] Furthermore, the adjustment coefficients for adjusting the distance relationship between the depth values ​​indicated in the depth map and the depth values ​​of the predetermined subject are not limited to the coefficients described above. For example, a coefficient that changes linearly or nonlinearly depending on the depth difference between the depth values ​​indicated in the depth map and the depth values ​​of the predetermined subject may be used.

[0053] <3. Application Examples> The technology according to the present disclosure can be applied to various fields. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving object, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, etc. Furthermore, the technology according to the present disclosure may be realized as a device mounted on equipment used in production processes in factories or equipment used in the construction field.

[0054] If applied to such fields, it will be possible to obtain images in focus on a desired subject at low cost and in a small space, reducing fatigue for drivers and workers and enabling safer autonomous driving.

[0055] The technology disclosed herein can also be applied to the medical field. For example, if applied to the case where an image of the surgical site is used during surgery, an image focused on the surgical site can be easily obtained, reducing the surgeon's fatigue and enabling the surgery to be performed safely and more reliably.

[0056] The technology according to the present disclosure can also be applied to the surveillance field. For example, if the technology is applied to a surveillance camera, an image in focus on the monitored object can be easily obtained.

[0057] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer incorporated in dedicated hardware and executed. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

[0058] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read-only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called packaged software.

[0059] In addition to being installed on a computer from a removable recording medium, the program may also be transferred wirelessly or by wire from a download site to a computer via a network such as a WAN (Wide Area Network) typified by cellular, a LAN (Local Area Network), or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

[0060] Note that the effects described in this specification are merely examples and are not limiting, and additional effects not described may exist. Furthermore, the present technology should not be interpreted as being limited to the above-described embodiments of the technology. The embodiments of the technology disclose the present technology in the form of examples, and it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration when determining the gist of the present technology.

[0061] Furthermore, the focus control device of the present technology can also have the following configuration. (1) A focus control device that performs focus control using a depth map generated from an image containing a specified subject, and has a control unit that switches the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked. (2) The focus control device described in (1) wherein, when in the tracking mode, the control unit performs focus control to focus on the specified subject searched for based on subject distance information generated using the depth map. (3) The focus control device according to (2), wherein the control unit generates the subject distance information using depth values ​​indicated in the depth map and depth values ​​of the predetermined subject. (4) The focus control device described in (3), wherein the control unit generates the subject distance information using an adjustment coefficient that adjusts the distance relationship between the depth value indicated in the depth map and the depth value of the specified subject. (5) The focus control device according to (4), wherein the control unit adjusts the adjustment coefficient to a distance difference corresponding to a depth difference between a depth value indicated in the depth map and a depth value of the subject. (6) The focus control device according to (4) or (5), wherein the control unit sets the adjustment coefficient in accordance with a composition of the captured image. (7) The focus control device according to any one of (2) to (6), wherein the control unit performs a filtering process in a time direction to generate the subject distance information. (8) The focus control device according to (7), wherein the control unit performs the filtering process on the depth map. (9) A focus control device described in any one of (1) to (8), wherein the control unit, when in the non-tracking mode, performs focus control to focus on the subject at the closest position indicated by the subject distance information generated using the depth map. (10) The focus control device according to (9), wherein the control unit performs focus control to bring the subject at the closest position within a search range set in the captured image into focus. (11) The focus control device according to any one of (1) to (10), wherein the control unit performs machine learning using the captured image to generate the depth map. [Explanation of symbols]

[0062] 10. Imaging device 11. Imaging optical system 12. Imaging unit 13. Image processing section 14...Display section 15. Recording section 16...Operation unit 17 System control section 18 Bus 131 Development processing section 132 Post-processing section 133 Detection processing section 134 Focus control section 134a Depth calculation section 134b Control section

Claims

1. a control unit that performs focus control using a depth map generated from a captured image including a predetermined subject, and switches the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked, and When in the tracking mode, the control unit performs focus control to focus on the predetermined subject searched for based on subject distance information generated by adjusting the distance relationship between the depth value indicated in the depth map and the depth value of the predetermined subject using an adjustment coefficient. Focus control device.

2. The control unit adjusts a distance difference corresponding to a depth difference between a depth value indicated in the depth map and a depth value of the subject using the adjustment coefficient. The focus control device according to claim 1 .

3. The control unit sets the adjustment coefficient according to the composition of the captured image. The focus control device according to claim 1 .

4. The control unit performs a filtering process in a time direction and generates the subject distance information. The focus control device according to claim 1 .

5. The control unit performs the filtering process on the depth map.

5. The focus control device according to claim 4.

6. When in the non-tracking mode, the control unit performs focus control to focus on the closest subject indicated by subject distance information generated using the depth map. The focus control device according to claim 1 .

7. The control unit performs focus control to focus on a subject at a closest position within a search range set in the captured image.

7. The focus control device according to claim 6.

8. The control unit performs machine learning using the captured image to generate the depth map. The focus control device according to claim 1 .

9. The method includes performing focus control using a depth map generated from an image containing a predetermined subject, and switching the focus control to different controls by a control unit between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked; When in the tracking mode, the control unit performs focus control to focus on the predetermined subject searched for based on subject distance information generated by adjusting the distance relationship between the depth value indicated in the depth map and the depth value of the predetermined subject using an adjustment coefficient. Focus control method.

10. A program for causing a computer to execute focus control of an imaging device, comprising: A step of performing focus control using a depth map generated from a captured image including a predetermined subject; a step of switching the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked, and a step of performing, in the tracking mode, focus control to focus on the predetermined subject searched for based on subject distance information generated by adjusting the distance relationship between the depth value indicated in the depth map and the depth value of the predetermined subject using an adjustment coefficient; A program for causing the computer to execute the above.

11. An imaging unit that generates a captured image; a control unit that generates a depth map from an image captured by the imaging unit, the image capturing unit includes a predetermined subject, performs focus control using the depth map, and switches the focus control between a tracking mode in which the subject is tracked and a non-tracking mode in which the subject is not tracked; and Equipped with When in the tracking mode, the control unit performs focus control to focus on the predetermined subject searched for based on subject distance information generated by adjusting the distance relationship between the depth value indicated in the depth map and the depth value of the predetermined subject using an adjustment coefficient. Imaging device.

Images