Imaging device, imaging method, and program

Abstract

The present technology is related to an imaging device, an imaging method, and a program capable of imaging in a desired exposure time even when power consumption is reduced. The present invention is provided with: a frame control unit that controls any one of a first frame on which a read operation and a shutter operation are performed, a second frame on which a read operation is performed, a third frame on which a shutter operation is performed, or fourth frames on which neither a read operation nor a shutter operation is performed; and a frame setting unit that sets the third frame as one frame among the fourth consecutive frames. The present technology may be applied to an imaging device, for example.

Description

Imaging device, imaging method, and program

[0001] The present technology relates to an imaging device, an imaging method, and a program, and, for example, to an imaging device, an imaging method, and a program that are capable of reducing power consumption.

[0002] Imaging elements such as CCD (Charge Coupled Device) image sensors and CMOS (Complementary Metal Oxide Semiconductor) image sensors are widely used in digital cameras, mobile phones, and the like. A column AD system is known for CMOS image sensors, which provides an ADC (Analog-to-Digital Converter) for each pixel column and operates the ADCs in parallel to increase the readout speed. For example, Patent Document 1 (JP-A-2005-102666) proposes reducing the power consumption of imaging devices such as CMOS image sensors.

[0003] Japanese Patent Application Laid-Open No. 2013-240002

[0004] There is a demand for further reduction in power consumption in imaging devices. There is also a demand for imaging devices that can perform desired imaging processing without imposing restrictions on the imaging processing, even when imaging with reduced power consumption is performed.

[0005] The present technology has been made in consideration of such circumstances, and is intended to reduce power consumption and prevent restrictions on desired imaging processing.

[0006] An imaging device according to one aspect of the present technology is an imaging device that includes a frame control unit that controls any one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and a frame setting unit that sets the third frame to one of the consecutive fourth frames.

[0007] An imaging method according to one aspect of the present technology is an imaging method in which an imaging device controls one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and sets the third frame to one of the consecutive fourth frames.

[0008] A program according to one aspect of the present technology is a program for causing a computer controlling an imaging device to execute processing including a step of controlling one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and setting the third frame to one of the consecutive fourth frames.

[0009] In an imaging device, an imaging method, and a program according to one aspect of the present technology, control is performed on one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and the third frame is set as one of the consecutive fourth frames.

[0010] The imaging device may be an independent device or an internal block constituting a single device.

[0011] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0012] FIG. 1 is a diagram illustrating a configuration of an embodiment of an imaging device to which the present technology is applied. FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging device to which the present technology is applied. FIG. 3 is a diagram illustrating imaging processing of the imaging device. FIG. 4 is a diagram illustrating imaging processing of the imaging device. FIG. 5 is a diagram illustrating setting of a shutter operating position of the imaging device. FIG. 6 is a diagram illustrating setting of a shutter operating position of the imaging device. FIG. 7 is a diagram illustrating setting of a shutter operating position of the imaging device. FIG. 8 is a diagram illustrating an example of the configuration of a PC. FIG. 9 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system. FIG. 10 is a block diagram illustrating an example of the functional configuration of a camera head and a CCU. FIG. 11 is a block diagram illustrating an example of a schematic configuration of a vehicle control system. FIG. 12 is an explanatory diagram illustrating an example of installation positions of an outside-of-vehicle information detection unit and an imaging unit.

[0013] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described.

[0014] <Configuration example of imaging device> The present technology is applicable to general electronic devices that use an imaging element in an image capture unit (photoelectric conversion unit), such as imaging devices such as digital still cameras and video cameras, mobile terminal devices with imaging functions, copiers that use an imaging element in an image reading unit, etc. The imaging element may be formed as a single chip, or may be in the form of a module having an imaging function in which the imaging unit and a signal processing unit or an optical system are packaged together.

[0015] Fig. 1 is a block diagram showing an example configuration of an embodiment of an imaging device 11 as an electronic device to which the present technology is applied. The imaging device 11 in Fig. 1 includes an optical unit 21 including a lens group and the like, an imaging element (imaging device) 22, and a DSP (Digital Signal Processor) circuit 23 which is a camera signal processing circuit. The imaging device 11 also includes a frame memory 24, a display unit 25, a recording unit 26, an operation unit 27, and a power supply unit 28. The DSP circuit 23, the frame memory 24, the display unit 25, the recording unit 26, the operation unit 27, and the power supply unit 28 are connected to each other via a bus line 29.

[0016] The optical unit 21 takes in incident light (image light) from a subject and forms an image on the imaging surface of the imaging element 22. The imaging element 22 converts the amount of incident light formed on the imaging surface by the optical unit 21 into an electrical signal on a pixel-by-pixel basis and outputs the signal as a pixel signal.

[0017] The display unit 25 is configured with a thin display such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays moving images or still images captured by the imaging element 22. The recording unit 26 records the moving images or still images captured by the imaging element 22 on a recording medium such as a hard disk or semiconductor memory.

[0018] The operation unit 27, under the user's operation, issues operation commands for various functions of the image sensor 22. The power supply unit 28 appropriately supplies various types of power to the DSP circuit 23, frame memory 24, display unit 25, recording unit 26, and operation unit 27 as operating power sources.

[0019] <Functions of the Imaging Device> FIG. 2 is a block diagram showing the functions of the imaging device 11. As shown in FIG.

[0020] The imaging device 11 includes an imaging unit 51, a signal processing unit 52, a synchronization signal generating unit 53, a control unit 54, a mode switching unit 55, a frame status management unit 56, a frame control unit 57, a thinning counter 58, and a shutter position setting unit 59. Each block is configured by hardware and / or software.

[0021] The imaging unit 51 includes an imaging element and a lens, receives light reflected from a subject, and outputs a signal corresponding to the amount of light received to the signal processing unit 52. The signal processing unit 52 performs processes such as defect correction, noise reduction, and high dynamic range synthesis (HDR) on the input signal, and outputs the signal to, for example, the display unit 25 or the recording unit 26 ( FIG. 1 ).

[0022] The synchronization signal generating unit 53 generates a synchronization signal and supplies it to the signal processing unit 52. The synchronization signal generated by the synchronization signal generating unit 53 is, for example, a vertical synchronization signal. The control unit 54 controls each unit within the imaging device 11.

[0023] The mode switching unit 55 executes a process of switching between a normal imaging mode and a thinning imaging mode, which will be described later. The frame status management unit 56 manages which of four types of frames, namely, a lead shutter frame, a lead frame, a thinning frame, and a shutter frame, which will be described later, is being processed. The frame control unit 57 executes a process according to the frame status managed by the frame status management unit 56.

[0024] When the thinning-out imaging mode is set, the thinning-out counter 58 counts the number of frames to be thinned out as frames not to be imaged. The shutter position setting unit 59 sets which of the frames set as thinned out in the thinning-out imaging mode should perform the shutter operation, in other words, which of the thinned-out frames should be used as the shutter frame.

[0025] <Operation of Imaging Device in Normal Imaging Mode> The operation of the imaging device 11 when performing imaging processing will be described with reference to Fig. 3. In the following description, a mode in which a frame is generated (output) for each synchronization signal will be referred to as a normal imaging mode, and a mode in which fewer frames are captured than the number of frames captured in the normal imaging mode will be referred to as a thinned-out imaging mode. Imaging in the normal imaging mode will be described with reference to Fig. 3, and imaging in the thinned-out imaging mode will be described with reference to Fig. 4.

[0026] In normal imaging mode, a frame is generated for each synchronization signal by repeatedly performing a read operation and a shutter operation in synchronization with the synchronization signal XVS, as shown in Fig. 3. The period from a predetermined synchronization signal to the next synchronization signal is defined as one frame, and a frame in which a read operation and a shutter operation are performed within that frame will be referred to as a lead / shutter frame hereinafter.

[0027] Control relating to the imaging of the lead shutter frame is performed by the frame control unit 57 when the normal imaging mode is set in the mode switching unit 55. The frame control unit 57 executes a read operation in accordance with the synchronization signal generated by the synchronization signal generation unit 53, and executes a shutter operation in accordance with the set exposure time.

[0028] <Operation of imaging device in thinning-out imaging mode> Imaging processing in the thinning-out imaging mode will be described with reference to Fig. 4. Processing in the thinning-out imaging mode is performed by the frame control unit 57 based on the frame status managed by the frame status management unit 56 when the thinning-out imaging mode is set by the mode switching unit 55.

[0029] In the example shown in FIG. 4, frames F12, F13, and F14 are set as thinned-out frames.

[0030] For frame F11, information indicating that it is a read / shutter frame is set in the frame status management unit 56, and based on this setting, the frame control unit 57 performs control, whereby the read operation and the shutter operation are performed within one frame.

[0031] Frame F12 is a thinned frame, but is treated as a read frame because the signal obtained by the shutter operation executed in the previous frame F11 is read out. Information indicating that frame F12 is a read frame is set in the frame status management unit 56, and based on this setting, the frame control unit 57 performs control to perform a read operation within one frame.

[0032] Frame F13 is a thinned-out frame, and is a frame in which no read operation or shutter operation is performed. Information indicating that frame F13 is a thinned-out frame is set in the frame status management unit 56, and based on this setting, the frame control unit 57 does not perform control, so that frame F13 is a frame in which no read operation or shutter operation is performed.

[0033] Frame F14 is a thinned-out frame, but is treated as a shutter frame because a shutter operation is performed to generate a signal to be read in the readout operation executed in the subsequent frame F15. Information indicating that frame F14 is a shutter frame is set in the frame status management unit 56, and based on this setting, the frame control unit 57 performs control to perform a shutter operation within one frame.

[0034] Thus, in this embodiment, four types of frames are set: lead shutter frame, lead frame, thinned frame, and shutter frame, and the frame types are managed by the frame status management unit 56. The frame status management unit 56 can be configured with, for example, a register. The frame control unit 57 controls the read operation and shutter operation according to the frame type managed by the frame status management unit 56.

[0035] In the thinned-out frames, shutter operations and read operations are not performed, in the shutter frames, read operations are not performed, and in the read frames, shutter operations are not performed. During these sections where shutter operations and read operations are not performed, driving related to the imaging process can be stopped. During sections where driving related to the imaging process can be stopped, analog circuits such as clock oscillation circuits such as PLL circuits and charge pump circuits such as step-up circuits and step-down circuits can be turned off, thereby achieving low power consumption.

[0036] In the thinning-out imaging mode, in the read frame, thinning frame, and shutter frame in which imaging is skipped, the section in which the read operation or the shutter operation is not performed is set as a low power consumption section in which at least one circuit such as the clock oscillation circuit or the analog circuit is turned off, and the read shutter frame also has a section in which the read operation or the shutter operation is not performed, and if there is a section (time) that can be set as a low power consumption section, the clock oscillation circuit or the analog circuit is turned off, and the section can be configured to be set as a low power consumption section and to be driven at low power consumption.

[0037] By providing frames in which imaging is skipped, it is possible to set a section in which the imaging device 11 is driven with at least low power consumption, thereby reducing power consumption of the imaging device 11. Furthermore, by providing frames in which imaging is skipped, it is possible to set frames in which only a read operation or only a shutter operation is performed, and it is possible to set a low power consumption section in these frames as well, thereby reducing power consumption of the imaging device 11.

[0038] <Shutter Frame Position> The position of the shutter frame will be described with reference to Fig. 5. The shutter frame is set by the shutter position setting unit 59, and as in the example shown in Fig. 4, it is set to the last thinned-out frame of the thinned-out frames set to skip imaging, in other words, the frame before the read operation is performed. The position of the shutter frame will be further described using an example in which six thinned-out frames are set as shown in Fig. 5.

[0039] Frames F12 to F17 are set as skip frames for skipping imaging. Of frames F12 to F17 set as skip frames, frame F11 is set as a lead frame, and frame F17 is set as a shutter frame. Frame F11 is set as a lead frame because a read operation is performed to read out a signal obtained by a shutter operation in a previous frame (not shown). Frame F17 is set as a shutter frame because it is the frame in which a shutter operation is performed to acquire a signal read out by a read operation performed in frame F18.

[0040] The following is an explanation of the frame status managed by the frame status management unit 56. In the example shown in Fig. 5, the frame status is managed as "2'b01" for a lead frame, "2'b00" for a thinned frame, and "2'b10" for a shutter frame.

[0041] "2'b" indicates a 2-bit binary notation, "01" after "2'b" indicates that there is no shutter operation but there is a read operation, "00" indicates that there is neither a shutter operation nor a read operation, and "10" indicates that there is a shutter operation but there is no read operation. Although not shown, in the case of a read shutter frame, both a read operation and a shutter operation are performed, so the information "2'b11" is managed.

[0042] The frame status management unit 56 is, for example, a register, and writes into the register the information "01" represented by 2 bits if it is a lead frame, the information "00" if it is a thinned frame, and the information "10" if it is a shutter frame. The frame control unit 57 determines the type of frame to be processed by referencing the frame status managed by the frame status management unit 56, and performs frame control based on that determination.

[0043] The thinning counter 58 is set to the thinning imaging mode and counts the number of frames set to skip (thin out) imaging. In the example shown in Fig. 5, the number of frames set to be thinned out is 6.

[0044] Frames F12 to F17 are set as frames to be thinned out, and the thinning counter 58 stores count values ​​of "7'd1", "7'd2", "7'd3", "7'd4", "7'd5", and "7'd6" in order from frame F12. "7'd" indicates a 7-bit decimal notation, and the number after "7'd" indicates the count value, with counting starting from 1. The count value indicates the ordinal number of the thinned frame among the consecutive thinned frames.

[0045] 5, frame F11 is managed as information "2'b01" by the frame status management unit 56, and is assigned a counter value of "7'd0" by the thinning counter 58. Since frame F11 is not a thinned frame whose imaging is skipped, no counting is performed by the thinning counter 58, and "0" is assigned as the counter value.

[0046] For frame F12, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd1". For frame F13, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd2".

[0047] For frame F14, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd3". For frame F15, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd4".

[0048] For frame F16, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd5". For frame F17, the frame status management unit 56 manages the information "2'b00", and the thinning counter 58 assigns a counter value of "7'd6".

[0049] For frame F18, the frame status management unit 56 manages the information "2'b01", and the thinning counter 58 assigns a counter value of "7'd0". Since frame F18 is not a thinned frame whose imaging is skipped, the counter value assigned is "0".

[0050] 5, the shutter frame is set to frame F17. The signal obtained by the shutter operation performed in frame F17 is read out in the read operation performed in the next frame F18. In this case, the exposure time is restricted to be within the length of one frame of frame F17 (the time from time t17 to time t18 when the synchronization signal XVS is output).

[0051] When an exposure time of one frame or more is to be set, the shutter position setting unit 59 can set a thinned-out frame corresponding to the exposure time as the shutter frame. In the example shown in Fig. 6, frame F14 is set as the shutter frame. In the example shown in Fig. 6, the frame status management unit 56 manages information of "2'b10" for frame F14, and the thinned-out counter 58 assigns a counter value of "7'd3" to frame F14. In the example shown in Fig. 6, frame F14 is set as a shutter frame in which a shutter operation is performed but a read operation is not performed, and is the third frame among the frames set as thinned-out frames.

[0052] By using frame F14 as the shutter frame, the exposure time can be secured to span the four frames of frame F14, frame F15, frame F16, and frame F17, thereby expanding the exposure time constraints.

[0053] The shutter position setting unit 59 is configured to be able to set a frame that can realize a desired exposure time as the shutter frame. Furthermore, the shutter position setting unit 59 can change the position of the shutter frame. This will be described with reference to FIG. 7.

[0054] The example shown in Fig. 7 shows a case where frame F14 is set as the shutter frame, as in the example shown in Fig. 6. The shutter position setting unit 59 specifies frame F14, in which the counter value of the thinning counter 58 is "7'd3", as the shutter frame, and in response to this specification, the frame status management unit 56 sets information "2'd10" as the frame status information.

[0055] If a change in exposure time is instructed between time t15 and time t16 (frame F15), the change in exposure time is reflected in the synchronization signal XVS issued at time t16. That is, the shutter position setting unit 59 re-designates frame F16, for which the counter value of the thinning counter 58 is "7'd5," as the shutter frame, and in response to this re-designation, the frame status management unit 56 changes the frame status information of frame F16 to "2'd10."

[0056] Since the frame status information of frame F16 has been changed to "2'd10", the frame control unit 57 executes the shutter operation in frame F16. By executing such processing, the position of the set shutter frame can be changed.

[0057] 7 shows the case where frame F16 is changed to the shutter frame, but depending on the changed exposure time, frame F17 may also be changed to the shutter frame. In this way, when the exposure time is changed, the shutter position setting unit 59 can change (set) a thinned-out frame at a position appropriate for the changed exposure time as the shutter frame for performing the shutter operation.

[0058] The shutter position setting unit 59 can assign a shutter frame to one of successive thinned frames. Therefore, even in a case where successive thinned frames are set, the shutter frame (shutter operation) can be set at a position where a desired exposure time can be achieved, and the desired image can be captured.

[0059] As described above, a thinning-out imaging mode is provided in which imaging processing of one or more frames is skipped, and in the thinning-out imaging mode, a low-power consumption period is set in which the clock oscillation circuit and analog circuits are turned off, thereby making it possible to reduce the power consumption of the imaging device 11.

[0060] Even in the case of a thinning-out imaging mode in which a low-power section is provided, it is possible to perform a desired imaging operation, for example, an imaging operation with a desired exposure time, thereby eliminating limitations on imaging processing due to low power consumption. Thus, according to this embodiment, it is possible to perform desired imaging processing while realizing low power consumption.

[0061] <Regarding the Recording Medium> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0062] 8 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program. In the computer, a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, and a RAM (Random Access Memory) 2003 are interconnected by a bus 2004. An input / output interface 2005 is also connected to the bus 2004. An input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010 are connected to the input / output interface 2005.

[0063] The input unit 2006 includes a keyboard, a mouse, a microphone, etc. The output unit 2007 includes a display, a speaker, etc. The storage unit 2008 includes a hard disk, a non-volatile memory, etc. The communication unit 2009 includes a network interface, etc. The drive 2010 drives removable media 2011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0064] In a computer configured as described above, the CPU 2001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004 and executing it.

[0065] The program executed by the computer (CPU 2001) can be provided by being recorded on, for example, a removable medium 2011 such as a package medium. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0066] In the computer, the program can be installed in the storage unit 2008 via the input / output interface 2005 by inserting the removable medium 2011 into the drive 2010. The program can also be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. Alternatively, the program can be installed in advance in the ROM 2002 or the storage unit 2008.

[0067] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0068] <Application Example to Endoscopic Surgery System> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0069] FIG. 9 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0070] 9 shows an operator (doctor) 11131 performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0071] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the example shown, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may also be configured as a so-called flexible scope having a flexible lens barrel.

[0072] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an object to be observed inside the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0073] An optical system and an image sensor are provided inside the camera head 11102, and light reflected from the object of observation (observation light) is collected onto the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. The image signal is sent to a camera control unit (CCU) 11201 as RAW data.

[0074] The CCU 11201 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and comprehensively controls the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102 and performs various types of image processing on the image signal, such as development processing (demosaic processing), to display an image based on the image signal.

[0075] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0076] The light source device 11203 is composed of a light source such as an LED (light emitting diode), and supplies irradiation light to the endoscope 11100 when photographing the surgical site, etc.

[0077] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 11100.

[0078] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 inflates the body cavity of the patient 11132 through the insufflation tube 11111 in order to ensure a clear field of view for the endoscope 11100 and a working space for the surgeon. The recorder 11207 is a device capable of recording various types of information related to the surgery. The printer 11208 is a device capable of printing various types of information related to the surgery in various formats such as text, images, or graphs.

[0079] The light source device 11203, which supplies illumination light to the endoscope 11100 when photographing the surgical site, can be configured from a white light source, such as an LED, a laser light source, or a combination of these. When the white light source is configured from a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, allowing the light source device 11203 to adjust the white balance of the captured image. In this case, it is also possible to time-share images corresponding to each RGB by irradiating the object of observation with laser light from each RGB laser light source and controlling the drive of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, color images can be obtained without providing a color filter to the image sensor.

[0080] Furthermore, the light source device 11203 may be controlled to change the intensity of light it outputs at predetermined time intervals. By controlling the driving of the image sensor of the camera head 11102 in synchronization with the timing of the change in light intensity to acquire images in a time-division manner and combining the images, it is possible to generate an image with a high dynamic range that is free from so-called blocked-up shadows and blown-out highlights.

[0081] The light source device 11203 may also be configured to supply light in a predetermined wavelength band corresponding to special light observation. Special light observation, for example, utilizes the wavelength dependence of light absorption in body tissues to irradiate light with a narrower band than the light irradiated during normal observation (i.e., white light), thereby capturing high-contrast images of specific tissues, such as blood vessels on the surface of mucous membranes, known as narrow-band imaging. Alternatively, special light observation may involve fluorescence observation, in which images are obtained using fluorescence generated by irradiating excitation light. Fluorescence observation may involve irradiating excitation light onto body tissues and observing the fluorescence from the tissue (autofluorescence observation), or by locally injecting a reagent such as indocyanine green (ICG) into the body tissue and irradiating the tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow-band light and / or excitation light corresponding to such special light observation.

[0082] FIG. 10 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0083] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other via a transmission cable 11400 so that they can communicate with each other.

[0084] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is composed of a combination of multiple lenses including a zoom lens and a focus lens.

[0085] The imaging unit 11402 may include one imaging element (a so-called single-chip type) or multiple imaging elements (a so-called multi-chip type). When the imaging unit 11402 is configured as a multi-chip type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining these signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to a 3D (dimensional) display. The 3D display allows the surgeon 11131 to more accurately grasp the depth of the biological tissue at the surgical site. Note that when the imaging unit 11402 is configured as a multi-chip type, multiple lens units 11401 may be provided corresponding to each imaging element.

[0086] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately after the objective lens.

[0087] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be adjusted appropriately.

[0088] The communication unit 11404 is configured by a communication device for transmitting and receiving various types of information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0089] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of imaging, and / or information specifying the magnification and focus of the captured image.

[0090] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by the user, or may be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0091] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404 .

[0092] The communication unit 11411 is configured by a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0093] Furthermore, the communication unit 11411 transmits to the camera head 11102 a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0094] The image processing unit 11412 performs various image processing operations on the image signal, which is RAW data transmitted from the camera head 11102 .

[0095] The control unit 11413 performs various controls related to the imaging of the surgical site, etc. by the endoscope 11100 and the display of the captured image obtained by imaging the surgical site, etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0096] Furthermore, the control unit 11413 displays the captured image showing the surgical site, etc., on the display device 11202 based on the image signal subjected to image processing by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition technologies. For example, the control unit 11413 can recognize surgical tools such as forceps, specific biological parts, bleeding, mist generated when using the energy treatment tool 11112, etc., by detecting the shape and color of the edges of objects included in the captured image. When displaying the captured image on the display device 11202, the control unit 11413 may use the recognition results to superimpose various surgical support information on the image of the surgical site. By superimposing the surgical support information and presenting it to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0097] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for electrical signal communication, an optical fiber for optical communication, or a composite cable of these.

[0098] In the illustrated example, communication is performed wired using a transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may also be performed wirelessly.

[0099] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0100] FIG. 11 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0101] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 11, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0102] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0103] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0104] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0105] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0106] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0107] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0108] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0109] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0110] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 11, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0111] FIG. 12 is a diagram showing an example of the installation position of the imaging unit 12031.

[0112] In FIG. 12, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0113] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0114] 12 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0115] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0116] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0117] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0118] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0119] In this specification, a system refers to an entire device made up of multiple devices.

[0120] Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present. (1) An imaging device comprising: a frame control unit that controls one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed; and a frame setting unit that sets the third frame to one of the consecutive fourth frames. (2) The imaging device described in (1) above, further comprising a counter that counts the ordinal number of the fourth frame among the consecutive fourth frames, and the frame setting unit sets the third frame by specifying a counter value of the counter. (3) The imaging device described in (1) or (2), in which, when an instruction to change the exposure time is received, the frame setting unit resets the third frame to the fourth frame that is appropriate for the changed exposure time. (4) The imaging device described in any of (1) to (3), in which imaging drive is stopped when the fourth frame is controlled by the frame control unit. (5) The imaging device according to any one of (1) to (3), wherein when the fourth frame is controlled by the frame control unit, at least one circuit of a clock oscillation circuit, a voltage step-up circuit, or a step-down circuit is stopped. (6) An imaging method, wherein the imaging device controls any one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and sets the third frame to one of the consecutive fourth frames.(7) A program for causing a computer that controls an imaging device to execute processing including a step of controlling any one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and setting the third frame to one of the consecutive fourth frames.

[0121] DESCRIPTION OF SYMBOLS 11 Imaging device, 21 Optical section, 22 Imaging element, 23 DSP circuit, 24 Frame memory, 25 Display section, 26 Recording section, 27 Operation section, 28 Power supply section, 29 Bus line, 51 Imaging section, 52 Signal processing section, 53 Synchronization signal generation section, 54 Control section, 55 Mode switching section, 56 Frame status management section, 57 Frame control section, 58 Thinning counter, 59 Shutter position setting section

Claims

1. An imaging device comprising: a frame control unit that controls one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed; and a frame setting unit that sets the third frame to one of the consecutive fourth frames.

2. The imaging device according to claim 1, further comprising a counter that counts the number of the fourth frame among the consecutive fourth frames, and the frame setting unit sets the third frame by specifying the counter value of the counter.

3. The imaging device according to claim 1, wherein, when an instruction to change the exposure time is received, the frame setting unit resets the fourth frame to the third frame, which is suitable for the changed exposure time.

4. The imaging device according to claim 1, wherein imaging drive is stopped when the fourth frame is controlled by the frame control unit.

5. The imaging device according to claim 1, wherein when the fourth frame is controlled by the frame control section, at least one of the clock oscillation circuit, the voltage boost circuit, and the voltage drop circuit is stopped.

6. An imaging method in which an imaging device controls one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and sets the third frame as one of the consecutive fourth frames.

7. A program for causing a computer that controls an imaging device to execute processing including the step of controlling one of a first frame in which a read operation and a shutter operation are performed, a second frame in which a read operation is performed, a third frame in which a shutter operation is performed, or a fourth frame in which neither a read operation nor a shutter operation is performed, and setting the third frame as one of the consecutive fourth frames.

Images