Image processing device, image processing method and electronic device

Abstract

The present technology relates to an image processing device, an image processing method, and an electronic device, which can suppress the degradation of the image quality of a wide dynamic range (WD) image when a wide dynamic range (WD) image is generated by synthesizing a pixel signal of a large pixel and a pixel signal of a small pixel. The image processing device according to the first aspect of the present technology is provided with: a pixel unit, the pixel unit includes a plurality of pixels, wherein a first pixel of the plurality of pixels is arranged to output a first pixel signal, a second pixel of the plurality of pixels is arranged to output a second pixel signal, and a size of the second pixel is smaller than a size of the first pixel; a temperature detection unit, the temperature detection unit detects the temperature of the pixel unit; and a signal processing unit, the signal processing unit processes the first pixel signal and the second pixel signal corresponding to the detected temperature to reduce noise. For example, the present technology can be applied to a CMOS image sensor.

Description

[0001] This application is a divisional application of a patent application with an application date of March 15, 2018, an invention name of “Image processing device, image processing method and electronic device” and an application number of 201880019966.6. Technical Field

[0002] The present technology relates to an image processing device, an image processing method, and an electronic device, and in particular, to an image processing device, an image processing method, and an electronic device that synthesize pixel signals output from pixels having different light receiving sensitivities to generate a WD (wide dynamic range) image. Background Art

[0003] As a method of generating a WD image, a method of arranging first and second pixels having different sensitivities on a pixel array such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor and synthesizing outputs of the pixels (ie, first and second images) is known.

[0004] Here, examples of setting pixels with different sensitivities include a method of setting pixels with a long exposure time (hereinafter, referred to as long accumulation pixels) and pixels with a short exposure time (hereinafter, referred to as short accumulation pixels) and a method of setting pixels including photoelectric conversion units with a large size such as PD (photodiode) (hereinafter, referred to as large pixels) and pixels including photoelectric conversion units with a small size such as PD (hereinafter, referred to as small pixels) (for example, see Patent Document 1).

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2004-222154 Summary of the invention

[0008] Technical issues

[0009] In the method of setting the long accumulation pixel and the short accumulation pixel, the two pixels have the same structure, and various characteristics depending on the temperature of the two pixels are common. Therefore, when synthesizing the pixel signals of the two pixels, it is not necessary to consider the temperature.

[0010] On the other hand, in the method of setting large pixels and small pixels, the two pixels have different structures. That is, the size of the photoelectric conversion unit such as PD is different between the two pixels, and various characteristics that vary depending on the temperature of the two pixels are different. Examples of various characteristics that vary depending on temperature include sensitivity characteristics, saturation charge amount Qs, conversion efficiency, white spot generation characteristics, and black spot generation characteristics. Hereinafter, these characteristics are collectively referred to as temperature-dependent characteristics.

[0011] Therefore, when synthesizing the pixel signal of the large pixel and the pixel signal of the small pixel, if the temperature dependency characteristic mentioned above is not taken into consideration, a step may occur at the junction of the pixel signal of the large pixel and the pixel signal of the small pixel according to the temperature, or the change of the pixel signal with respect to the change of the brightness of the object may not be kept linear. In this case, deterioration of image quality occurs, such as generation of false color on the generated WD image.

[0012] The present technology has been accomplished in view of the above-mentioned circumstances, and an object thereof is to suppress degradation in the image quality of a WD image when a pixel signal of a large pixel and a pixel signal of a small pixel are synthesized to generate a WD image.

[0013] Solution to the problem

[0014] According to the first aspect of the present technology, an image processing device includes: a first acquisition unit, which acquires a first pixel signal output from a first pixel; a second acquisition unit, which acquires a second pixel signal output from a second pixel having a size smaller than that of the first pixel; a temperature detection unit, which detects temperature; a synthetic gain determination unit, which determines a synthetic gain corresponding to the detected temperature; and a synthesis unit, which synthesizes the first pixel signal and the second pixel signal multiplied by the synthetic gain.

[0015] According to the first aspect of the present technology, the image processing method includes performing the following steps by the image processing device: a first acquisition step of acquiring a first pixel signal output from a first pixel; a second acquisition step of acquiring a second pixel signal output from a second pixel having a size smaller than that of the first pixel; a temperature detection step of detecting a temperature; a synthetic gain determination step of determining a synthetic gain corresponding to the detected temperature; and a synthesis step of synthesizing the first pixel signal and the second pixel signal multiplied by the synthetic gain.

[0016] An electronic device according to a second aspect of the present technology is an electronic device equipped with a solid-state imaging element, wherein the solid-state imaging element includes: a pixel unit, in which a plurality of first pixels and a plurality of second pixels are arranged, and the second pixels are smaller in size than the first pixels; a first acquisition unit, which acquires a first pixel signal output from the first pixel; a second acquisition unit, which acquires a second pixel signal output from the second pixel; a temperature detection unit, which detects temperature; a synthetic gain determination unit, which determines a synthetic gain corresponding to the detected temperature; and a synthesis unit, which synthesizes the first pixel signal and the second pixel signal multiplied by the synthetic gain.

[0017] In the first and second aspects of the present technology, a first pixel signal output from a first pixel is acquired; a second pixel signal output from a second pixel having a size smaller than that of the first pixel is acquired; a temperature is detected; a composite gain corresponding to the detected temperature is determined; and the first pixel signal and the second pixel signal multiplied by the composite gain are composited.

[0018] Advantageous Effects of the Invention

[0019] According to the first aspect of the present technology, it is possible to suppress degradation in the image quality of a WD image obtained by synthesizing a pixel signal of a large pixel and a pixel signal of a small pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram showing a configuration example of an image processing device to which the present technology is applied.

[0021] Figure 2 is a graph showing an example of a temperature-dependent correction amount corresponding to temperature.

[0022] Figure 3 : is a flowchart describing a WD image generation process performed by the image processing device to which the present technology is applied.

[0023] Figure 4 This is a graph showing the results of a conventional synthesis method at 60°C.

[0024] Figure 5 is a graph showing the results of a conventional synthesis method at 105°C.

[0025] Figure 6 : is a graph showing the result of the WD image generation process at 105°C.

[0026] Figure 7 is a diagram describing a schematic configuration example of an endoscopic surgery system.

[0027] Figure 8 is a block diagram describing an example of the functional configuration of a camera head and a camera control unit (CCU).

[0028] Fig. 9 is a block diagram describing a schematic configuration example of a vehicle control system.

[0029] Fig.10 1 is a diagram for assisting in explaining an example of the installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0030] Hereinafter, the best mode for carrying out the present technology (hereinafter, referred to as an embodiment) will be described in detail with reference to the accompanying drawings.

[0031] <Configuration Example of Image Processing Device According to Embodiment of the Present Technology>

[0032] Figure 1 is a block diagram showing a configuration example of an image processing device according to an embodiment of the present technology.

[0033] The image processing device includes a pixel unit 11, a camera control unit 12, a temperature detection unit 13, a large pixel signal processing unit 14, a large pixel table 15, a small pixel signal processing unit 16, a small pixel table 17, a synthesis gain determination unit 18, a synthesis gain table 19 and a synthesis unit 20.

[0034] The pixel unit 11 includes a solid-state imaging element such as a CMOS image sensor in which large pixels and small pixels having different light receiving sensitivities are arranged according to a predetermined rule. It should be noted that the large pixel and the small pixel each include any one of R (red), G (green) and B (blue) color filters or filters for acquiring wavelength bands other than visible light, and generate pixel signals including color components of R, G or B or wavelength bands other than visible light. The pixel unit 11 outputs the pixel signal of the large pixel to the large pixel signal processing unit 14, and outputs the pixel signal of the small pixel to the small pixel signal processing unit 16. In the following, the pixel signal of the large pixel will be referred to as the large pixel signal, and the pixel signal of the small pixel will be referred to as the small pixel signal.

[0035] The imaging control unit 12 controls the operation of the pixel unit 11. Specifically, the imaging control unit 12 controls the exposure time of the large pixel of the pixel unit 11 and the exposure time of the small pixel of the pixel unit 11, and notifies the synthesis gain determination unit 18 of each exposure time.

[0036] The temperature detection unit 13 detects the temperature of the pixel unit 11 , and notifies the large pixel signal processing unit 14 , the small pixel signal processing unit 16 , and the synthesis gain determination unit 18 of the detected temperature.

[0037] The large pixel signal processing unit 14 performs predetermined signal processing corresponding to the temperature on the large pixel signal input from the pixel unit 11 , and outputs the large pixel signal subjected to the signal processing to the synthesis unit 20 .

[0038] For example, the large pixel signal processing unit 14 acquires an OB (optical black body) clamp value that varies depending on a change in temperature from the large pixel table 15, and performs OB clamp processing on the large pixel signal. Further, for example, the large pixel signal processing unit 14 acquires a minimum gain corresponding to a saturation signal amount Qs that varies depending on a change in temperature from the large pixel table 15, and multiplies the large pixel signal by the acquired minimum gain.

[0039] Further, the large pixel signal processing unit 14 notifies the synthesis gain determination unit 18 of the minimum gain which is the multiplier of the large pixel signal.

[0040] In the large pixel table 15, various temperature-dependent parameters based on 45° C. (at which temperature it can be assumed that there is no difference between the temperature-dependent characteristics of large pixels and small pixels) are pre-stored as a table. The various temperature-dependent parameters are used in the signal processing performed by the large pixel signal processing unit 14. It should be noted that, instead of a table, the various temperature-dependent parameters may be stored as a function of temperature.

[0041] The small pixel signal processing unit 16 performs predetermined signal processing corresponding to the temperature on the small pixel signal input from the pixel unit 11 , and outputs the small pixel signal subjected to the signal processing to the synthesis unit 20 .

[0042] For example, the small pixel signal processing unit 16 acquires the OB clamp value that varies depending on the change in temperature from the small pixel table 17, and performs OB clamp processing on the small pixel signal. Further, for example, the small pixel signal processing unit 16 acquires the minimum gain corresponding to the saturation signal amount Qs that varies depending on the change in temperature from the small pixel table 17, and multiplies the small pixel signal by the acquired minimum gain.

[0043] Further, the small-pixel signal processing unit 16 notifies the synthesis gain determination unit 18 of the minimum gain which is the multiplier of the small-pixel signal.

[0044] In the small pixel table 17, various temperature-dependent parameters based on 45° C. (at which temperature it can be assumed that there is no difference in temperature-dependent characteristics between large pixels and small pixels) are pre-stored as a table. The various temperature-dependent parameters are used in the signal processing performed by the small pixel signal processing unit 16. It should be noted that, instead of a table, the various temperature-dependent parameters may be stored as a function of temperature.

[0045] The synthesis gain determination unit 18 acquires a temperature dependent correction amount different for each color component of R, G, and B corresponding to the temperature notified from the temperature detection unit 13 from the synthesis gain table 19. Further, the synthesis gain determination unit 18 determines a synthesis gain for each color component by applying the temperature dependent correction amount, the exposure time of each of the large pixel and the small pixel notified from the imaging control unit 12, the minimum gain notified from the large pixel signal processing unit 14, the minimum gain notified from the small pixel signal processing unit 16, and the synthesis gain at 45° C. stored in advance to the following formula (1), and notifies the determined synthesis gain to the synthesis unit 20.

[0046] Composite gain = Composite gain at 45°C × (exposure time of large pixel / exposure time of small pixel) × (minimum gain of large pixel / minimum gain of large pixel) × temperature-dependent correction amount (1)

[0047] In the synthetic gain table 19 , temperature-dependent correction amounts that have obtained appropriate values ​​corresponding to the temperature are stored in advance as a table.

[0048] Figure 2 An example of a temperature-dependent correction amount corresponding to a temperature is shown. The horizontal axis of the graph represents temperature [°C], and the vertical axis represents the correction amount. In the graph, the correction amount at 45°C as a reference value is 1, and the correction amount is set to increase stepwise as the temperature rises. Conversely, when the temperature drops below 45°C as a reference value, the correction amount is set to decrease.

[0049] It should be pointed out that when the exposure time of the large pixel, the exposure time of the small pixel, the minimum gain of the large pixel and the minimum gain of the small pixel are each a fixed value or a variable that depends on temperature, instead of having the composite gain table 19 save the temperature-related correction amount corresponding to the temperature, the composite gain table 19 can save the pre-calculated composite gain corresponding to the temperature.

[0050] The synthesis unit 20 synthesizes the large pixel signal that has undergone a predetermined signal processing and the result obtained by multiplying the small pixel signal that has undergone a predetermined signal processing by a synthesis gain to generate a WD image. For example, the WD image is generated by adopting the large pixel signal when the brightness of the subject is lower than the brightness when the large pixel signal is saturated, and adopting the small pixel signal multiplied by the synthesis gain when the brightness is equal to or higher than the brightness when the large pixel is saturated. Further, for example, a synthesis rate of 1 or less corresponding to the brightness is set, and a result obtained by multiplying the large pixel signal that has undergone a predetermined signal processing by an addition rate and a result obtained by multiplying the small pixel signal that has undergone a predetermined signal processing by a synthesis gain (1-synthesis rate) are added to generate a WD image.

[0051] <WD Image Generation Processing Performed by Image Processing Apparatus According to Embodiment of the Present Technology>

[0052] Figure 3 is a flowchart describing a WD image generation process performed by the image processing apparatus.

[0053] In step S1, the pixel unit 11 performs imaging according to the control from the imaging control unit 12, and outputs a large pixel signal and a small pixel signal. The large pixel signal is acquired by the large pixel signal processing unit 14, and the small pixel signal is acquired by the small pixel signal processing unit 16. In step S2, the temperature detection unit 13 detects the temperature of the pixel unit 11, and notifies the detected temperature to the large pixel signal processing unit 14, the small pixel signal processing unit 16, and the synthesis gain determination unit 18.

[0054] In step S3, the large pixel signal processing unit 14 performs predetermined signal processing corresponding to the temperature on the large pixel signal input from the pixel unit 11, and outputs the large pixel signal after the signal processing to the synthesis unit 20. In step S4, the small pixel signal processing unit 16 performs predetermined signal processing corresponding to the temperature on the small pixel signal input from the pixel unit 11, and outputs the small pixel signal after the signal processing to the synthesis unit 20.

[0055] In step S5, the synthesis gain determination unit 18 acquires the temperature-dependent correction amount corresponding to the temperature notified from the temperature detection unit 13 from the synthesis gain table 19. Further, the synthesis gain determination unit 18 determines the synthesis gain by applying the temperature-dependent correction amount, the exposure time of each of the large pixel and the small pixel notified from the imaging control unit 12, the minimum gain notified from the large pixel signal processing unit 14, the minimum gain notified from the small pixel signal processing unit 16, and the synthesis gain at 60° C. stored in advance to the formula (1), and notifies the synthesis unit 20 of the determined synthesis gain.

[0056] In step S6, the synthesis unit 20 synthesizes the large pixel signal subjected to the predetermined signal processing and the result obtained by multiplying the small pixel signal subjected to the predetermined signal processing by the synthesis gain to generate a WD image. In this way, the WD image generation process is completed.

[0057] <Effect of WD Image Generation Processing Performed by the Image Processing Apparatus According to the Embodiment of the Present Technology>

[0058] The effects of the above-mentioned WD image generation processing will be described.

[0059] Figure 4The figure shows the change in the brightness of the subject and the change in the pixel signal level when the large pixel signal and the small pixel signal are synthesized by the existing method without considering the temperature dependence characteristics of the large pixel and the small pixel at 45°C as a reference value of the temperature (at which it can be assumed that there is no difference between the temperature dependence characteristics of the large pixel and the small pixel). In the figure, the solid lines RL, GL and BL respectively represent the R, G and B components of the large pixel signal, and the dotted lines RS, GS and BS respectively represent the R, G and B components of the small pixel signal multiplied by the synthesis gain.

[0060] It should be noted that when the large pixel signal and the small pixel signal are synthesized by WD image generation processing at 45°C, the same Figure 4 Similar results were obtained in .

[0061] That is, the solid line RL and the dotted line RS, the solid line GL and the dotted line GS, and the solid line BL and the dotted line BS are connected together without any step, and the pixel signal level can be kept linear with respect to the luminance change.

[0062] Figure 5 The figure shows the change in the brightness of the subject and the change in the pixel signal level when the large pixel signal and the small pixel signal are synthesized by the existing method that does not consider the temperature dependency characteristics of the large pixel and the small pixel at 105°C, which greatly exceeds the reference value of 45°C. In the figure, the solid lines RL, GL and BL respectively represent the R, G and B components of the large pixel signal, and the dotted lines RS, GS and BS respectively represent the R, G and B components of the small pixel signal multiplied by the synthesis gain.

[0063] At 105°C, due to the difference in temperature dependence characteristics between large pixels and small pixels, Figure 5 As shown, the solid line RL and the dotted line RS, the solid line GL and the dotted line GS, and the solid line BL and the dotted line BS are not well connected, and degradation of image quality such as generation of false color occurs near the joints.

[0064] Figure 6 The change in subject brightness and the change in pixel signal level when the large pixel signal and the small pixel signal are synthesized by the above-mentioned WD synthesis processing at 105° C., which greatly exceeds the reference value of 45° C., are shown. In the figure, the solid lines RL, GL, and BL respectively represent the R, G, and B components of the large pixel signal, and the dotted lines RS, GS, and BS respectively represent the R, G, and B components of the small pixel signal multiplied by the synthesis gain.

[0065] At 105°C, although there is a difference between the temperature dependency characteristics of the large pixel and the small pixel, since the synthetic gain as a multiplier of the small pixel signal is determined corresponding to the temperature, Figure 6 As shown, Figure 5In contrast, the solid line RL and the dotted line RS, the solid line GL and the dotted line GS, and the solid line BL and the dotted line BS are smoothly connected, and the pixel signal level can be kept linear with respect to the brightness change. Therefore, it is possible to suppress the degradation of image quality such as the generation of false color near the joint.

[0066] <Modification>

[0067] In the above description, Figure 1 The pixel unit 11 of the image processing device shown in FIG. 1 has already included a solid-state imaging element such as a CMOS image sensor. However, the solid-state imaging element may include Figure 1 That is, the solid-state imaging element may include Figure 1 All the components shown in .

[0068] <Application examples of endoscopic surgery systems>

[0069] The technology according to the present disclosure (the present technology) is applicable to various products. For example, the technology according to the present disclosure can be applied to an endoscopic surgery system.

[0070] Figure 7 : is a diagram showing a schematic configuration example of an endoscopic surgery system to which the technology according to an embodiment of the present disclosure (the present technology) can be applied.

[0071] exist Figure 7 , a state is shown in which an operator (doctor) 11131 is using an endoscopic surgery system 11000 to perform surgery on a patient 11132 on a bed 11133. As shown in the figure, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical tools 11110 such as a pneumoperitoneum tube 11111 and an energy treatment tool 11112, a support arm device 11120 supporting the endoscope 11100 thereon, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0072] The endoscope 11100 includes a lens barrel 11101, a region of a predetermined length from the distal end of the lens barrel is inserted into the body cavity of a patient 11132, and a camera head 11102, the camera head being connected to the proximal end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a rigid scope having a rigid lens barrel 11101 is illustrated. However, the endoscope 11100 may also be configured as a flexible scope having a flexible lens barrel 11101.

[0073] The lens barrel 11101 has an opening at its distal end in which an objective lens is mounted. The light source device 11203 is connected to the endoscope 11100 so that the light generated by the light source device 11203 is introduced into the distal end of the lens barrel 11101 through a light guide extending into the interior of the lens barrel 11101, and irradiated onto an observation object in the body cavity of the patient 11132 through the objective lens. It should be noted that the endoscope 11100 may be a direct-viewing endoscope, or may be a perspective endoscope or a side-view endoscope.

[0074] The optical system and the imaging element are arranged inside the camera head 11102 so that the reflected light (observation light) from the observation object is gathered on the imaging element through the optical system. The observation light is photoelectrically converted by the imaging element to generate an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal is transmitted to the CCU 11201 as raw (RAW) data.

[0075] The CCU 11201 includes a central processing unit (CPU), a graphics processing unit (GPU), etc., and centrally controls the operations of the endoscope 11100 and the display device 11202. Further, for example, the CCU 11201 receives an image signal from the camera head 11102, and performs various image processing such as development processing (demosaic processing) on ​​the image signal to display an image based on the image signal.

[0076] The display device 11202 displays thereon an image based on an image signal that has been image-processed by the CCU 11201 under the control of the CCU 11201 .

[0077] For example, the light source device 11203 includes a light source such as a light emitting diode (LED) and provides the endoscope 11100 with illumination light for imaging the surgical area.

[0078] The input device 11204 is an input interface of the endoscopic surgery system 11000. The user can input various information or instructions to the endoscopic surgery system 11000 through the input device 11204. For example, the user may input instructions to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 11100.

[0079] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 to burn or cut tissue, seal blood vessels, etc. The pneumoperitoneum device 11206 supplies gas into the body cavity of the patient 11132 through the pneumoperitoneum tube 11111 to expand the body cavity in order to ensure the field of view of the endoscope 11100 and to ensure the working space of the operator. The recorder 11207 is a device capable of recording various information related to the operation. The printer 11208 is a device capable of printing various information related to the operation in various forms such as text, images, or graphics.

[0080] It should be noted that the light source device 11203 that provides irradiation light when imaging the surgical area to the endoscope 11100 can be composed of a white light source, for example, a white light source composed of an LED, a laser light source, or a combination thereof. In the case where the white light source is composed of a combination of red, green, and blue (RGB) laser light sources, since the output intensity and output timing of each color (each wavelength) can be controlled with high precision, the white balance of the captured image can be adjusted by the light source device 11203. Further, in this case, if the laser beams from the respective RGB laser light sources are irradiated on the observed object in a time-division manner, the drive of the imaging element of the camera head 11102 is controlled synchronously with the irradiation timing. Then, images corresponding to the R, G, and B colors, respectively, can also be captured in a time-division manner. According to this method, a color image can be obtained even if a color filter is not configured for the imaging element.

[0081] Furthermore, the drive of the light source device 11203 can be controlled so as to change the intensity of the light to be output at predetermined intervals. By controlling the drive of the imaging element of the camera head 11102 in synchronization with the timing of the change in light intensity to acquire images in a time-division manner and synthesize the images, it is possible to create an image with a high dynamic range without underexposed shadows and overexposed highlights.

[0082] Further, the light source device 11203 can be configured to provide light of a predetermined wavelength band corresponding to special light observation. For example, in special light observation, by utilizing the wavelength dependence of light absorption of body tissue, light of a narrow band compared to the irradiation light (i.e., white light) during ordinary observation is irradiated, and narrow-band observation (narrow-band imaging) is performed on predetermined tissues such as blood vessels in the surface layer of the mucosa with high contrast. Alternatively, in special light observation, fluorescence observation for obtaining an image from fluorescence generated by irradiating excitation light can be performed. In fluorescence observation, fluorescence observation (autofluorescence observation) of body tissue can be performed by irradiating excitation light on body tissue, or a fluorescent image can be obtained by locally injecting a reagent such as indocyanine green (ICG) into the body tissue and irradiating excitation light corresponding to the fluorescence wavelength of the reagent on the body tissue. The light source device 11203 can be configured to provide such narrow-band light and / or excitation light suitable for special light observation as described above.

[0083] Figure 8 It is shown Figure 7 A block diagram of a functional configuration example of a camera head 11102 and a CCU 11201 shown in FIG.

[0084] The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a driving unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected via a transmission cable 11400 so as to communicate with each other.

[0085] The lens unit 11401 is an optical system provided at a connection position with the lens barrel 11101. Observation light entering from the distal end of the lens barrel 11101 is guided to the camera head 11102 and introduced into the lens unit 11401. The lens unit 11401 is composed of a combination of a plurality of lenses including a zoom lens and a focus lens.

[0086] The imaging unit 11402 is configured as an imaging element. The number of imaging elements included in the imaging unit 11402 can be one (single-board type) or multiple (multi-board type). For example, in the case where the imaging unit 11402 is configured as a multi-board type, image signals corresponding to each R, G and B are generated by the imaging element, and the image signals can be synthesized to obtain a color image. The imaging unit 11402 can also be configured to have a pair of imaging elements for acquiring a right-eye image signal and a left-eye image signal corresponding to a three-dimensional (3D) display. If a 3D display is performed, then the operator 11131 can more accurately grasp the depth of the living tissue in the surgical area. It should be noted that in the case where the imaging unit 11402 is configured as a stereoscopic type, a plurality of lens unit 11401 systems are set corresponding to each imaging element.

[0087] Further, the camera unit 11402 may not necessarily be disposed on the camera head 11102. For example, the camera unit 11402 may be disposed just behind the objective lens inside the lens barrel 11101.

[0088] The driving unit 11403 is composed of an actuator, and moves the zoom lens and the focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. Therefore, the magnification and focus of the image captured by the imaging unit 11402 can be appropriately adjusted.

[0089] The communication unit 11404 is constituted by a communication device for sending and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal acquired from the imaging unit 11402 to the CCU 11201 as RAW data via the transmission cable 11400.

[0090] In addition, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. For example, the control signal includes information related to imaging conditions, such as information specifying a frame rate of an image to be captured, information specifying an exposure value when capturing an image, and / or information specifying a magnification and a focus of an image to be captured.

[0091] It should be noted that the imaging conditions such as the frame rate, exposure value, magnification or focus may be specified by the user or may be automatically set based on the acquired image signal by the control unit 11413 of the CCU 11201. In the latter case, an automatic exposure (AE) function, an automatic focus (AF) function and an automatic white balance (AWB) function are provided in the endoscope 11100.

[0092] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 through the communication unit 11404 .

[0093] The communication unit 11411 is constituted by a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted thereto from the camera head 11102 through the transmission cable 11400.

[0094] Further, the communication unit 11411 transmits a control signal for controlling the driving of the camera head 11102 to the camera head 11102. The image signal and the control signal may be transmitted through electrical communication, optical communication, or the like.

[0095] The image processing unit 11412 performs various image processing on the image signal in the form of RAW data transmitted thereto from the camera head 11102 .

[0096] The control unit 11413 performs various controls related to image capturing of the operation area, etc. through the endoscope 11100 and display of captured images obtained by image capturing of the operation area, etc. For example, the control unit 11413 creates a control signal for controlling the driving of the camera head 11102 .

[0097] Further, the control unit 11413 controls the display device 11202 to display a captured image in which the surgical area, etc., is imaged based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 can use various image recognition technologies to recognize various objects in the captured image. For example, the control unit 11413 can recognize surgical tools such as surgical forceps, specific living areas, bleeding, fog when using the energy treatment tool 11112, etc. by detecting the shape, color, etc. of the edge of the object contained in the captured image. When the control unit 11413 controls the display device 11202 to display the captured image, it can use the recognition result to display various surgical support information in an overlapping manner with the image of the surgical area. In the case where the surgical support information is displayed in an overlapping manner and presented to the operator 11131, the burden of the operator 11131 can be reduced and the operator 11131 can reliably perform the operation.

[0098] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 to each other is an electric signal cable for electric signal communication, an optical fiber for optical communication, or a composite cable for electric communication and optical communication.

[0099] Here, although in the illustrated example, communication is performed by wired communication using the transmission cable 11400, communication between the camera head 11102 and the CCU 11201 may be performed by wireless communication.

[0100] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. For example, the technology according to the present disclosure is applicable to the endoscope 11100, the camera head 11102 (the imaging unit 11402), and the CCU 11201 (the image processing unit 11412) in the above configuration.

[0101] <Application examples for mobile objects>

[0102] The technology according to the present disclosure (the present technology) is applicable to various products. For example, the technology according to the present disclosure can be implemented as a device installed on any type of mobile object such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, a robot, etc.

[0103] Fig. 9 : is a block diagram showing a schematic configuration example of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present disclosure can be applied.

[0104] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig. 9In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound and image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as the functional configuration of the integrated control unit 12050.

[0105] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for the following devices: a drive force generating device such as an internal combustion engine or a drive motor for generating a drive force for the vehicle; a drive force transmitting mechanism for transmitting the drive force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating a braking force for the vehicle, etc.

[0106] The body system control unit 12020 controls the operation of various devices provided to the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a taillight, a brake light, a turn signal light, or a fog light. In this case, radio waves transmitted from a portable device as a substitute for a key or signals of various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls the door lock device, the power window device, and lights of the vehicle.

[0107] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle having the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the camera unit 12031. The vehicle exterior information detection unit 12030 causes the camera unit 12031 to image the exterior of the vehicle and receives the imaged image. Based on the received image, the vehicle exterior information detection unit 12030 can perform detection processing on objects such as people, vehicles, obstacles, signs, or symbols on the road surface, or detection processing of the distance to these objects.

[0108] The imaging unit 12031 is an optical sensor for receiving light and outputting an electrical signal corresponding to the amount of light received. The imaging unit 12031 may output the electrical signal as an image, or may output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0109] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 is connected to a driver state detection unit 12041 that detects the driver's state. The driver state detection unit 12041 includes, for example, a camera that images the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off.

[0110] The microcomputer 12051 can calculate the control target value of the driving force generating device, the steering mechanism or the braking device based on the information about the inside or outside of the vehicle (the information is obtained by the outside information detection unit 12030 or the inside information detection unit 12040), and output the control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform the coordinated control aimed at realizing the functions of the advanced driver assistance system (ADAS), which includes: collision avoidance or collision mitigation of the vehicle, following driving based on the vehicle-to-vehicle distance, vehicle speed maintenance driving, vehicle collision warning or vehicle lane departure warning, etc.

[0111] In addition, the microcomputer 12051 can perform collaborative control intended for autonomous driving, which enables the vehicle to drive autonomously by controlling a driving force generating device, a steering mechanism, or a braking device, etc. based on information about the interior or exterior of the vehicle (the information is obtained by the exterior information detection unit 12030 or the interior information detection unit 12040), without relying on the driver's operation.

[0112] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 based on information about the exterior of the vehicle, which is obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare by controlling the headlights to change from high beam to low beam according to the position of the preceding vehicle or the oncoming vehicle detected by the exterior information detection unit 12030.

[0113] The audio and video output unit 12052 transmits an output signal of at least one of audio and video to an output device that can visually or auditorily notify the vehicle's passengers or the outside of the vehicle of the information. Fig. 9 In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are shown as output devices. For example, the display portion 12062 may include at least one of an in-vehicle display and a head-up display.

[0114] Fig.10 12031 is a diagram showing an example of the installation position of the camera unit 12031.

[0115] exist Fig.10 In the figure, the camera unit 12031 includes camera units 12101, 12102, 12103, 12104 and 12105.

[0116] The camera units 12101, 12102, 12103, 12104 and 12105 are, for example, arranged at positions on the front nose, side mirrors, rear bumper and rear door of the vehicle 12100 and at positions on the upper part of the windshield inside the vehicle. The camera unit 12101 arranged on the front nose and the camera unit 12105 arranged on the upper part of the windshield inside the vehicle mainly obtain images in front of the vehicle 12100. The camera units 12102 and 12103 arranged on the side mirrors mainly obtain images on the sides of the vehicle 12100. The camera unit 12104 arranged on the rear bumper or rear door mainly obtains images at the rear of the vehicle 12100. The front images obtained by the camera units 12101 and 12105 are mainly used to detect front vehicles, pedestrians, obstacles, signals, traffic signs or lanes, etc.

[0117] By the way, Fig.10 An example of the photographing range of the camera units 12101 to 12104 is shown. The photographing range 12111 indicates the photographing range of the camera unit 12101 provided to the front nose. The photographing ranges 12112 and 12113 respectively indicate the photographing ranges of the camera units 12102 and 12103 provided to the side mirrors. The photographing range 12114 indicates the photographing range of the camera unit 12104 provided to the rear bumper or the rear door. For example, by superimposing the image data photographed by the camera units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above is obtained.

[0118] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0119] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the time change of the distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, thereby extracting the nearest three-dimensional object as the leading vehicle, in particular, the three-dimensional object exists on the driving path of the vehicle 12100 and travels in substantially the same direction as the vehicle 12100 at a predetermined speed (e.g., equal to or greater than 0 km / h). In addition, the microcomputer 12051 can pre-set the inter-vehicle distance to be maintained in front of the leading vehicle, and perform automatic braking control (including follow-up stop control) or automatic acceleration control (including follow-up start control), etc. Therefore, cooperative control intended for automatic driving can be performed, which allows the vehicle to travel autonomously without relying on the driver's operation, etc.

[0120] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on the distance information obtained from the camera units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be visually identified by the driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually identify. Then, the microcomputer 12051 determines the collision risk representing the risk of collision with each obstacle. When the collision risk is equal to or higher than the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering through the drive system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collision.

[0121] At least one of the camera units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 may identify a pedestrian by determining whether there is a pedestrian in the captured images of the camera units 12101 to 12104. For example, such identification of pedestrians is performed by a program that extracts feature points in the captured images of the camera units 12101 to 12104 that are infrared cameras and a program that determines whether it is a pedestrian by performing pattern matching processing on a series of feature points that represent the outline of an object. When the microcomputer 12051 determines that there is a pedestrian in the captured images of the camera units 12101 to 12104 and thus identifies the pedestrian, the sound image output unit 12052 controls the display unit 12062 so that a square contour line for emphasis is displayed in a manner superimposed on the identified pedestrian. The sound image output unit 12052 may also control the display unit 12062 so that an icon or the like representing a pedestrian is displayed at a desired position.

[0122] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. For example, the technology according to the present disclosure is applicable to the camera unit 12031 and the like in the above configuration.

[0123] It should be noted that the embodiments of the present technology are not limited to the above-described embodiments and various modifications may be made without departing from the essence of the present technology.

[0124] It should be noted that the present technology can also adopt the following configurations.

[0125] (1) An image processing device comprising:

[0126] a first acquisition unit, which acquires a first pixel signal output from a first pixel;

[0127] a second acquisition unit that acquires a second pixel signal output from a second pixel having a size smaller than that of the first pixel;

[0128] a temperature detection unit that detects temperature;

[0129] a synthetic gain determination unit that determines a synthetic gain corresponding to the detected temperature; and

[0130] A synthesis unit synthesizes the first pixel signal and the second pixel signal multiplied by the synthesis gain.

[0131] (2) The image processing device according to (1) above further comprises:

[0132] A pixel unit, a plurality of the first pixels and a plurality of the second pixels are arranged in the pixel unit.

[0133] (3) The image processing device according to (1) or (2) above, wherein

[0134] The first pixel and the second pixel have different temperature dependence characteristics.

[0135] (4) The image processing device according to any one of (1) to (3) above, further comprising:

[0136] A table storing temperature-dependent corrections associated with temperatures, wherein

[0137] The synthesized gain determination unit acquires the temperature-dependent correction amount corresponding to the detected temperature from the table, and calculates the synthesized gain by using the acquired temperature-dependent correction amount.

[0138] (5) The image processing device according to any one of (1) to (3) above, further comprising:

[0139] A table storing the synthesized gain associated with temperature, wherein

[0140] The synthesized gain determination unit acquires the synthesized gain corresponding to the detected temperature from the table.

[0141] (6) The image processing device according to any one of (1) to (5) above, further comprising:

[0142] a first signal processing unit that performs predetermined signal processing on the first pixel signal; and

[0143] a second signal processing unit that performs predetermined signal processing on the second pixel signal, wherein

[0144] The synthesis unit synthesizes the first pixel signal subjected to the predetermined signal processing and the second pixel signal subjected to the predetermined signal processing multiplied by the synthesis gain.

[0145] (7) The image processing device according to (6) above, wherein

[0146] At least one of the first signal processing unit and the second signal processing unit performs an OB (Optical Black Body) clamp process corresponding to the detected temperature.

[0147] (8) The image processing device according to (6) or (7) above, wherein

[0148] At least one of the first signal processing unit and the second signal processing unit performs a process of multiplying by a minimum gain corresponding to the detected temperature.

[0149] (9) An image processing method of an image processing device, comprising:

[0150] The image processing device performs the following steps:

[0151] A first acquisition step of acquiring a first pixel signal output from a first pixel;

[0152] a second acquisition step of acquiring a second pixel signal output from a second pixel having a size smaller than that of the first pixel;

[0153] A temperature detection step of detecting a temperature;

[0154] a synthetic gain determining step of determining a synthetic gain corresponding to the detected temperature; and

[0155] A synthesis step of synthesizing the first pixel signal and the second pixel signal multiplied by the synthesis gain.

[0156] (10) An electronic device equipped with a solid-state imaging element, the solid-state imaging element comprising:

[0157] A pixel unit, a plurality of first pixels and a plurality of second pixels are arranged in the pixel unit, and the second pixels are smaller in size than the first pixels;

[0158] a first acquisition unit, which acquires a first pixel signal output from the first pixel;

[0159] a second acquisition unit, which acquires a second pixel signal output from the second pixel;

[0160] a temperature detection unit that detects temperature;

[0161] a synthetic gain determination unit that determines a synthetic gain corresponding to the detected temperature; and

[0162] A synthesis unit synthesizes the first pixel signal and the second pixel signal multiplied by the synthesis gain.

[0163] Reference numerals list

[0164] 11 pixel unit

[0165] 12 Camera control unit

[0166] 13. Temperature detection unit

[0167] 14 Large pixel signal processing unit

[0168] 15 large pixel table

[0169] 16 small pixel signal processing unit

[0170] 17 Small Pixel Table

[0171] 18 Synthesized gain determination unit

[0172] 19 Crafting Table

[0173] 20 Synthesis Units

Claims

1. An image processing device, comprising: A pixel unit, the pixel unit comprising a plurality of pixels, wherein a first pixel of the plurality of pixels is configured to output a first pixel signal, a second pixel of the plurality of pixels is configured to output a second pixel signal, and a size of the second pixel is smaller than a size of the first pixel; a temperature detection unit, wherein the temperature detection unit detects the temperature of the pixel unit; a signal processing unit, wherein the signal processing unit processes the first pixel signal and the second pixel signal corresponding to the detected temperature; a synthetic gain determination unit that determines a synthetic gain corresponding to the detected temperature; and A synthesis unit synthesizes the first pixel signal and the second pixel signal multiplied by the synthesis gain.

2. The image processing device according to claim 1, wherein: The image processing device stores a temperature-dependent correction amount that differs based on the detected temperature.

3. The image processing device according to claim 2, wherein: The correction amount for a high temperature is greater than the correction amount for a low temperature.

4. The image processing device according to any one of claims 1 to 3, wherein: The signal processing unit includes a first signal processing unit that performs processing on the first pixel signal corresponding to the detected temperature and a second signal processing unit that performs processing on the second pixel signal corresponding to the detected temperature.

5. The image processing device according to any one of claims 1 to 3, wherein The first pixel and the second pixel have different temperature dependence characteristics.

6. The image processing device according to claim 2 or 3, wherein: The corrections are stored in a table, where The composite gain determination unit acquires the correction amount corresponding to the detected temperature from the table, and calculates the composite gain by using the correction amount.

7. The image processing apparatus according to claim 4, wherein At least one of the first signal processing unit and the second signal processing unit performs an optical black body clamping process corresponding to the detected temperature.

8. The image processing apparatus according to claim 4, wherein At least one of the first signal processing unit and the second signal processing unit performs a process of multiplying by a minimum gain corresponding to the detected temperature.

9. An image processing method of an image processing device, comprising: Acquire a first pixel signal output from a first pixel included in a pixel unit of the image processing device; Acquire a second pixel signal output from a second pixel included in the pixel unit, wherein the size of the second pixel is smaller than the size of the first pixel; detecting the temperature of the pixel unit; performing processing corresponding to the detected temperature on the first pixel signal and the second pixel signal; determining a composite gain corresponding to the detected temperature; and The first pixel signal and the second pixel signal multiplied by the synthesis gain are synthesized. 10 . An electronic device equipped with a solid-state imaging element, the solid-state imaging element comprising the image processing device according to claim 1 .

Images