Medical imaging system, medical imaging device, and operating method

Abstract

The present invention relates to a medical imaging system, a medical imaging device, and an operating method that enable the capture of medical images that can better observe the surgical field. The medical imaging system includes: a first imaging element for receiving NIR light and outputting a first image signal; a second imaging element for receiving ordinary light and outputting a second image signal; a third imaging element for receiving at least ordinary light and outputting a third image signal; and a signal processing unit for using at least two of the first image signal, the second image signal, and the third image signal to perform depth of field extension processing for generating an extended depth of field image. The optical path length from the mounting surface to the first imaging element is configured to be longer than the optical path length from the mounting surface to the second imaging element, and shorter than the optical path length from the mounting surface to the third imaging element. The present technology can be applied to, for example, a medical imaging system capable of capturing EDOF images.

Description

Technical Field

[0001] The present disclosure relates to a medical imaging system, a medical imaging apparatus, and an operating method, and more particularly, to a medical imaging system, a medical imaging apparatus, and an operating method that enable capturing a medical image in which a surgical field can be better observed. Background Art

[0002] Typically, medical observation devices such as endoscopes or microscopes acquire images with a shallow depth of field. On the other hand, in surgeries using endoscopes, microscopes, etc., the surgical field is often deep, so it is necessary to capture medical images with a deep depth of field.

[0003] In view of this, as disclosed in Patent Document 1, an endoscope, a microscope, or the like has been proposed that uses an extended depth of field (EDOF) optical system with a phase mask to extend the depth of field to increase the depth of field.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-158764 Summary of the Invention

[0007] Problems to be solved by the present invention

[0008] Meanwhile, resolution and depth of field are determined by the F-number of the optical system, and there is a trade-off between them. Specifically, when the F-number is set to a high value and the resolution is increased, an image with a shallow depth of field is obtained, making it impossible to obtain image quality sufficient for observing the surgical field. Therefore, it is necessary to capture medical images that have both high resolution and a deep depth of field, and that enable better observation of the surgical field.

[0009] The present invention has been made in view of the above circumstances, and an object of the present invention is to enable the capture of medical images that enable better observation of the surgical field.

[0010] Solution to the problem

[0011] According to one aspect of the present disclosure, a medical imaging system and a medical imaging device include: a first imaging element that receives light of a first wavelength band in incident light incident from a mounting surface and outputs a first image signal; a second imaging element that receives light of a second wavelength band different from the first wavelength band in the incident light and outputs a second image signal; a third imaging element that receives light of at least the second wavelength band in the incident light and outputs a third image signal; and a signal processing unit that performs depth of field extension processing to generate an extended depth of field image obtained by extending the depth of field using at least two of the first image signal, the second image signal, and the third image signal, wherein the optical path length from the mounting surface to the first imaging element is longer than the optical path length from the mounting surface to the second imaging element, and shorter than the optical path length from the mounting surface to the third imaging element.

[0012] An operating method according to one aspect of the present disclosure is an operating method for a medical imaging system, the method comprising: receiving light of a first wavelength band in incident light incident from a mounting surface and outputting a first image signal; receiving light of a second wavelength band different from the first wavelength band in the incident light and outputting a second image signal; receiving light of at least a second wavelength band in the incident light and outputting a third image signal; and performing depth of field extension processing to generate an extended depth of field image obtained by extending the depth of field using at least two of the first image signal, the second image signal, and the third image signal, wherein the medical imaging system is configured such that the optical path length from the mounting surface to the first imaging element is longer than the optical path length from the mounting surface to the second imaging element and shorter than the optical path length from the mounting surface to the third imaging element.

[0013] According to one aspect of the present disclosure, a first image signal is output by receiving light in a first wavelength band from incident light incident from a mounting surface, a second image signal is output by receiving light in a second wavelength band different from the first wavelength band from the incident light, a third image signal is output by receiving at least light in the second wavelength band from the incident light, and depth of field extension processing is performed using at least two of the first, second, and third image signals to generate an extended depth of field image obtained by extending the depth of field. Furthermore, the optical path length from the mounting surface to the first imaging element is set to be longer than the optical path length from the mounting surface to the second imaging element, and shorter than the optical path length from the mounting surface to the third imaging element. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram showing a configuration example of a medical imaging system to which the present technology is applied according to the first embodiment.

[0015] Figure 2 is a diagram for describing the optical distance from the mount.

[0016] Figure 3 is a flowchart for describing a method for capturing medical images.

[0017] Figure 4 is a block diagram showing a configuration example of a medical imaging system according to the second embodiment.

[0018] Figure 5 is a block diagram showing a configuration example of a medical imaging system according to the third embodiment.

[0019] Figure 6 is a block diagram showing a configuration example of an imaging device.

[0020] Figure 7 is a diagram illustrating a usage example of an image sensor.

[0021] Figure 8 is a diagram showing a schematic configuration example of an endoscope system.

[0022] Figure 9 It shows Figure 8 The block diagram shows an example of the functional configuration of a camera head and a camera control unit (CCU).

[0023] Figure 10 is a diagram illustrating an example of a schematic configuration of a microsurgery system. DETAILED DESCRIPTION

[0024] Specific embodiments to which the present technology is applied will be described in detail below with reference to the accompanying drawings.

[0025] <First Configuration Example of Medical Imaging System>

[0026] Figure 1 is a block diagram showing a configuration example of a medical imaging system to which the present technology is applied according to the first embodiment.

[0027] Figure 1 The medical imaging system 11 shown in FIG is assumed to be used for, for example, an endoscope or a surgical microscope, and includes a medical imaging device 12 and a signal processing device 13. For example, in the medical imaging system 11, light from a surgical field (not shown) enters the medical imaging device 12 along an optical axis indicated by a dashed line, and an image obtained by imaging the surgical field is supplied from the medical imaging device 12 to the signal processing device 13. Then, an EDoF image obtained by performing depth-of-field extension processing in the signal processing device 13, a color image obtained from normal light, and an IR image obtained from infrared light are output from the medical imaging system 11.

[0028] The medical imaging apparatus 12 includes a mount 21 , a spectroscopic system 22 , and imaging modules 23 - 1 to 23 - 3 .

[0029] The mount 21 is, for example, a connection portion that connects an imaging lens having a focusing function to focus on a desired affected part in the surgical field to the medical imaging device 12 .

[0030] The optical splitting system 22 splits the light incident on the medical imaging apparatus 12 via the mount 21 into light to be incident on the imaging module 23 - 1 , light to be incident on the imaging module 23 - 2 , and light to be incident on the imaging module 23 - 3 .

[0031] The optical separation system 22 includes a first prism 31 , a second prism 32 , a third prism 33 , a wavelength separation mirror 34 , and a half mirror 35 .

[0032] For example, in the optical splitting system 22, the wavelength separation mirror 34 is provided between the first prism 31 and the second prism 32, so that the light reflected by the wavelength separation mirror 34 is incident on the imaging module 23-2 through the first prism 31. In addition, in the optical splitting system 22, the half mirror 35 is provided between the second prism 32 and the third prism 33, through which the light transmitted through the wavelength separation mirror 34 and reflected by the half mirror 35 is incident on the imaging module 23-3 through the second prism 32, and the light transmitted through the half mirror 35 is incident on the imaging module 23-1 through the third prism 33.

[0033] The wavelength separation mirror 34 separates light having a predetermined wavelength. For example, the wavelength separation mirror 34 separates infrared light and visible light by reflecting light in the infrared wavelength range and transmitting light in the visible wavelength range. Thus, the infrared light separated by the wavelength separation mirror 34 is incident on the imaging module 23-2. Furthermore, the visible light separated by the wavelength separation mirror 34 is reflected by the half mirror 35 and enters the imaging module 23-3. It also passes through the half mirror 35 and enters the imaging module 23-1.

[0034] The half mirror 35 can be set so that the reflectivity and the transmittance are the same (50:50). The half mirror 35 can also be set, for example, so that the reflectivity is higher than the transmittance, or the reflectivity is lower than the transmittance. By using a half mirror 35 in which the reflectivity is set higher than the transmittance, the amount of light entering the imaging module 23-3 can be greater than the amount of light entering the imaging module 23-1. By using a half mirror 35 in which the reflectivity is set lower than the transmittance, the amount of light entering the imaging module 23-1 can be greater than the amount of light entering the imaging module 23-3. It should be noted that since it is basically difficult to make the reflectivity and transmittance of the half mirror 35 variable, the setting of the reflectivity and transmittance is a parameter adjustment item in the design stage.

[0035] The imaging module 23-1 includes an imaging element 43-1 and a filter 44-1, which are housed in a space enclosed by a housing 41-1 and a cover glass 42-1. In addition, the imaging module 23-2 and the imaging module 23-3 have similar configurations.

[0036] Filter 44-1 of imaging module 23-1 is an RGB filter with a Bayer array as shown in the figure, and imaging element 43-1 captures a color image via filter 44-1. Filter 44-2 of imaging module 23-2 is an IR filter that transmits infrared light, and imaging element 43-2 captures an IR image via filter 44-2. Filter 44-3 of imaging module 23-3 is an RGB filter with a Bayer array as shown in the figure, and imaging element 43-3 captures a color image via filter 44-3.

[0037] Furthermore, the medical imaging device 12 is configured so that the optical distance from the mount 21 to the imaging module 23-1 is shorter than the optical distance from the mount 21 to the imaging module 23-3. With this configuration, when the focus is adjusted so that the desired affected area is focused near the color image captured by the imaging module 23-1, the focus shifts in the image captured by the imaging module 23-3. That is, in the color image captured by the imaging module 23-3, the focused position is located at a point farther from the desired affected area.

[0038] Hereinafter, the color image captured by the imaging module 23-1 is referred to as a near color image, and the color image captured by the imaging module 23-3 is referred to as a far color image as appropriate. Therefore, in the medical imaging system 11, the near color image output from the imaging module 23-1, the IR image output from the imaging module 23-2, and the far color image output from the imaging module 23-3 are provided from the medical imaging device 12 to the signal processing device 13.

[0039] The signal processing device 13 performs depth-of-field extension processing to generate an EDOF image in which the depth of field is extended by selecting a pixel value having a higher contrast among the pixel values ​​of the near-color image and the pixel values ​​of the far-color image at the same pixel position. As a selection method for selecting the pixel value having a higher contrast, for example, a method of extracting an edge component from each of the near-color image and the far-color image and extracting an image region having the largest edge in each image region can be used.

[0040] Furthermore, the signal processing device 13 can perform an enhancement filter process on the image without offset to eliminate the contrast difference that can be seen from the contrast difference map, and output the image after the enhancement filter process as an EDOF image. It should be noted that since its purpose is to increase the depth of field, weighted addition is not performed in the enhancement filter process. In addition, weighted addition can be performed to smooth the switching portion between the near color image and the far color image. For example, the following process can be performed: based on the near color image, an unsharp mask is applied to the image area where the far color image is selected, and the intensity of the emphasis component (far-near) is changed and added. This makes it possible to control the emphasis component and avoid image quality degradation.

[0041] In the medical imaging system 11, by setting the reflectivity and transmittance of the half mirror 35 so that the amount of light entering the imaging module 23-1 is greater than the amount of light entering the imaging module 23-3, a near-color image can be made brighter. In the medical imaging system 11, by setting the reflectivity and transmittance of the half mirror 35 so that the amount of light entering the imaging module 23-3 is greater than the amount of light entering the imaging module 23-1, a far-color image can be made brighter. As described above, in the medical imaging system 11, by appropriately increasing the sensitivity of the far-color image or the near-color image, a more satisfactory signal-to-noise ratio can be achieved.

[0042] As the imaging module 23-2 that captures IR images, a monochrome image sensor is assumed to be used, but a color image sensor may be used instead of the monochrome image sensor. Here, the description assumes that an IR image is captured. On the other hand, IR light used, for example, in fluorescence observation is NIR light (near infrared: a region included in visible light by definition) of around 780 to 800 nm. Therefore, in the case where a color image sensor is used as the imaging module 23-2, an image can be captured by receiving most of the NIR light. Therefore, the medical imaging device 12 can have a structure that uses a color image sensor that has a common configuration with the imaging modules 23-1 to 23-3.

[0043] In the medical imaging system 11, the imaging modules 23-1 to 23-3 are preferably arranged (in terms of optical distance from the mount 21) in the order of the imaging module 23-1 that captures a near color image, the imaging module 23-2 that captures an IR image, and the imaging module 23-3 that captures a far color image. Figure 2 As shown in A of FIG. Moreover, in the case where the imaging module 23-2 captures a V (violet: light with an excitation wavelength of ultraviolet wavelength) image instead of an IR image, the imaging modules are preferably arranged in the order of the imaging module 23-1 that captures a near color image, the imaging module 23-2 that captures a V image, and the imaging module 23-3 that captures a far color image, as shown in FIG. Figure 2As shown in B. In addition, the optical distance from the mount 21 is preferably within an acceptable range as the flange rear distance.

[0044] In the above configuration, preferably, the imaging module 23-2 that captures the IR image is located on the side with a longer optical distance relative to the center between the imaging module 23-1 and the imaging module 23-2 ( Figure 2 In addition, in the configuration in which the imaging module 23-2 captures the V image, it is preferable that the imaging module 23-2 that captures the V image is located on the side with a shorter optical distance relative to the center between the imaging module 23-1 and the imaging module 23-2 (on the right side). Figure 2 on the left side of the ).

[0045] The imaging module 23-2 may be configured to be movable so that the capture of IR images and the capture of V images can be switched. With this configuration, if the imaging module 23-2 is a color image sensor, both IR images and V images can be captured by moving the imaging module 23-2. It should be noted that even in a configuration where the imaging module 23-2 is immovable, both IR images and V images can be captured by changing the correction parameters.

[0046] The medical imaging system 11 can switch between an EDOF mode for outputting an EDOF image generated using a near-color image and a far-color image, an EDOF+IR mode for outputting an EDOF image generated using a near-color image, a far-color image, and an IR image, a near-IR mode for outputting a near-color image and an IR image, and a far-IR mode for outputting a far-color image and an IR image. For example, the medical imaging system 11 can switch between these modes appropriately according to the details of the surgery and select one of these modes.

[0047] When the imaging module 23-2 that captures the IR image is a color image sensor, the medical imaging system 11 can generate an EDOF image using the near color image and the IR image, the far color image and the IR image, or the near color image, the far color image, and the IR image. The medical imaging system 11 can simultaneously output both the EDOF image generated from the near color image and the IR image and the EDOF image generated from the far color image and the IR image.

[0048] In the case where the imaging module 23-2 is not a color image sensor but a monochrome image sensor, the medical imaging system 11 can extract edge components from the IR image captured by the imaging module 23-2 and perform correction corresponding to the edge components on the near color image and the far color image. The medical imaging system 11 can extend the depth of field of the IR image by performing depth-of-field extension processing on the IR image using the near color image or the far color image.

[0049] Will refer to Figure 3 The flowchart of FIG. 1 describes a method for capturing medical images by the medical imaging system 11 .

[0050] In step S11 , the imaging module 23 - 1 outputs a near-color image acquired by receiving the light in the visible wavelength range transmitted through the wavelength separation mirror 34 to the signal processing device 13 .

[0051] In step S12 , the imaging module 23 - 2 outputs the IR image acquired by receiving the light in the infrared wavelength range reflected by the wavelength separation mirror 34 to the signal processing device 13 .

[0052] In step S13 , the imaging module 23 - 3 outputs a far-color image acquired by receiving the light in the visible wavelength range transmitted through the wavelength separation mirror 34 to the signal processing device 13 .

[0053] In step S14, the signal processing device 13 performs depth-of-field extension processing to generate an EDOF image in which the depth of field is extended using the near color image and the far color image. The signal processing device 13 then outputs the EDOF image and the IR image.

[0054] <Second Configuration Example of Medical Imaging System>

[0055] Figure 4 is a block diagram showing a configuration example of a medical imaging system to which the present technology is applied according to the second embodiment. Figure 4 In the medical imaging system 11A shown in FIG. Figure 1 The same components as those in the medical imaging system 11 are identified by the same reference numerals, and detailed descriptions thereof will be omitted.

[0056] like Figure 4 As shown, the medical imaging system 11A includes a medical imaging device 12A and a signal processing device 13. The medical imaging device 12A includes a wavelength separation mirror 34A constituting a spectroscopic system 22A, and the wavelength separation mirror 34A and the signal processing device 13 are connected. Figure 1 The wavelength separation mirror 34 in the medical imaging device 12A is different in that it reflects light in the visible wavelength range and transmits light in the infrared wavelength range. Therefore, in the medical imaging device 12A, the imaging module 23A-2, on which the visible light reflected by the wavelength separation mirror 34A is incident, captures a color image, and the imaging modules 23A-1 and 23A-3, on which the infrared light transmitted by the wavelength separation mirror 34A is incident, capture an IR image.

[0057] That is, the filter 44A-1 of the imaging module 23A-1 and the filter 44A-3 of the imaging module 23A-3 are IR filters. In addition, the filter 44A-2 of the imaging module 23A-2 is an RGB filter having a Bayer array as shown in the figure.

[0058] The medical imaging device 12A is configured so that the optical distance from the mount 21 to the imaging module 23A-3 is longer than the optical distance from the mount 21 to the imaging module 23A-1. With this configuration, when the focus is adjusted so that the vicinity of the desired affected part is in focus in the IR image captured by the imaging module 23A-1, the focus shifts in the image captured by the imaging module 23A-3. In other words, the focus position is located at a point farther from the desired affected part in the IR image captured by the imaging module 23A-3.

[0059] In the medical imaging system 11A configured as described above, the signal processing device 13 can generate an IR EDOF image in which the depth of field is extended by performing depth-of-field extension processing using the IR image captured by the imaging module 23A-1 and the IR image captured by the imaging module 23A-3. That is, the medical imaging system 11A can output the color image captured by the imaging module 23A-2, the IR images captured by the imaging modules 23A-1 and 23A-3, and the IR EDOF image.

[0060] In the medical imaging system 11A, the reflectivity and transmittance of the half mirror 35 can be as shown in FIG. Figure 1 With this configuration, the IR image captured by the imaging module 23A-3 can be made brighter, or the IR image captured by the imaging module 23A-1 can be made brighter, and a more satisfactory SN ratio can be achieved by appropriately increasing the sensitivity of either IR image.

[0061] <Third Configuration Example of Medical Imaging System>

[0062] Figure 5 is a block diagram showing a configuration example of a medical imaging system to which the present technology is applied according to the third embodiment. Figure 5 In the medical imaging system 11B shown in FIG. Figure 1 The same components as those of the medical imaging system 11 in FIG. 1 are identified by the same reference numerals, and detailed descriptions thereof will be omitted.

[0063] like Figure 5 As shown, the medical imaging system 11B includes a medical imaging device 12B and a signal processing device 13. In the medical imaging device 12B, the spectroscopic system 22B and the Figure 1 The difference between the optical splitting system 22 in FIG. 1 and FIG. 2 is that the half mirror 35 is arranged between the first prism 31 and the second prism 32, and the wavelength separation mirror 34 is arranged between the second prism 32 and the third prism 33. Figure 1The medical imaging device 12 in FIG. 1 is different in that the filter 44B- 2 of the imaging module 23B- 2 is an RGB filter with a Bayer array as shown, and the filter 44B- 3 of the imaging module 23B- 3 is an IR filter.

[0064] Therefore, in the medical imaging device 12B, the light before being separated in wavelength by the wavelength separation mirror 34 is reflected by the half mirror 35 and enters the imaging module 23B-2, and a color image is captured. That is, both the normal light component and the IR component can be extracted from the color image captured by the imaging module 23B-2.

[0065] Furthermore, the medical imaging device 12B is configured so that the optical distance from the mount 21 to the imaging module 23B-2 is longer than the optical distance from the mount 21 to the imaging module 23-1. With this configuration, when the focus is adjusted so that the vicinity of the desired affected part is in focus in the color image captured by the imaging module 23-1, the focus shifts in the image captured by the imaging module 23B-2. In other words, the focus position is located at a point farther from the desired affected part in the color image captured by the imaging module 23B-2.

[0066] Hereinafter, the color image captured by the imaging module 23-1 is referred to as a near color image, and the color image captured by the imaging module 23B-2 is referred to as a far color image, as necessary. Therefore, in the medical imaging system 11B, the near color image output from the imaging module 23-1, the far color image output from the imaging module 23B-2, and the IR image output from the imaging module 23B-3 are provided from the medical imaging device 12B to the signal processing device 13.

[0067] The medical imaging system 11B configured as described above can be used to emit normal light and IR light in a time-division manner. For example, when emitting normal light, the signal processing device 13 performs depth-extended field processing to generate an EDOF image of normal light using the normal light components of the near and far color images. On the other hand, when emitting IR light, the signal processing device 13 performs depth-extended field processing to generate an IR EDOF image using the IR image and the IR components of the far color image.

[0068] Note that although a description has been given assuming that IR light is emitted, image capturing can also be performed by the imaging module 23B-2 including the filter 44B-2 as an RGB filter because substantially red or NIR light having a peak at 760 nm is used.

[0069] The depth of field extension process performed by the signal processing device 13 to generate an EDOF image in the medical imaging system 11B will be described.

[0070] First, the signal processing device 13 combines the near-color image output from the imaging module 23 - 1 and the IR image output from the imaging module 23B- 3 .

[0071] Next, the signal processing device 13 generates a selection map in which the pixel values ​​of pixels having higher contrast are selected from the pixel values ​​of pixels at the same pixel position in the composite image obtained by combining the near color image and the IR image, and the far color image output from the imaging module 23B-2, and generates a contrast difference map indicating the difference in contrast between the pixels. For pixels in which the far color image is selected, the selection map is a table of the same size as the input image with a value of 1. The contrast difference map is a table of the same size as the input image with a contrast difference.

[0072] The signal processing device 13 then generates an EDOF image for normal light and an EDOF image for IR from the selection map through first and second processing. In the first processing, for pixels with a value of 0 in the selection map, an image without offset is output. In the second processing, for pixels with a value of 1 in the selection map, the normal light and IR images are separated from the far color image based on the ratio of the near color image and IR image when generating a composite image, and the corresponding images are output.

[0073] As described above, the medical imaging system 11 according to each of the above embodiments can simultaneously acquire three images: a near color image, an IR image, and a far color image (an image with different focal points for generating an EDOF image). Therefore, the medical imaging system 11 can obtain two images: an image with high resolution and a deep depth of field, and an image with another light source.

[0074] Therefore, the medical imaging system 11 can simultaneously observe images with high resolution and a deep depth of field using ordinary light, and observe deep blood vessels and the like using IR images. For example, while focusing on an image that has not been subjected to EDOF, another image with a deep depth of field can also be focused. Therefore, the medical imaging system 11 can eliminate the need for focus adjustment when switching between ordinary light and IR. Furthermore, when superimposed, focused images from both light sources can be obtained.

[0075] Note that, for example, Japanese Patent Application Laid-Open No. 2017-209154 discloses a technique for performing EDOF by changing the optical path length to change the focus position, but does not describe separation of light of different wavelengths. Alternatively, the medical imaging system 11 may be adapted for EDOF with a combination of multiple light sources.

[0076] <Configuration Example of Electronic Equipment>

[0077] The above-described medical imaging apparatus 12 can be applied to various electronic devices including an imaging system such as a digital still camera and a digital camera, a mobile phone with an imaging function, and another device with an imaging function.

[0078] Figure 6 is a block diagram showing a configuration example of an imaging device mounted on an electronic device.

[0079] like Figure 6 As shown, the imaging device 101 includes an optical system 102, an imaging element 103, a signal processing circuit 104, a monitor 105, and a memory 106, and can capture still images and moving images.

[0080] The optical system 102 includes one or more lenses, guides image light (incident light) from a subject to the imaging element 103 , and forms an image on a light receiving surface (sensor unit) of the imaging element 103 .

[0081] The medical imaging device 12 is used as the imaging element 103. Electrons are accumulated for a certain time in the imaging element 103 according to an image formed on the light receiving surface via the optical system 102. Then, a signal corresponding to the electrons accumulated in the imaging element 103 is supplied to the signal processing circuit 104.

[0082] The signal processing circuit 104 performs various types of signal processing on the pixel signal output from the imaging element 103. The image (image data) obtained by the signal processing by the signal processing circuit 104 is supplied to the monitor 105 and displayed on the monitor 105, or supplied to the memory 106 and stored (recorded) in the memory 106.

[0083] The imaging device 101 configured as described above can capture, for example, medical images, wherein the surgical field can be better observed by applying the medical imaging device 12 .

[0084] <Use Examples of Image Sensors>

[0085] Figure 7 : is a diagram showing a usage example of the image sensor (imaging element).

[0086] Image sensors may be used in a variety of situations for sensing light, such as visible light, infrared light, ultraviolet light, and X-rays as described below.

[0087] Devices that capture images for viewing, such as digital cameras and mobile devices with camera capabilities

[0088] Devices used for transportation, such as: in-vehicle sensors that capture images of the environment in front of, behind, and around the car, as well as the interior of the car, for use in safe driving, such as automatic stopping and identifying the driver's condition; surveillance cameras that monitor moving vehicles or roads; or distance measurement sensors that measure the distance between vehicles.

[0089] Devices for home appliances (such as TVs, refrigerators, and air conditioners) that capture images of user gestures and perform operations based on the gestures

[0090] Devices used for medical and therapeutic purposes, such as endoscopes and devices that perform angiography by receiving infrared light

[0091] Devices for security, such as surveillance cameras for crime prevention and cameras for personal authentication

[0092] Devices for beauty use, such as skin measurement devices that capture skin images and microscopes that capture scalp images

[0093] Devices used for sports, such as action cameras and wearable cameras for sports use

[0094] Devices used in agriculture, such as cameras used to monitor field and crop conditions

[0095] <Application Examples>

[0096] The technology according to the present disclosure can be applied to a medical imaging system. A medical imaging system is a medical system that uses imaging technology and is, for example, an endoscope system or a microscope system.

[0097] [Endoscope System]

[0098] Reference Figure 8 and Figure 9 An example of an endoscope system will be described. Figure 8 is a diagram showing a schematic configuration example of an endoscope system 5000 to which the technology according to the present disclosure can be applied. Figure 9 50 is a diagram showing a configuration example of an endoscope 5001 and a camera control unit (CCU) 5039. Figure 8 FIG. 5 shows a situation in which an operator (eg, a doctor) 5067, who is a surgical participant, performs surgery on a patient 5071 on a bed 5069 using an endoscope system 5000. Figure 8 As shown, the endoscope system 5000 includes an endoscope 5001 as a medical imaging device, a CCU 5039 , a light source device 5043 , a recording device 5053 , an output device 5055 , and a support device 5027 for supporting the endoscope 5001 .

[0099] During endoscopic surgery, an insertion aid tool called a trocar 5025 is inserted into a patient 5071. Then, a scope 5003 connected to the endoscope 5001 and surgical tools 5021 are inserted into the body of the patient 5071 through the trocar 5025. The surgical tools 5021 include: an energy device such as an electric scalpel; and forceps.

[0100] A surgical image, which is a medical image captured by an endoscope 5001 inside a patient 5071, is displayed on a display device 5041. An operator 5067 performs surgery on a surgical target using a surgical tool 5021 while viewing the surgical image displayed on the display device 5041. The medical image is not limited to a surgical image and may be a diagnostic image captured during diagnosis.

[0101] [Endoscope]

[0102] The endoscope 5001 is a camera for capturing the inside of the body of a patient 5071, and is, for example, a camera head including: a collecting optical system 50051 for collecting incident light, a zoom optical system 50052 capable of optically zooming by changing the focal length of the camera, a focusing optical system 50053 capable of focusing by changing the focal length of the camera, and a light receiving sensor 50054, as shown in FIG. Figure 9As shown. Endoscope 5001 focuses light from a connected scope 5003 onto a light-receiving sensor 50054 to generate pixel signals, and outputs these pixel signals to CCU 5039 via a transmission system. Scope 5003 is an insertable unit with an objective lens at its tip, which directs light from a connected light source device 5043 into the body of a patient 5071. Scope 5003 may be, for example, a rigid endoscope for a rigid scope or a flexible endoscope for a flexible scope. The pixel signals only need to be based on signals output from the pixels, and may be, for example, raw signals or image signals. The transmission system connecting endoscope 5001 to CCU 5039 may include a memory, and this memory may store parameters related to endoscope 5001 and CCU 5039. The memory may be provided at the connection portion of the transmission system or on the cable. For example, the transmission system's memory may store parameters of the endoscope 5001 before shipment or parameters that change when current is applied, and the operation of the endoscope may be altered based on the parameters read from the memory. The combination of the camera and the transmission system can be called an endoscope. The light receiving sensor 50054 is a sensor for converting the received light into a pixel signal, and is, for example, a complementary metal oxide semiconductor (CMOS) imaging sensor. The light receiving sensor 50054 is preferably an imaging sensor capable of color imaging with a Bayer array. For example, the light receiving sensor 50054 is also preferably an imaging sensor having a plurality of pixels corresponding to a resolution of 4K (3840 horizontal pixels × 2160 vertical pixels), 8K (7680 horizontal pixels × 4320 vertical pixels), or square 4K (3840 or more horizontal pixels × 3840 or more vertical pixels). The light receiving sensor 50054 can be one sensor chip or multiple sensor chips. For example, a prism can be provided to divide the incident light into predetermined wavelength bands, and these wavelength bands can be imaged by different light receiving sensors. Multiple light receiving sensors can be provided for stereoscopic viewing. The light receiving sensor 50054 can be a sensor having a chip structure including an arithmetic processing circuit for image processing, or can be a sensor for time of flight (ToF). The transmission system is, for example, an optical cable system or a wireless transmission system. Wireless transmission only needs to be able to send the pixel signal generated by the endoscope 5001, and for example, the endoscope 5001 can be wirelessly connected to the CCU 5039, or the endoscope 5001 can be connected to the CCU 5039 via a base station in the operating room. At this time, the endoscope 5001 can not only send pixel signals, but also simultaneously send information related to the pixel signals (for example, processing priority and / or synchronization signals of the pixel signals). In the endoscope, the observer can be integrated with the camera, and the light receiving sensor can be set at the far end of the observer.

[0103] [CCU (Camera Control Unit)]

[0104] The CCU 5039 is a control device for controlling the endoscope 5001 and the light source device 5043 connected to the CCU 5039 in an integrated manner, and is, for example, a device including Figure 9 The CCU 5039 is an image processing device shown as a field programmable gate array (FPGA) 50391, a central processing unit (CPU) 50392, a random access memory (RAM) 50393, a read-only memory (ROM) 50394, a graphics processing unit (GPU) 50395, and an interface (I / F) 50396. The CCU 5039 can control the display device 5041, the recording device 5053, and the output device 5055 connected to the CCU 5039 in an integrated manner. The CCU 5039 controls the illumination timing, illumination intensity, type of illumination light source, etc. of the light source device 5043. The CCU 5039 also performs image processing such as development processing (e.g., demosaicing processing) and correction processing on the pixel signals output from the endoscope 5001, and outputs the processed image signals (e.g., images) to an external device such as the display device 5041. The CCU 5039 also sends control signals to the endoscope 5001 to control the driving of the endoscope 5001. The control signal is information about imaging conditions such as the magnification or focal length of the camera. The CCU 5039 may have a down-conversion function for the image and may be configured to simultaneously output a higher resolution (e.g., 4K) image to the display device 5041 and a lower resolution (e.g., high definition (HD)) image to the recording device 5053.

[0105] In addition, the CCU 5039 can be connected to an external device via an IP converter for converting a signal into a predetermined communication protocol (such as the Internet Protocol (IP)). The connection between the IP converter and the external device can be established using a wired network, or a part or all of the network can be established using a wireless network. For example, the IP converter on the CCU 5039 side can have a wireless communication function and can send the received image to the IP switch or the output-side IP converter via a wireless communication network such as the fifth-generation mobile communication system (5G) or the sixth-generation mobile communication system (6G).

[0106] [Light source device]

[0107] The light source device 5043 is a device capable of emitting light having a predetermined wavelength band and includes, for example, multiple light sources and a light source optical system for guiding the light from the light sources. Examples of the light source are xenon lamps, light-emitting diode (LED) light sources, or laser diode (LD) light sources. The light source device 5043 includes, for example, LED light sources corresponding to the three primary colors of red (R), green (G), and blue (B), and controls the output intensity and timing of each light source to emit white light. In addition to a light source for emitting ordinary light for ordinary light observation, the light source device 5043 may also include a light source capable of emitting special light for special light observation. Special light is light of a predetermined wavelength band different from ordinary light used for ordinary light observation, such as near-infrared light (light with a wavelength of 760 nm or longer), infrared light, blue light, or ultraviolet light. Ordinary light is, for example, white light or green light. In narrowband light observation, a form of special light observation, the alternating emission of blue and green light allows for high-contrast imaging of specific tissues, such as blood vessels on mucosal surfaces, by exploiting the wavelength dependence of light absorption in living tissue. In fluorescence observation as a special light observation, since excitation light is emitted to excite the reagent injected into the biological tissue, and fluorescence emitted from the biological tissue or the reagent is received to obtain a fluorescence image, fluorescence observation can facilitate the operator to observe biological tissue that is difficult for the operator to observe using ordinary light. For example, in fluorescence observation using infrared light, near-infrared light is emitted as excitation light to excite the reagent injected into the biological tissue, such as indocyanine green (ICG), so that fluorescence observation can facilitate the observation of the deep structure of the biological tissue. In addition, in fluorescence observation, a reagent (such as PDD or 5-aminolevulinic acid (5-ALA)) that emits fluorescence in the red wavelength band by being excited by special light in the blue wavelength band can be used. The type of irradiation light of the light source device 5043 is set by the control of the CCU 5039. The CCU 5039 can also have a mode for controlling the light source device 5043 and the endoscope 5001 to alternately perform ordinary light observation and special light observation. In this case, it is preferable to superimpose information based on the pixel signal obtained by observation with special light and the pixel signal obtained by observation with normal light.

[0108] [Recording device]

[0109] The recording device 5053 is a device for recording pixels acquired from the CCU 5039 and is, for example, a recorder. The recording device 5053 records images acquired from the CCU 5039 on a hard disk drive (HDD), an ultra-high-density disk (SDD), and / or an optical disk. The recording device 5053 can be connected to a network accessible from equipment outside the operating room within the hospital. The recording device 5053 can have a down-conversion function or an up-conversion function.

[0110] [Display device]

[0111] The display device 5041 is a device capable of displaying images, and is, for example, a display monitor. Under the control of the CCU 5039, the display device 5041 displays an image based on pixel signals processed by the CCU 5039. The display device 5041 may include a camera and a microphone as an input device, which allows input of commands through line of sight recognition, voice recognition, and gestures.

[0112] [Output device]

[0113] The output device 5055 is a device for outputting information acquired from the CCU 5039 and is, for example, a printer. For example, the output device 5055 prints a print image on paper based on the pixel signal acquired from the CCU 5039.

[0114] [Support device]

[0115] The support device 5027 is an articulated arm comprising a base 5029, an arm 5031 extending from the base 5029, and a holding portion 5032 mounted at the distal end of the arm 5031. The base 5029 includes an arm control device 5045. The arm control device 5045 includes a processor, such as a CPU, and operates according to a predetermined computer program to control the driving of the arm 5031. The support device 5027 uses the arm control device 5045 to control parameters such as the length of the link 5035 constituting the arm 5031, the rotation angle of the joint 5033, and the torque, thereby controlling, for example, the position and posture of the endoscope 5001 held by the holding portion 5032. This allows the position or posture of the endoscope 5001 to be changed to a desired position or posture, enabling insertion of the scope 5003 into the patient 5071 and changing the observed area within the body. The support device 5027 serves as an endoscope support arm for supporting the endoscope 5001 during operation. Therefore, the support device 5027 can play the role of a scopist as an assistant to hold the endoscope 5001. The support device 5027 can be a device for holding the microscope device 5301 to be described later, and can be called a medical support arm. The support device 5027 can be controlled by the arm control device 5045 using an autonomous control method, or can be controlled using a control method in which the arm control device 5045 performs control based on the user's input. The control method can be, for example, a master-slave method, in which the support device 5027 acting as a slave device is controlled based on the movement of the master device in the user's hand. The support device 5027 can be remotely controlled from outside the operating room.

[0116] An example of the endoscope system 5000 to which the technology according to the present disclosure can be applied has been described above. For example, the technology according to the present disclosure can be applied to a microscope system.

[0117] [Microscope system]

[0118] Figure 10 1 is a diagram showing an example of a schematic configuration of a microsurgery system to which the technology according to the present disclosure can be applied. In the following description, the same reference numerals are attached to the same structural elements as those of the endoscope system 5000, and repeated description is omitted.

[0119] Figure 10 The diagram schematically shows an operator 5067 using a microsurgery system 5300 to perform surgery on a patient 5071 on a bed 5069. Figure 10 The cart 5037 is not shown among the components of the microscope surgery system 5300, and the microscope device 5301 is shown in a simplified manner instead of the endoscope 5001. The microscope device 5301 may refer to the microscope 5303 provided at the distal end of the link 5035, or may refer to the overall configuration including the microscope 5303 and the support device 5027.

[0120] like Figure 10 As shown, during surgery, a microscope surgery system 5300 is used to display an image of a surgical site captured by a microscope device 5301 in an enlarged manner on a display device 5041 installed in an operating room. The display device 5041 is installed in a position facing an operator 5067, and the operator 5067 performs various treatments on the surgical site, such as resecting an affected part, while observing the state of the surgical site using the image displayed on the display device 5041.

[0121] The corresponding examples of the endoscope system 5000 and the microsurgery system 5300 to which the technology of the present disclosure can be applied have been described above. The systems to which the technology of the present disclosure can be applied are not limited to such examples. For example, instead of the endoscope 5001 or the microscope 5303, the support device 5027 can support another observation device or another surgical tool at its distal end. As other observation devices, for example, there can be cited forceps, forceps, pneumoperitoneum tubes for pneumoperitoneum, energy treatment instruments for incising tissues and sealing blood vessels by cauterization, etc. By using a support device to support the observation device or surgical tool, its position can be fixed more stably and the burden on the medical staff can be reduced compared to the case where the medical staff manually supports the observation device or surgical tool. The technology of the present disclosure can be applied to a support device that supports such components other than a microscope.

[0122] The technology according to the present disclosure can be applied to the endoscope 5001 or the microscope device 5301 in the above configuration. As a result, medical images that enable better observation of the surgical field can be captured, and operations can be performed more safely and reliably.

[0123] <Example of configuration combination>

[0124] It should be noted that the present technology can also have the following configurations. (1)

[0126] A medical imaging system comprising:

[0127] a first imaging element that receives light of a first wavelength band among incident light incident from the mounting surface and outputs a first image signal;

[0128] a second imaging element, receiving light of a second wavelength band different from the first wavelength band in the incident light and outputting a second image signal;

[0129] a third imaging element, receiving at least the light in the second wavelength band in the incident light and outputting a third image signal; and

[0130] a signal processing unit that performs depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals of the first image signal, the second image signal, and the third image signal, wherein

[0131] An optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element and shorter than an optical path length from the mounting surface to the third imaging element. (2)

[0133] The medical imaging system according to (1), wherein

[0134] The second imaging element and the third imaging element receive light of visible wavelengths, and

[0135] The signal processing unit generates the extended depth of field image by selecting a pixel value having a higher contrast among pixel values ​​of pixels at the same pixel position. (3)

[0137] The medical imaging system according to (1) or (2), further comprising:

[0138] A light splitting system branches the incident light toward each of the first imaging element, the second imaging element, and the third imaging element, wherein

[0139] The optical splitting system is configured so that an amount of light incident on the third imaging element is greater than an amount of light incident on the second imaging element. (4)

[0141] The medical imaging system according to any one of (1) to (3), wherein

[0142] The first imaging element receives near infrared (NIR) light. (5)

[0144] The medical imaging system according to any one of (1) to (4), wherein

[0145] The first imaging element receives violet (V) light. (6)

[0147] The medical imaging system according to any one of (1) to (5), wherein

[0148] Each of the second imaging element and the third imaging element includes a filter, the first imaging element does not include a filter, and

[0149] The signal processing unit performs the depth of field extension processing using the second image signal and the third image signal. (7)

[0151] The medical imaging system according to any one of (1) to (5), wherein

[0152] Each of the first imaging element, the second imaging element, and the third imaging element includes a filter, and

[0153] The signal processing unit performs the depth of field extension processing using the first image signal and the third image signal. (8)

[0155] The medical imaging system according to any one of (1) to (5), wherein

[0156] The first imaging element and the third imaging element do not include a filter, the second imaging element includes a filter, and

[0157] The signal processing unit performs the depth of field extension processing using the first image signal and the third image signal. (9)

[0159] The medical imaging system according to any one of (1) to (8), wherein

[0160] The signal processing unit changes which combination of the first image signal, the second image signal, and the third image signal is used to perform the depth of field extension processing according to a mode. (10)

[0162] A medical imaging device comprising:

[0163] a first imaging element that receives light of a first wavelength band among incident light incident from the mounting surface and outputs a first image signal;

[0164] a second imaging element, receiving light of a second wavelength band different from the first wavelength band in the incident light and outputting a second image signal;

[0165] a third imaging element, receiving at least the light in the second wavelength band in the incident light and outputting a third image signal; and

[0166] a signal processing unit that performs depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals of the first image signal, the second image signal, and the third image signal, wherein

[0167] An optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element and shorter than an optical path length from the mounting surface to the third imaging element. (11)

[0169] A method for operating a medical imaging system, the method comprising:

[0170] receiving light of a first wavelength band among incident light incident from the installation surface through a first imaging element and outputting a first image signal;

[0171] receiving light of a second wavelength band different from the first wavelength band in the incident light through a second imaging element and outputting a second image signal;

[0172] receiving at least the light of the second wavelength band in the incident light through a third imaging element and outputting a third image signal; and

[0173] performing depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals among the first image signal, the second image signal, and the third image signal, wherein

[0174] The medical imaging system is configured such that an optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element and shorter than an optical path length from the mounting surface to the third imaging element.

[0175] It should be noted that the embodiments of the present disclosure are not limited to the embodiments described, and various modifications can be made without departing from the gist of the present disclosure. In addition, the effects described in this specification are merely illustrative and not restrictive, and there may be other effects.

[0176] Reference Symbol List

[0177] 11 Medical Imaging Systems

[0178] 12 Medical imaging devices

[0179] 13 Signal processing device

[0180] 21 Mounting parts

[0181] 22 Spectroscopic System

[0182] 23 Imaging Module

[0183] 31 First Prism

[0184] 32 Second Prism

[0185] 33 Third Prism

[0186] 34 wavelength separation mirror

[0187] 35 Half Mirror

[0188] 41 housing

[0189] 42 cover glass

[0190] 43 Imaging Elements

[0191] 44 filters.

Claims

1. A medical imaging system comprising: a first imaging element that receives light of a first wavelength band among incident light incident from the mounting surface and outputs a first image signal; a second imaging element, receiving light of a second wavelength band different from the first wavelength band in the incident light and outputting a second image signal; a third imaging element, receiving at least the light in the second wavelength band in the incident light and outputting a third image signal; as well as a signal processing unit that performs depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals among the first image signal, the second image signal, and the third image signal, wherein An optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element, and shorter than an optical path length from the mounting surface to the third imaging element.

2. The medical imaging system according to claim 1, wherein The second imaging element and the third imaging element receive light of visible wavelengths, and The signal processing unit generates the extended depth of field image by selecting a pixel value having a higher contrast among pixel values ​​of pixels at the same pixel position.

3. The medical imaging system of claim 1 , further comprising: A light splitting system branches the incident light toward each of the first imaging element, the second imaging element, and the third imaging element, wherein: The optical splitting system is configured so that an amount of light incident on the third imaging element is greater than an amount of light incident on the second imaging element.

4. The medical imaging system of claim 1 , wherein: The first imaging element receives near infrared (NIR) light.

5. The medical imaging system of claim 1 , wherein: The first imaging element receives violet (V) light.

6. The medical imaging system of claim 1 , wherein: Each of the second imaging element and the third imaging element includes a filter, the first imaging element does not include a filter, and The signal processing unit performs the depth of field extension processing using the second image signal and the third image signal.

7. The medical imaging system of claim 1 , wherein: Each of the first imaging element, the second imaging element, and the third imaging element includes a filter, and The signal processing unit performs the depth of field extension processing using the first image signal and the third image signal.

8. The medical imaging system of claim 1 , wherein: The first imaging element and the third imaging element do not include a filter, the second imaging element includes a filter, and The signal processing unit performs the depth of field extension processing using the first image signal and the third image signal.

9. The medical imaging system of claim 1 , wherein: The signal processing unit changes which combination of the first image signal, the second image signal, and the third image signal is used to perform the depth of field extension processing according to a mode.

10. A medical imaging device comprising: a first imaging element that receives light of a first wavelength band among incident light incident from the mounting surface and outputs a first image signal; a second imaging element, receiving light of a second wavelength band different from the first wavelength band in the incident light and outputting a second image signal; a third imaging element, receiving at least the light in the second wavelength band in the incident light and outputting a third image signal; as well as a signal processing unit that performs depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals among the first image signal, the second image signal, and the third image signal, wherein An optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element, and shorter than an optical path length from the mounting surface to the third imaging element.

11. A method for operating a medical imaging system, comprising: receiving light of a first wavelength band among incident light incident from the installation surface through a first imaging element and outputting a first image signal; receiving light of a second wavelength band different from the first wavelength band in the incident light through a second imaging element and outputting a second image signal; receiving at least the light of the second wavelength band in the incident light through a third imaging element and outputting a third image signal; as well as performing depth-of-field extension processing to generate an extended depth-of-field image obtained by extending the depth of field using at least two image signals among the first image signal, the second image signal, and the third image signal, wherein The medical imaging system is configured such that an optical path length from the mounting surface to the first imaging element is longer than an optical path length from the mounting surface to the second imaging element, and shorter than an optical path length from the mounting surface to the third imaging element.

Images