Medical imaging system and medical imaging device

Abstract

The present disclosure relates to a medical imaging system, a medical imaging device, and an operation method capable of capturing a medical image with which an operative field can be more satisfactorily observed. The medical imaging system comprises: a first imaging element for receiving light through a color filter and outputting a first image signal; a second imaging element for receiving light not through the color filter and outputting a second image signal; a branch optical system for branching incident light entering from a mount surface into light entering the first imaging element and light entering the second imaging element; and a signal processing unit for performing depth-of-field enlarging processing for generating a depth-of-field enlarged image in which a depth of field is enlarged by using the first image signal and the second image signal. Then, the optical path length from the mount surface to the first imaging element is configured to be shorter than the optical path length from the mount surface to the second imaging element. This technology is applicable to, for example, a medical imaging system capable of capturing an EDOF image.

Description

[Technical Field]

[0001] The present disclosure relates to a medical imaging system and Medical Imaging Equipment Place In particular, a medical imaging system that can capture medical images that allow better observation of the surgical field. and Medical Imaging Equipment Place Regarding. [Background technology]

[0002] Generally, medical observation devices such as endoscopes and microscopes capture images with a shallow depth of field. However, because the surgical field used in surgery using endoscopes and microscopes often has depth, there is a demand for capturing medical images with a deep depth of field.

[0003] Therefore, as disclosed in Patent Document 1, in order to increase the depth of field, endoscopes and microscopes have been proposed that utilize an EDOF (Extended Depth of Field) optical system with a phase mask to increase the depth of field. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-158764 Summary of the Invention [Problem to be solved by the invention]

[0005] Resolution and depth of field are determined by the F-number of an optical system, and there is a trade-off between them. In other words, increasing the resolution by setting a brighter F-number results in an image with a shallow depth of field, making it impossible to obtain a practically sufficient image quality for observing the surgical field. Therefore, there is a need for medical images that combine high resolution and a deep depth of field to enable better observation of the surgical field.

[0006] The present disclosure has been made in view of the above circumstances, and makes it possible to capture medical images that allow better observation of the surgical field. [Means for solving the problem]

[0007] A medical imaging system and a medical imaging device according to one aspect of the present disclosure include a first imaging element that receives light through a color filter and outputs a first image signal, a second imaging element that receives light without passing through a color filter and outputs a second image signal, a branching optical system that branches incident light entering from a mount surface into light that enters the first imaging element and light that enters the second imaging element, and a signal processing unit that performs depth-of-field extension processing to generate an extended depth-of-field image with an extended depth of field using the first image signal and the second image signal, wherein the optical path length from the mount surface to the first imaging element is shorter than the optical path length from the mount surface to the second imaging element.

[0009] In one aspect of the present disclosure, a first image signal is output by a first image sensor that receives branched light from incident light entering through a mount surface, a second image sensor that receives branched light separately from the first image signal and outputs a second image signal, and a depth-of-field extended process is performed to generate an extended depth-of-field image with an extended depth of field using the first and second image signals, and the optical path length from the mount surface to the first image sensor is configured to be shorter than the optical path length from the mount surface to the second image sensor. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example configuration of a first embodiment of a medical imaging system to which the present technology is applied. [Figure 2] FIG. 10 is a block diagram showing a configuration example of a second embodiment of a medical imaging system. [Figure 3] FIG. 10 is a block diagram showing an example of the configuration of a third embodiment of a medical imaging system. [Figure 4]10 is a flowchart illustrating a method for capturing a medical image. [Figure 5] FIG. 1 is a block diagram illustrating an example of the configuration of an imaging device. [Figure 6] FIG. 1 is a diagram illustrating an example of use of an image sensor. [Figure 7] FIG. 1 is a diagram illustrating an example of a schematic configuration of an endoscope system. [Figure 8] 8 is a block diagram showing an example of the functional configuration of the camera head and the CCU shown in FIG. 7. FIG. [Figure 9] FIG. 1 is a diagram illustrating an example of a schematic configuration of a microsurgery system. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <First Configuration Example of Medical Imaging System> FIG. 1 is a block diagram showing an example of the configuration of a first embodiment of a medical imaging system to which the present technology is applied.

[0013] 1 is intended for use in, for example, an endoscope or a surgical microscope, and is configured to include a medical imaging device 12 and a signal processing device 13. For example, in the medical imaging system 11, light from a surgical field (not shown) enters the medical imaging device 12 along the optical axis indicated by the dashed dotted line, and an image of the surgical field captured by the medical imaging device 12 is supplied to the signal processing device 13. An EDoF 4K image obtained as a result of depth-of-field extension processing performed by the signal processing device 13 is then output from the medical imaging system 11.

[0014] The medical imaging device 12 is configured to include a mount 21, a branching optical system 22, and imaging modules 23-1 and 23-2.

[0015] The mount 21 is a connection part for connecting, to the medical imaging device 12, an imaging lens having a focusing function for focusing on an affected area in the surgical field, for example.

[0016] The branching optical system 22 branches the light incident on the medical imaging device 12 via the mount 21 into light incident on the imaging module 23-1 and light incident on the imaging module 23-2.

[0017] The branching optical system 22 is composed of a first prism 31, a second prism 32, and a half mirror 33. For example, the branching optical system 22 is configured such that the half mirror 33 is disposed between the first prism 31 and the second prism 32, and light reflected by the half mirror 33 enters the imaging module 23-2 via the first prism 31, and light transmitted through the half mirror 33 enters the imaging module 23-1 via the second prism 32.

[0018] The half mirror 33 can be set so that the reflectance and transmittance are the same (50:50), or, for example, the reflectance is set higher than the transmittance, or the reflectance is set lower than the transmittance. A half mirror 33 with a reflectance set higher than the transmittance can allow more light to be incident on the imaging module 23-2 than on the imaging module 23-1. A half mirror 33 with a reflectance set lower than the transmittance can allow more light to be incident on the imaging module 23-1 than on the imaging module 23-2. Note that, since it is basically difficult to change the reflectance and transmittance of the half mirror 35, the settings of the reflectance and transmittance are parameters that are adjusted during design.

[0019] The imaging module 23-1 is configured by accommodating an imaging element 43-1 and a filter 44-1 in a space sealed by a housing 41-1 and a cover glass 42-1. The imaging module 23-2 is configured in the same manner.

[0020] The filter 44-1 of the imaging module 23-1 is a transparent filter that transmits light in all wavelength ranges (or may be configured without a filter), and the imaging element 43-1 captures a luminance image (monochrome image) through the filter 44-1. The filter 44-2 of the imaging module 23-2 is an RGB filter with a Bayer array as shown in the figure, and the imaging element 43-2 captures a color image through the RGB filter.

[0021] Furthermore, the medical imaging device 12 is configured so that the optical distance from the mount 21 to the imaging module 23-2 is shorter than the optical distance from the mount 21 to the imaging module 23-1. As a result, when the focal length is adjusted so that the brightness image captured by the imaging module 23-1 is focused on a point near the desired affected area, the imaging module 23-2 will be out of focus. That is, the color image captured by the imaging module 23-2 will be focused on a point closer than the desired affected area.

[0022] Hereinafter, the luminance image captured by the imaging module 23-1 will be referred to as a middle luminance image, and the color image captured by the imaging module 23-2 will be referred to as a near color image. Therefore, in the medical imaging system 11, the middle luminance image output from the imaging module 23-1 and the near color image output from the imaging module 23-2 are supplied from the medical imaging device 12 to the signal processing device 13.

[0023] The signal processing device 13 performs depth of field extension processing to generate an EDOF image with an extended depth of field using the middle luminance image and the near color image.

[0024] First, the signal processing device 13 performs continuity detection from the middle luminance image and uses the results to demosaic (develop) the near color image. The signal processing device 13 calculates the difference between adjacent pixels in the horizontal, vertical, and diagonal directions, and performs continuity detection by determining that the direction in which the pixel difference is smallest is the direction in which continuity (edge direction) exists. For example, in the case of a horizontal edge, the pixel difference in the horizontal direction is smallest. The signal processing device 13 uses a middle luminance image with all pixels for diagonal lines, which is a point that determines the performance of the demosaic, and can achieve higher accuracy than a green checkered pattern in a Bayer array.

[0025] Next, the signal processing device 13 converts the near color image from RGB to YCbCr, and converts the middle luminance image to Y (≈MONO).The signal processing device 13 then selects the near color image or the middle luminance image, whichever has higher contrast, as the Y of the image to be output.For example, the signal processing device 13 detects edges in each of the near color image and the middle luminance image, and determines that an edge with a large edge has a high contrast.

[0026] Next, the signal processing device 13 outputs CbCr. For example, in the case of a near color image, each CbCr is set as the CbCr to be output. On the other hand, in the case of a middle luminance image, the contrast difference between the near color image and the middle luminance image is calculated, and a process (combining process) to reduce and eliminate the calculated contrast difference is performed on the CbCr of the second signal, and the CbCr is set as the CbCr to be output. Then, the signal processing device 13 performs unsharp processing to subtract Y of the near color image from Y of the middle luminance image (Y(Middle)-Y(Near)), and add the result to CbCr.

[0027] By performing such depth of field extension processing, the signal processing device 13 can generate an EDOF image.

[0028] The method and parameters for detecting continuity from the middle brightness image by the signal processing device 13 may be changed depending on the mode selected by the user or the surgical situation obtained by image recognition. For example, in the ENT mode, the inside of a thin tube is often observed, and the focus is likely to differ between the periphery and the center of the image. Therefore, the signal processing device 13 may perform continuity detection by looking only at the center of the image. Furthermore, the signal processing device 13 may perform image recognition to detect the presence of a surgical tool such as forceps in the center of the image, and perform continuity detection based on the area around the tip of the surgical tool.

[0029] In the medical imaging system 11, the reflectance and transmittance of the half mirror 33 are set so that more light is incident on the imaging module 23-1 than on the imaging module 23-2, thereby making the middle brightness image brighter. In the medical imaging system 11, the reflectance and transmittance of the half mirror 33 are set so that more light is incident on the imaging module 23-2 than on the imaging module 23-1, thereby making the near color image brighter. In this way, in the medical imaging system 11, noise can be reduced by setting the reflectance and transmittance of the half mirror 33 according to, for example, the difference in sensitivity between the imaging modules 23-1 and 23-2.

[0030] <Second Configuration Example of Medical Imaging System> Fig. 2 is a block diagram showing a configuration example of a second embodiment of a medical imaging system to which the present technology is applied. In the medical imaging system 11A shown in Fig. 2, components common to the medical imaging system 11 in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0031] As shown in Fig. 2, a medical imaging system 11A includes a medical imaging device 12A and a signal processing device 13. In the medical imaging device 12A, a branching optical system 22A has a configuration different from that of the medical imaging device 12 in Fig. 1 in that it includes a first prism 31, a second prism 32A, a half mirror 33, a third prism 34, and a half mirror 35. That is, the branching optical system 22A branches light incident on the medical imaging device 12 via the mount 21 into three directions.

[0032] 1 in that it includes an imaging module 23-3 in addition to the imaging modules 23-1 and 23-2. The imaging module 23-3 is capable of capturing color images, similar to the imaging module 23-2.

[0033] The branching optical system 22A is configured such that a half mirror 33 is disposed between the first prism 31 and the second prism 32A, and light reflected by the half mirror 33 enters the imaging module 23-2 via the first prism 31. The branching optical system 22A is configured such that a half mirror 35 is disposed between the second prism 32A and the third prism 34, and light reflected by the half mirror 35 enters the imaging module 23-3 via the second prism 32, and light transmitted through the half mirror 35 enters the imaging module 23-1 via the third prism 34.

[0034] Similar to the medical imaging apparatus 12 of FIG. 1, the medical imaging apparatus 12A can set the reflectance and transmittance of the half mirrors 33 and 35 according to the difference in sensitivity between the imaging modules 23-1 to 23-3.

[0035] Furthermore, the medical imaging device 12A is configured so that the optical distance from the mount 21 to the imaging module 23-3 is longer than the optical distance from the mount 21 to the imaging module 23-1. As a result, when the focal length is adjusted so that the brightness image captured by the imaging module 23-1 is focused on a point near the desired affected area, the imaging module 23-3 will be out of focus. That is, the color image captured by the imaging module 23-3 will be focused on a point farther away than the desired affected area.

[0036] Hereinafter, the color image captured by the imaging module 23-3 will be referred to as a far color image where appropriate. Therefore, in the medical imaging system 11A, the middle brightness image output from the imaging module 23-1, the near color image output from the imaging module 23-2, and the far color image output from the imaging module 23-3 are supplied from the medical imaging device 12A to the signal processing device 13.

[0037] The signal processing device 13 performs depth of field extension processing to generate an EDOF image with an extended depth of field, similar to the signal processing device 13 of FIG. 1, using the middle luminance image, the near color image, and the far color image.

[0038] The medical imaging device 12A may be configured to further include an IR imaging module (image sensor for a long wavelength light source) and have four imaging modules 23. In this configuration, it is preferable to arrange the IR imaging module between the imaging module 23-1 for the middle luminance image and the imaging module 23-3 for the far color image. The medical imaging device 12A may be configured to further include an V imaging module (image sensor for a short wavelength light source) and have four imaging modules 23. In this configuration, it is preferable to arrange the V imaging module between the imaging module 23-1 for the middle luminance image and the imaging module 23-2 for the near color image.

[0039] Furthermore, in the medical imaging device 12A, it is preferable that the relationship (BC)>(CA) be satisfied among the distance A from the mount surface to the imaging module 23-2 for near color images, the distance B from the mount surface to the imaging module 23-1 for middle luminance images, and the distance C from the mount surface to the imaging module 23-3 for far color images. That is, this indicates that the imaging module 23-1 for middle luminance images is disposed on the far side of the center between the imaging module 23-2 for near color images and the imaging module 23-3 for far color images. This is because the imaging module for far color images is often important.

[0040] <Third Configuration Example of Medical Imaging System> Fig. 3 is a block diagram showing a configuration example of a third embodiment of a medical imaging system to which the present technology is applied. In the medical imaging system 11B shown in Fig. 3, components common to the medical imaging system 11 in Fig. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0041] As shown in Fig. 3, a medical imaging system 11B includes a medical imaging device 12B and a signal processing device 13. In the medical imaging device 12B, the branching optical system 22 is configured differently from the medical imaging device 12 of Fig. 1 in that it includes a first prism 31, a second prism 32B, a half mirror 33, a third prism 34, and a half mirror 35. That is, like the branching optical system 22A of Fig. 2, the branching optical system 22B branches light incident on the medical imaging device 12 via the mount 21 into three directions.

[0042] 1 in that it includes an imaging module 23-3 in addition to imaging modules 23B-1 and 23-2. Like imaging module 23-2, imaging module 23-3 is capable of capturing color images. Furthermore, imaging module 23B-1's filter 44B-1 is a Bayer array RGB filter as shown in the figure, and imaging element 43-1 is configured to capture color images via filter 44B-1, making it different from imaging module 23-1 in the configuration shown in FIG.

[0043] Furthermore, medical imaging device 12B is configured so that the optical distance from mount 21 to imaging module 23-3 is longer than the optical distance from mount 21 to imaging module 23B-1. As a result, when the focal length is adjusted so that the color image captured by imaging module 23B-1 is focused on a point near the desired affected area, imaging module 23-3 will be out of focus. That is, the color image captured by imaging module 23-3 will be focused on a point farther away than the desired affected area.

[0044] Hereinafter, the color image captured by the imaging module 23B-1 will be referred to as a middle color image, and the color image captured by the imaging module 23-3 will be referred to as a far color image. Therefore, in the medical imaging system 11, the middle color image output from the imaging module 23B-1, the near color image output from the imaging module 23-2, and the far color image output from the imaging module 23-3 are supplied from the imaging module 23-2 to the signal processing device 13.

[0045] Furthermore, in the medical imaging device 12B, the imaging module 23-3 for the far color image and the imaging module 23-2 for the near color image are positioned one pixel apart in a horizontal plane direction (up / down or left / right) perpendicular to the optical axis relative to the imaging module 23B-1 for the middle color image.

[0046] The signal processing device 13 performs depth of field extension processing to generate an EDOF image with an extended depth of field using the middle color image, the near color image, and the far color image.

[0047] First, the signal processing device 13 selects the image with the highest contrast and the image with the second highest contrast using green (Gr, Gb) in the Bayer array in the Middle color image, Near color image, and Far color image. Here, the two images selected are either a pair of a Near color image and a Middle color image, or a pair of a Middle color image and a Far color image.

[0048] Next, the signal processing device 13 performs steady-state detection using an image of all pixels consisting of green pixel values, rather than a checkered pattern, by combining green (Gr, Gb) in the Bayer array of each of the two selected images.

[0049] Then, the signal processing device 13 demosaices the image with the highest contrast using the result of continuity detection, and then generates RGB as an image to be output.

[0050] In the medical imaging system 11B, the reflectance and transmittance of the half mirror 33 are preferably set so that a middle color image is always selected, and therefore more light is incident on the imaging module 23B-1 for middle color images.

[0051] A method for capturing medical images using the medical imaging system 11B will be described with reference to the flowchart of FIG.

[0052] In step S11, the imaging module 23B-1 acquires a middle color image and outputs it to the signal processing device 13.

[0053] In step S12, the imaging module 23-2 acquires a Near color image and outputs it to the signal processing device 13.

[0054] In step S13, the imaging module 23-3 acquires a far color image and outputs it to the signal processing device 13.

[0055] In step S14, the signal processing device 13 performs depth of field extension processing to generate an EDOF image with an extended depth of field using the Middle color image, the Near color image, and the Far color image, and outputs the EDOF image.

[0056] As described above, the medical imaging system 11 of each of the above-described embodiments can simultaneously acquire three images with different focuses: a middle brightness image (middle color image), a near color image, and a far color image. From these three images, the medical imaging system 11 can generate an image with a high resolution and depth of field, enabling good observation using the generated image.

[0057] For example, Japanese Patent Application Laid-Open No. 2012-169989 discloses a technique for obtaining a good interpolated image without false colors by using the luminance signal of a monochrome image sensor when demosaicing the signal obtained from a Bayer array RGB image sensor, but does not disclose anything about extending the depth of field.In contrast, the medical imaging system 11 can obtain high-resolution images as well as images with an extended depth of field.

[0058] <Example of electronic device configuration> The medical imaging device 12 as described above can be applied to various electronic devices, such as imaging systems such as digital still cameras and digital video cameras, mobile phones with imaging capabilities, or other devices with imaging capabilities.

[0059] FIG. 5 is a block diagram showing an example of the configuration of an imaging device mounted on an electronic device.

[0060] As shown in FIG. 5, the imaging device 101 includes an optical system 102, an imaging element 103, a signal processing circuit 104, a monitor 105, and a memory 106, and is capable of capturing still images and moving images.

[0061] The optical system 102 is configured to have one or more lenses, and guides image light (incident light) from a subject to the image sensor 103, forming an image on the light receiving surface (sensor section) of the image sensor 103.

[0062] The above-described medical imaging device 12 is applied as the imaging element 103. Electrons are accumulated in the imaging element 103 for a certain period of time in accordance with an image formed on the light receiving surface via the optical system 102. A signal corresponding to the electrons accumulated in the imaging element 103 is then supplied to the signal processing circuit 104.

[0063] The signal processing circuit 104 performs various types of signal processing on the pixel signals output from the image sensor 103. The image (image data) obtained by the signal processing performed by the signal processing circuit 104 is supplied to a monitor 105 to be displayed, or supplied to a memory 106 to be stored (recorded).

[0064] By applying the above-described medical imaging device 12 to the imaging device 101 configured in this manner, it is possible to capture medical images that allow better observation of the surgical field, for example.

[0065] <Examples of using image sensors> FIG. 6 is a diagram showing an example of using the image sensor (imaging element) described above.

[0066] The image sensor described above can be used in various cases for sensing light such as visible light, infrared light, ultraviolet light, and X-rays, for example, as follows.

[0067] ·Digital cameras, mobile devices with camera functions, and other devices that take images for viewing purposes - Devices used for traffic purposes, such as in-vehicle sensors that take pictures of the front, rear, surroundings, and interior of a vehicle for safe driving such as automatic stopping, and for recognizing the driver's condition, surveillance cameras that monitor moving vehicles and roads, and distance measuring sensors that measure distances between vehicles. A device used in home appliances such as TVs, refrigerators, and air conditioners to capture user gestures and operate the appliances according to those gestures. -Medical and healthcare equipment, such as endoscopes and devices that take blood vessel images using infrared light - Security devices such as surveillance cameras for crime prevention and cameras for person authentication Cosmetic devices such as skin measuring devices that take pictures of the skin and microscopes that take pictures of the scalp Sports equipment such as action cameras and wearable cameras for sports purposes Agricultural equipment such as cameras for monitoring the condition of fields and crops

[0068] <Application example> The technology disclosed herein can be applied to a medical imaging system, which is a medical system that uses imaging technology, such as an endoscope system or a microscope system.

[0069] [Endoscope system] An example of an endoscopic system will be described with reference to FIGS. 7 and 8. FIG. 7 is a diagram illustrating an example of the schematic configuration of an endoscopic system 5000 to which the technology according to the present disclosure can be applied. FIG. 8 is a diagram illustrating an example of the configuration of an endoscope 5001 and a CCU (Camera Control Unit) 5039. FIG. 7 illustrates a state in which an operator (e.g., a doctor) 5067, who is a surgical participant, is performing surgery on a patient 5071 on a patient bed 5069 using the endoscopic system 5000. As shown in FIG. 7, the endoscopic system 5000 includes an endoscope 5001, which is a medical imaging device, a CCU 5039, a light source device 5043, a recording device 5053, an output device 5055, and a support device 5027 that supports the endoscope 5001.

[0070] In endoscopic surgery, an insertion aid called a trocar 5025 is inserted into a patient 5071. Then, a scope 5003 and surgical tools 5021 connected to an endoscope 5001 are inserted into the body of the patient 5071 via the trocar 5025. The surgical tools 5021 are, for example, energy devices such as an electric scalpel, forceps, etc.

[0071] A surgical image, which is a medical image showing the inside of the body of a patient 5071 photographed by an endoscope 5001, is displayed on a display device 5041. An operator 5067 performs a procedure on the surgical target using a surgical tool 5021 while viewing the surgical image displayed on the display device 5041. Note that the medical image is not limited to a surgical image, and may be a diagnostic image photographed during a diagnosis.

[0072] [Endoscopy] The endoscope 5001 is a camera that captures images of the inside of the body of a patient 5071. For example, as shown in FIG. 8 , the endoscope 5001 is a camera head including a focusing optical system 50051 that focuses incident light, a zoom optical system 50052 that changes the focal length of the camera to enable optical zoom, a focus optical system 50053 that changes the focal length of the camera to enable focus adjustment, and a light-receiving element 50054. The endoscope 5001 generates pixel signals by focusing light onto the light-receiving element 50054 via a connected scope 5003, and outputs the pixel signals to the CCU 5039 via a transmission system. The scope 5003 has an objective lens at its tip and is an insertion section that guides light from a connected light source device 5043 into the body of the patient 5071. The scope 5003 is, for example, a rigid scope in the case of a rigid endoscope, or a flexible scope in the case of a flexible endoscope. The pixel signals may be signals based on signals output from pixels, such as RAW signals or image signals. Furthermore, a configuration may be adopted in which a memory is installed in the transmission system connecting the endoscope 5001 and the CCU 5039, and parameters related to the endoscope 5001 and the CCU 5039 are stored in the memory. The memory may be disposed, for example, at a connection portion of the transmission system or on a cable. For example, parameters at the time of shipment of the endoscope 5001 and parameters that change when power is applied may be stored in the memory of the transmission system, and the operation of the endoscope may be changed based on parameters read from the memory. Furthermore, the endoscope and the transmission system may be collectively referred to as an endoscope. The light receiving element 50054 is a sensor that converts received light into pixel signals, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type imaging element. It is preferable that the light receiving element 50054 be an imaging element having a Bayer array and capable of color imaging. Furthermore, the light receiving element 50054 is preferably an imaging element having a number of pixels corresponding to a resolution of, for example, 4K (3840 horizontal pixels × 2160 vertical pixels), 8K (7680 horizontal pixels × 4320 vertical pixels), or square 4K (3840 or more horizontal pixels × 3840 or more vertical pixels). The light receiving element 50054 may be a single sensor chip or multiple sensor chips. For example, a prism may be provided to separate incident light into predetermined wavelength bands, and each wavelength band may be imaged by a different light receiving element.Furthermore, multiple light-receiving elements may be provided for stereoscopic vision. The light-receiving element 50054 may be a sensor including an arithmetic processing circuit for image processing within its chip structure, or may be a ToF (Time of Flight) sensor. The transmission system may be, for example, an optical fiber cable or wireless transmission. Wireless transmission may be performed via any means capable of transmitting pixel signals generated by the endoscope 5001. For example, the endoscope 5001 and the CCU 5039 may be connected wirelessly, or the endoscope 5001 and the CCU 5039 may be connected via a base station in the operating room. In this case, the endoscope 5001 may simultaneously transmit not only the pixel signals but also information related to the pixel signals (e.g., the processing priority of the pixel signals, a synchronization signal, etc.). The endoscope may be configured such that the scope and camera head are integrated, or a light-receiving element is provided at the tip of the scope.

[0073] [CCU (Camera Control Unit)] The CCU 5039 is a control device that comprehensively controls the connected endoscope 5001 and light source device 5043, and is, for example, an information processing device having an FPGA 50391, a CPU 50392, a RAM 50393, a ROM 50394, a GPU 50395, and an I / F 50396, as shown in FIG. 8 . The CCU 5039 may also comprehensively control the connected display device 5041, recording device 5053, and output device 5055. For example, the CCU 5039 controls the irradiation timing, irradiation intensity, and type of irradiation light source of the light source device 5043. The CCU 5039 also performs image processing such as development processing (e.g., demosaic processing) and correction processing on pixel signals output from the endoscope 5001, and outputs the processed pixel signals (e.g., images) to an external device such as the display device 5041. The CCU 5039 also transmits control signals to the endoscope 5001 to control the driving of the endoscope 5001. The control signal is, for example, information regarding imaging conditions such as the magnification and focal length of the camera. The CCU 5039 may have an image down-conversion function and may be configured to be able to simultaneously output a high-resolution (e.g., 4K) image to the display device 5041 and a low-resolution (e.g., HD) image to the recording device 5053.

[0074] The CCU 5039 may also be connected to an external device via an IP converter that converts signals into a predetermined communication protocol (e.g., IP (Internet Protocol)). The connection between the IP converter and the external device may be configured as a wired network, or a part or all of the network may be configured as a wireless network. For example, the IP converter on the CCU 5039 side may have a wireless communication function, and may transmit the received video to an IP switcher or an output-side IP converter via a wireless communication network such as a fifth-generation mobile communication system (5G) or a sixth-generation mobile communication system (6G).

[0075] [Light source device] The light source device 5043 is a device capable of emitting light in a predetermined wavelength band and includes, for example, multiple light sources and a light source optical system that guides the light from the multiple light sources. The light sources are, for example, a xenon lamp, an LED light source, or an LD light source. The light source device 5043 has, for example, LED light sources corresponding to the three primary colors R, G, and B, and emits white light by controlling the output intensity and output timing of each light source. The light source device 5043 may also include a light source that can emit special light used for special light observation, in addition to a light source that emits normal light used for normal light observation. The special light is light in a predetermined wavelength band different from the normal light used for normal light observation, such as near-infrared light (light with a wavelength of 760 nm or more), infrared light, blue light, or ultraviolet light. The normal light is, for example, white light or green light. Narrowband light observation, which is a type of special light observation, alternately emits blue light and green light, allowing high-contrast imaging of specific tissues, such as blood vessels on the surface of mucous membranes, by utilizing the wavelength-dependence of light absorption in body tissue. Furthermore, in fluorescence observation, which is a type of special light observation, excitation light that excites a reagent injected into body tissue is irradiated, and fluorescence emitted by the body tissue or the reagent is received to obtain a fluorescence image, thereby making it easier for the surgeon to visualize body tissue that is difficult for the surgeon to visualize under normal light. For example, in infrared observation using infrared light, near-infrared light is irradiated as excitation light that excites a reagent such as indocyanine green (ICG) injected into body tissue, making it easier to visualize structures deep within the body tissue. In addition, in fluorescence observation, a reagent (e.g., PDD or 5-ALA) that is excited by special light in the blue wavelength band and emits fluorescence in the red wavelength band may be used. The type of irradiated light for the light source device 5043 is set under the control of the CCU 5039. The CCU 5039 may have a mode in which normal light observation and special light observation are alternately performed by controlling the light source device 5043 and the endoscope 5001. In this case, it is preferable that information based on pixel signals obtained under special light observation be superimposed on pixel signals obtained under normal light observation.

[0076] [Recording Device] The recording device 5053 is a device, such as a recorder, that records pixels acquired from the CCU 5039. The recording device 5053 records images acquired from the CCU 5039 on a HDD, an SDD, or an optical disk. The recording device 5053 may be connected to a network within the hospital so as to be accessible from devices outside the operating room. The recording device 5053 may also have an image down-conversion or up-conversion function.

[0077] [Display device] The display device 5041 is a device capable of displaying an image, such as a display monitor. Under the control of the CCU 5039, the display device 5041 displays an image based on pixel signals that have been subjected to image processing by the CCU 5039. The display device 5041 may also function as an input device that enables gaze recognition, voice recognition, and instruction input using gestures by including a camera and a microphone.

[0078] [Output device] The output device 5055 is a device, such as a printer, that outputs information acquired from the CCU 5039. The output device 5055 prints, for example, a print image based on the pixel signal acquired from the CCU 5039 onto paper.

[0079] [Support device] The support device 5027 is an articulated arm including a base 5029 having an arm control device 5045, an arm 5031 extending from the base 5029, and a holder 5032 attached to the tip of the arm 5031. The arm control device 5045 is configured with a processor such as a CPU and controls the drive of the arm 5031 by operating according to a predetermined program. The support device 5027 controls the position and posture of the endoscope 5001 held by the holder 5032, for example, by controlling parameters such as the length of each link 5035 constituting the arm 5031 and the rotation angle and torque of each joint 5033 using the arm control device 5045. This allows the endoscope 5001 to be changed to a desired position or posture, allowing the scope 5003 to be inserted into the patient 5071 and the observation area within the body to be changed. The support device 5027 functions as an endoscope support arm that supports the endoscope 5001 during surgery. This allows the support device 5027 to take the place of an assistant scopist who holds the endoscope 5001. The support device 5027 may also be a device that supports a microscope device 5301, which will be described later, and may also be called a medical support arm. The control of the support device 5027 may be an autonomous control method by the arm control device 5045, or a control method in which the arm control device 5045 controls the support device 5027 based on user input. For example, the control method may be a master-slave method in which the support device 5027 as a slave device is controlled based on the movement of a master device in the user's hand. The support device 5027 may also be remotely controlled from outside the operating room.

[0080] The above describes an example of the endoscope system 5000 to which the technology according to the present disclosure can be applied. For example, the technology according to the present disclosure may be applied to a microscope system.

[0081] [Microscope system] 9 is a diagram showing an example of a schematic configuration of a microsurgical system to which the technology according to the present disclosure can be applied. In the following description, components similar to those in the endoscope system 5000 are assigned the same reference numerals, and redundant description thereof will be omitted.

[0082] 9 shows a schematic diagram of an operator 5067 performing surgery on a patient 5071 on a patient bed 5069 using a microsurgical system 5300. For simplicity, the illustration of the cart 5037 of the microsurgical system 5300 is omitted, and the illustration of the microscope device 5301 that replaces the endoscope 5001 is simplified. However, the microscope device 5301 in this description may refer to the microscope unit 5303 provided at the tip of the link 5035, or may refer to the entire configuration including the microscope unit 5303 and the support device 5027.

[0083] 9, during surgery, a microsurgery system 5300 is used to display an enlarged image of the surgical site captured by a microscope device 5301 on a display device 5041 installed in an operating room. The display device 5041 is installed in a position facing the surgeon 5067, who performs various procedures on the surgical site, such as resecting the affected area, while observing the state of the surgical site using the image displayed on the display device 5041.

[0084] The above describes examples of an endoscopic system 5000 and a microsurgery system 5300 to which the technology of the present disclosure can be applied. Note that the systems to which the technology of the present disclosure can be applied are not limited to these examples. For example, the support device 5027 may support another observation device or another surgical tool at its tip instead of the endoscope 5001 or the microscope unit 5303. Examples of such other observation devices include forceps, a surgeon, an insufflation tube for insufflation, or an energy treatment tool for incising tissue or sealing blood vessels by cauterization. Supporting these observation devices and surgical tools with a support device allows them to be more stably fixed in position than when medical staff support them manually, and also reduces the burden on medical staff. The technology of the present disclosure may be applied to support devices that support components other than the microscope unit.

[0085] Of the configurations described above, the technology according to the present disclosure can be suitably applied to the endoscope 5001 and the microscope device 5301. This makes it possible to capture medical images that allow better observation of the surgical field, and enables surgery to be performed more safely and reliably.

[0086] <Configuration combination example> The present technology can also be configured as follows. (1) a first image sensor that receives light that has passed through a color filter and outputs a first image signal; a second image sensor that receives light without passing through a color filter and outputs a second image signal; a branching optical system that branches incident light that has entered through a mount surface into light that is incident on the first image sensor and light that is incident on the second image sensor; a signal processing unit that performs depth of field extension processing to generate an extended depth of field image by using the first image signal and the second image signal; Equipped with The optical path length from the mount surface to the first image sensor is shorter than the optical path length from the mount surface to the second image sensor. Medical imaging systems. (2) the branching optical system has an optical element that reflects a part of the light and transmits a part of the light, The optical element has a transmittance set according to a difference in sensitivity between the first image sensor and the second image sensor. The medical imaging system according to (1) above. (3) The signal processing unit detects continuity from the second image signal and performs development processing on the first image signal using the result of the detection. A medical imaging system according to (1) or (2) above. (4) The signal processing unit obtains a difference between adjacent pixels in at least one of the horizontal, vertical, and diagonal directions, and performs continuity detection by determining that the direction in which the pixel difference is smallest has continuity. The medical imaging system according to (3) above. (5) The signal processing unit calculates a contrast difference between the first image signal and the second image signal and performs processing to reduce the contrast difference. The medical imaging system according to (3) above. (6) The signal processor changes parameters and methods of constancy detection based on a mode selected by a user or a surgical situation obtained from image recognition. The medical imaging system according to (3) above. (7) further comprising a third image sensor that receives light that has passed through a color filter and outputs a third image signal; The optical path length from the mount surface to the second image sensor is longer than the optical path length from the mount surface to the third image sensor. A medical imaging system according to any one of (1) to (6) above. (8) The signal processing unit creates one all-pixel image using the Gr and Gb pixel values among the pixel values of R, Gr, Gb, and B constituting the second image signal and the third image signal, and performs continuity detection. The medical imaging system according to (7) above. (9) a first image sensor that receives light that has passed through a color filter and outputs a first image signal; a second image sensor that receives light without passing through a color filter and outputs a second image signal; a branching optical system that branches incident light that has entered through a mount surface into light that is incident on the first image sensor and light that is incident on the second image sensor; a signal processing unit that performs depth of field extension processing to generate an extended depth of field image by using the first image signal and the second image signal; Equipped with The optical path length from the mount surface to the first image sensor is shorter than the optical path length from the mount surface to the second image sensor. Medical imaging equipment. (10) 1. A method of operating a medical imaging system, comprising: receiving light that is branched from incident light that is incident from the mount surface and outputting a first image signal by a first image sensor; receiving light branched separately from the light, and outputting a second image signal by a second image sensor; performing a depth of field extension process for generating an extended depth of field image with an extended depth of field using the first image signal and the second image signal; Including, The medical imaging system is configured such that an optical path length from the mount surface to the first imaging element is shorter than an optical path length from the mount surface to the second imaging element. How it works.

[0087] It should be noted that the present embodiment is not limited to the above-described embodiment, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained. [Explanation of symbols]

[0088] 11 medical imaging system, 12 medical imaging device, 13 signal processing device, 20 mount, 21 branching optical system, 22 imaging module, 31 first prism, 32 second prism, 33 half mirror, 34 third prism, 35 half mirror, 41 housing, 42 cover glass, 43 imaging element, 44 filter

Claims

1. a first image sensor that receives light that has passed through a color filter and outputs a first image signal; a second image sensor that receives light without passing through a color filter and outputs a second image signal; a branching optical system that branches incident light that has entered through a mount surface into light that is incident on the first image sensor and light that is incident on the second image sensor; a signal processing unit that performs depth of field extension processing to generate an extended depth of field image by using the first image signal and the second image signal; Equipped with The optical path length from the mount surface to the first image sensor is shorter than the optical path length from the mount surface to the second image sensor. Medical imaging systems.

2. the branching optical system has an optical element that reflects a part of the light and transmits a part of the light, The optical element has a transmittance set according to a difference in sensitivity between the first image sensor and the second image sensor. The medical imaging system of claim 1 .

3. The signal processing unit detects continuity from the second image signal and performs development processing of the first image signal using the result of the detection. The medical imaging system of claim 1 .

4. The signal processing unit acquires a difference between adjacent pixels in at least one of a horizontal, vertical, and diagonal direction, and performs continuity detection by determining that the direction in which the pixel difference is smallest has continuity. The medical imaging system of claim 3 .

5. The signal processing unit calculates a contrast difference between the first image signal and the second image signal and performs processing to reduce the contrast difference. The medical imaging system of claim 3 .

6. The signal processor changes parameters and methods of constancy detection based on a mode selected by a user or a surgical situation obtained from image recognition. The medical imaging system of claim 3 .

7. further comprising a third image sensor that receives light that has passed through a color filter and outputs a third image signal; The optical path length from the mount surface to the third image sensor is longer than the optical path length from the mount surface to the second image sensor. The medical imaging system of claim 1 .

8. The signal processing unit creates one all-pixel image using the Gr and Gb pixel values among the R, Gr, Gb, and B pixel values constituting the second image signal and the third image signal, and performs continuity detection. The medical imaging system of claim 7.

9. a first image sensor that receives light that has passed through a color filter and outputs a first image signal; a second image sensor that receives light without passing through a color filter and outputs a second image signal; a branching optical system that branches incident light that has entered through a mount surface into light that is incident on the first image sensor and light that is incident on the second image sensor; a signal processing unit that performs depth of field extension processing to generate an extended depth of field image by using the first image signal and the second image signal; Equipped with The optical path length from the mount surface to the first image sensor is shorter than the optical path length from the mount surface to the second image sensor. Medical imaging equipment.

Images