Medical imaging system, medical imaging processing method, and medical information processing device

Abstract

A medical imaging system, comprising: a light source configured to irradiate a surgical area with observation light in a first band or special light in a second band different from the first band; an image capture device configured to generate a special light image based on the reflected special light, the reflected special light being the special light reflected from at least a part of the surgical area and received by the image capture device; and a control processing circuit configured to generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the special light image.

Description

Technical Field

[0001] The present technology relates to a medical observation system, a medical observation method, and an information processing device, and more particularly to a medical observation system, a medical observation method, and an information processing device capable of obtaining three-dimensional information with high reliability.

[0002] Cross-reference to Related Applications

[0003] This application claims the benefit of Japanese Priority Patent Application JP2019-115405, filed on Jun. 21, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0004] In surgeries using medical observation devices such as endoscopes and microscopes, the following techniques have been proposed: generating three-dimensional information of a surgical area based on image information or sensor information, and further providing useful information for the surgery to an operator by using the generated three-dimensional information.

[0005] For example, Patent Document 1 discloses a technique of obtaining the optical axis angle information of an endoscope by Simultaneous Localization and Mapping (SLAM) and controlling the image quality of a displayed image.

[0006] Citation List

[0007] Patent Documents

[0008] Patent Document 1: WO 2017 / 168986 Summary of the Invention

[0009] Technical Problem

[0010] Incidentally, in surgeries using endoscopes and microscopes, there is a technique of identifying deep blood vessels or damaged sites that are difficult to see in visible light images based on special light images obtained by using special light in a band different from that of visible light (white light), such as infrared (IR).

[0011] The present technology has been made in view of the above circumstances, and is capable of providing three-dimensional information with high reliability.

[0012] Solution to the Problem

[0013] According to an embodiment of the present technology, there is provided a medical imaging system including: a light source configured to irradiate a surgical area with observation light in a first band or special light in a second band different from the first band; an image capturing device configured to generate a special light image based on the reflected special light, the reflected special light being special light reflected from at least a part of the surgical area and received by the image capturing device; and a control processing circuit configured to generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the special light image.

[0014] According to another embodiment of the present technology, there is provided an information processing device including: a controller that generates three-dimensional information based on a special light image obtained by capturing an image of a surgical area during irradiation with special light from a light source unit, the light source unit irradiating the surgical area with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band.

[0015] In an embodiment of the present technology, the surgical area is irradiated with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band, and three-dimensional information is generated based on the special light image captured during the irradiation with the special light.

[0016] In another embodiment of the present technology, three-dimensional information is generated based on a special light image obtained by capturing an image of a surgical area during irradiation with special light from a light source unit, the light source unit irradiating the surgical area with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a diagram showing a configuration example of a medical observation system according to an embodiment of the present technology.

[0018] Figure 2 is a diagram showing an example of irradiation with special light.

[0019] Figure 3 is a block diagram showing a functional configuration example of a camera control unit (CCU).

[0020] Figure 4 is a flowchart for describing image display processing of a surgical area.

[0021] Figure 5 is a diagram showing an example of irradiation with special light and visible light.

[0022] Figure 6 is a block diagram showing another functional configuration example of the CCU.

[0023] Figure 7 is a diagram showing a configuration example of a prism and a dichroic mirror.

[0024] Figure 8 is a flowchart for describing image display processing of a surgical area.

[0025] Figure 9 is a diagram showing an example of combined SLAM.

[0026] Figure 10 is a block diagram showing yet another functional configuration example of the CCU.

[0027] Figure 11 It is a flowchart for describing the image display processing of the surgical area.

[0028] Figure 12 It is a diagram showing an example of feature points for three-dimensional map generation / position and pose estimation processing.

[0029] Figure 13 It is a diagram showing examples of feature points detected from visible light images and feature points detected from special light images.

[0030] Figure 14 It is a diagram showing feature points superimposed on a special light image.

[0031] Figure 15 It is for describing in Figure 11 The flowchart of the three-dimensional map generation / position and pose estimation processing executed in step S56.

[0032] Figure 16 It is a flowchart for describing different three-dimensional map generation / position and pose estimation processing.

[0033] Figure 17 It is a diagram showing another configuration example of the surgical support system according to the embodiment of the present technology.

[0034] Figure 18 It is a block diagram showing an example of the hardware configuration of the information processing device. Detailed implementation manners

[0035] Hereinafter, embodiments for implementing the present technology will be described. The description will be given in the following order.

[0036] 1. Medical observation system

[0037] 2. Simultaneous localization and mapping (SLAM) with special light

[0038] 3. Combined SLAM

[0039] 4. Variant examples

[0040] 5. Application examples

[0041] <Medical observation system>>

[0042] <System configuration>

[0043] Figure 1 It is a diagram showing an example of the configuration of the medical observation system according to the embodiment of the present technology.

[0044] Figure 1 An example of an endoscopic surgery system for abdominal endoscopic surgery is shown, for example, performing this abdominal endoscopic surgery in a medical environment instead of existing laparotomy.

[0045] In Figure 1 medical observation system 1, instead of performing laparotomy on the abdominal wall as in the past, puncturing tools called trocars 25a and 25b are connected to multiple positions on the abdominal wall. Then, a laparoscope (also referred to as an endoscope hereinafter) 11, an energy treatment tool 22, forceps 23, etc., which are observation medical devices for observing the inside of the patient's body, are inserted into the body through the holes provided for trocars 25a and 25b.

[0046] The operator performs treatment, such as removing the affected part U (tumor, etc.), while viewing an image of the affected part U inside the patient's body in real time, and the image is captured by the endoscope 11. The endoscope 11, the energy treatment tool 22, and the forceps 23 are held by the operator, a robot, etc.

[0047] It should be noted that the operator refers to a health professional who performs surgery in the operating room. The operator includes members participating in the surgery. For example, in addition to surgeons, assistants, scopists, and nurses, there are also doctors who monitor the surgery from other places different from the operating room. In Figure 1 the example, the endoscope 11 is held by a scopist, for example. The endoscope 11 includes: a camera head, and the camera head includes: a lens to be inserted into the patient's body and an imaging device that receives light guided by the lens to perform imaging. It should be noted that the lens can be of a hard type or a soft type. In addition, the lens and the imaging device can be integrated.

[0048] In the operating room where such endoscopic surgery is performed, a cart 31 on which equipment for endoscopic surgery is installed, a hospital bed 33 on which the patient lies, a foot switch 35, etc. are provided. For example, devices such as a camera control unit (CCU) 13, a light source device 17 (light source), a device for surgical tools 21, a pneumoperitoneum device 24, a display device 15, a recorder 26, and a printer 27 are placed on the cart 31 as medical devices.

[0049] An image signal of the affected part U captured by the observation optical system of the endoscope 11 is transmitted to the CCU 13 via a camera cable that is a signal transmission cable. The CCU 13 can be connected to the endoscope 11 via the camera cable and can also be connected to the endoscope 11 via a wireless communication path. The CCU 13 performs signal processing on the image signal output from the endoscope 11 and outputs the image signal on which signal processing has been performed to the display device 15. With this configuration, an image of the surgical area of the affected part U is displayed on the display device 15.

[0050] It should be noted that the CCU 13 outputs the image signal that has undergone signal processing to the recorder 26, so that the recorder 26 records the surgical area image of the affected part U as image data (for example, the data of a moving image). In addition, the CCU 13 outputs the image signal that has undergone signal processing to the printer 27, so that the printer 27 prints the surgical area image of the affected part U.

[0051] The light source device 17 is connected to the endoscope 11 via an optical fiber cable and irradiates the affected part U with light of various wavelengths when switched. For example, the light from the light source device 17 can be used as auxiliary light.

[0052] The surgical tool device 21 corresponds to, for example, a high-frequency output device that outputs a high-frequency current to the energy treatment tool 22, and the energy treatment tool 22 cuts the affected part U using electrothermal energy.

[0053] The pneumoperitoneum device 24 includes a gas supply / intake device and supplies gas to, for example, the abdominal area inside the patient's body.

[0054] The foot switch 35 controls the CCU 13, the surgical tool device 21, etc. using the foot operation of the operator, assistant, etc. as a trigger signal.

[0055] <Image Processing in a Medical Observation System>

[0056] In the CCU 13 of the medical observation system having such a configuration, SLAM is performed based on the image obtained by imaging the surgical area using the endoscope 11. Through SLAM, a three-dimensional map is generated, that is, three-dimensional information indicating the shape of the space including the surgical area inside the patient's body is generated, and the position and posture of the endoscope 11 at each moment are estimated. The three-dimensional information is information including three-dimensional coordinate information (for example, a three-dimensional map).

[0057] For example, when displaying images of organs, etc. obtained by computed tomography (CT), magnetic resonance imaging (MRI), etc. respectively, the estimation result of the position and posture of the endoscope 11 obtained by SLAM is used to control the orientation of the image. The organ images obtained by CT, etc. are displayed on the display device 15 in an orientation corresponding to the position and posture of the endoscope 11.

[0058] For example, SLAM in the CCU 13 is performed based on a special light image, which is an image captured by irradiating a surgical area with special light. Since SLAM is performed based on the special light image captured during the irradiation with special light, this SLAM can be called special light SLAM. It should be noted that the algorithm of image-based SLAM only needs to be an algorithm that can generate a three-dimensional map and its own position and pose on a three-dimensional map based on the feature values in the image, and existing algorithms can be adopted. For example, an algorithm such as machine learning SLAM can be adopted, which generates a three-dimensional map and estimates its own position by using parallel tracking and mapping (PTAM), dense tracking and mapping (DTAM), oriented FAST and rotated BRIEF (ORB)-SLAM, or machine learning using a multi-layer neural network. In addition, not only images can be combined, but also information from an inertial measurement unit (IMU) sensor or a depth sensor can be combined.

[0059] Figure 2 FIG. is a diagram showing an example of irradiation with special light.

[0060] As Figure 2 shown by the arrow of, the special light of the light source device 17 as the light source irradiates the surgical area from the tip of the endoscope 11, and imaging is performed in this state.

[0061] If infrared (IR) light is used as the special light, as Figure 2 shown by the arrow of, the special light does not reflect on the surface of the surgical area. The special light reaches a predetermined structure such as a blood vessel deep inside and reflects on the deep structure. The deep structure that does not appear (is difficult to appear) in the visible light image captured by visible light irradiation is highlighted in the special light image. It should be noted that, for example, infrared light is near-infrared light and has a wavelength band with a peak wavelength of 760 nm to 800 nm.

[0062] In the CCU 13, SLAM is performed based on the special light image, and a three-dimensional map is generated, in which the edges of deep structures and the like are used as feature points. In addition, the generated three-dimensional map is used to estimate the position and pose.

[0063] The surface shape of the surgical area changes through treatments such as resection of damaged parts during the surgery, while the blood vessels in deep or similar parts are basically preserved and unchanged. When performing SLAM based on the special light illumination image, a three-dimensional map can be generated according to the feature points that are less affected by the changes in the surgical area, and a SLAM result with high reliability can be obtained.

[0064] It should be noted that various types of light in a band (the first band) different from the band of visible light (the second band) are used for the special light. The details of the special light will be described later.

[0065] <Special Light SLAM>

[0066] <Example of Performing Special Light Observation>

[0067] Functional Configuration of CCU

[0068] Figure 3 is a block diagram showing an example of the functional configuration of CCU 13.

[0069] As Figure 3 shown, an information processing unit 71 is implemented in CCU 13. For example, when a central processing unit (CPU) constituting CCU 13 executes a predetermined program stored in a memory, the information processing unit 71 is implemented. In other words, the functions of CCU 13 are implemented by a control processing circuit. The control processing circuit is, for example, a circuit including a CPU and a memory.

[0070] The information processing unit 71 includes: a special light development processing unit 81, a development processing unit 82 for special light observation, a three-dimensional map generation unit 83, a three-dimensional map storage unit 84, a screen position calculation unit 85, a three-dimensional information storage unit 86, and a display controller 87.

[0071] An image signal output from an image sensor 62 for special light of an imaging unit 51 (image capturing device) constituting an endoscope 11 (medical imaging device) is input to the special light development processing unit 81 and the development processing unit 82 for special light observation. The imaging unit 51 includes a special light irradiation unit 61 and an image sensor 62 for special light. The special light irradiation unit 61 irradiates a surgical area with special light. The image sensor 62 for special light images the surgical area during irradiation with special light.

[0072] The special light development processing unit 81 generates an RGB image as a special light image based on the original signal provided from the image sensor 62 for special light, and outputs the data of the special light image to the three-dimensional map generation unit 83.

[0073] The development processing unit 82 for special light observation generates an observation image based on the original signal provided from the image sensor 62 for special light, and the observation image visualizes the state of the surgical area. In the observation image, the state of the surgical area is displayed in a visible manner. In Figure 3 the example, special light is used as observation light. The data of the observation image generated by the development processing unit 82 for special light observation is provided to the display controller 87.

[0074] The three-dimensional map generation unit 83 performs special light SLAM based on the special light image. For example, the surgical area irradiated with the special light is repeatedly imaged. Using the special light images sequentially provided from the special light development processing unit 81, the special light SLAM performed by the three-dimensional map generation unit 83 is executed.

[0075] The three-dimensional map generation unit 83 analyzes the special light image to set points having a predetermined eigenvalue as feature points, thereby generating a three-dimensional map including these feature points. The three-dimensional map generation unit 83 outputs the generated three-dimensional map to the three-dimensional map storage unit 84 and causes the three-dimensional map storage unit 84 to store the three-dimensional map. The three-dimensional map stored in the three-dimensional map storage unit 84 is updated according to the processing order of the three-dimensional map generation unit 83.

[0076] In addition, the three-dimensional map generation unit 83 estimates the position and orientation of the endoscope 11 based on the three-dimensional map stored in the three-dimensional map storage unit 84 and the like, and outputs position / orientation information (information indicating the estimation result of the position and orientation) to the on-screen position calculation unit 85.

[0077] The on-screen position calculation unit 85 reads the three-dimensional map of the organ pre-generated using CT, MRI, etc. before the surgery from the three-dimensional information storage unit 86 to obtain the three-dimensional map. In addition, the on-screen position calculation unit 85 obtains three-dimensional position information corresponding to the navigation information input to the three-dimensional information storage unit 86 or three-dimensional position information specified on the screen by the operator as a user, for example.

[0078] The on-screen position calculation unit 85 calculates the position and orientation of the endoscope 11 and the direction of the three-dimensional map, where the position and orientation of the endoscope 11 are indicated by the position / orientation information provided from the three-dimensional map generation unit 83, and the direction of the three-dimensional map corresponds to the position indicated by the three-dimensional position information, etc. The on-screen position calculation unit 85 transforms the three-dimensional map so as to be viewed according to the calculated direction, and outputs the transformed three-dimensional map to the display controller 87. Calculating the direction of the three-dimensional map means calculating the rotation of the three-dimensional map. The transformed three-dimensional map is a two-dimensional image of the three-dimensional map viewed from a predetermined direction.

[0079] The display controller 87 synthesizes the three-dimensional map transformed by the on-screen position calculation unit 85 with the observation image for which data is provided from the development processing unit 82 for special light observation, and causes the display device 15 to display the synthesized observation image.

[0080] Examples of special light

[0081] Here, the special light for capturing the special light image will be described.

[0082] (1) Case of using infrared light (infrared rays)

[0083] If light such as infrared light that can observe deep blood vessel structures is used as the special light, a three-dimensional map that is not affected by changes in the surface state can be obtained. In other words, SLAM that is less affected by surgical treatment is achieved. Similar to infrared light, light with a wavelength longer than that of visible light can be used as the special light. In this case, the wavelength band of the special light is larger than that of visible light.

[0084] (2) Case of using blue light

[0085] If blue light that can highlight superficial blood vessels for observation is used as the special light, feature points for SLAM are set for blood vessels and the like that appear in the special light image. Since blood vessels are basically preserved during the operation, SLAM that is less affected by surgical treatment is achieved.

[0086] (3) Case of using high-transmittance light

[0087] If light with a higher transmittance than that of visible light, i.e., high-transmittance light, is used as the special light, even if fog or mist appears inside the organ, a special light image that vividly shows the surgical area can be obtained. Since feature points can be obtained based on the special light image, interruption of SLAM can be suppressed. Depending on the details of the treatment (such as the appearance of fog inside the organ), and in some cases where visible light images are used, it is difficult to continue SLAM. This can be prevented from happening.

[0088] (4) Case of using polarized light as the special light

[0089] If polarized light is used as the special light, feature points in the specular reflection area can be obtained from the special light image. This allows SLAM to be performed in an area including the specular reflection area. For example, during the operation, the organ has fluid, and an area that causes specular reflection can be formed in the surgical area. If visible light images are used, it is difficult to detect feature points in the specular reflection area in some cases. This situation can be prevented by using the special light image.

[0090] The polarized light used as the special light is generated, for example, by using a polarization filter. The visible light emitted from the light source passes through the polarization filter, thereby generating polarized light.

[0091] (5) Case of using light that forms a known spatial pattern as the special light

[0092] Light (structured light) that projects a known spatial pattern (such as a checkerboard pattern or a dot pattern) can be used as the special light. In this case, the three-dimensional shape of the surgical area can be detected more accurately.

[0093] (6) Case of using pulse-modulated light as the special light

[0094] If pulsed modulated light is used as the special light, the distance to the surgical area can be directly measured based on the phase difference between the reflected light and the irradiated light. A three-dimensional map is generated based on the distance measurement results for each position.

[0095] As described above, various types of light in a band different from that of visible light can be used as the special light.

[0096] It should be noted that visible light is white light and has a band such that, for example, the lower limit falls within a predetermined wavelength in the range of approximately 360 nm to 400 nm and the upper limit falls within a predetermined wavelength in the range of approximately 760 nm to 830 nm. Various types of light in a band different from that of such visible light can be used as the special light.

[0097] Alternatively, instead of using light in a band different from that of visible light as the special light, light with a different light source type (such as an LED, a laser, etc.) from that of visible light can be used as the special light. Alternatively, light with an irradiation intensity different from that of visible light can also be used as the special light. Light used for emitting a spatial pattern different from the spatial pattern emitted using visible light can also be used as the special light.

[0098] Operation of the CCU

[0099] Here, the surgical area image display process of the CCU 13 will be described with reference to Figure 4 the flowchart of.

[0100] In step S1, the special light irradiation unit 61 of the imaging unit 51 irradiates the surgical area with the special light.

[0101] In step S2, the image sensor 62 for the special light images the surgical area during the irradiation with the special light.

[0102] In step S3, the special light development processing unit 81 of the information processing unit 71 generates a special light image based on the raw signal provided from the image sensor 62 for the special light.

[0103] In step S4, the development processing unit 82 for special light observation generates an observation image based on the raw signal provided from the image sensor 62 for the special light. This observation image can be the same as the special light image in S2 or a different special light image.

[0104] In step S5, the three-dimensional map generation unit 83 analyzes the special light image provided from the special light development processing unit 81 to generate a three-dimensional map.

[0105] In step S6, the three-dimensional map generation unit 83 estimates the position and posture of the endoscope 11 based on the three-dimensional map.

[0106] In step S7, the on-screen position calculation unit 85 converts the orientation of the three-dimensional map based on the estimation results of the position and orientation by the three-dimensional map generation unit 83.

[0107] In step S8, the display controller 87 synthesizes the converted three-dimensional map provided by the on-screen position calculation unit 85 and the observation image provided by the development processing unit 82 for special light observation.

[0108] In step S9, the display controller 87 causes the display device 15 to display the synthesized observation image.

[0109] By special light SLAM, that is, SLAM based on the above-mentioned special light images, highly reliable SLAM results can be obtained.

[0110] <Example of performing visible light observation>

[0111] An observation image can be generated based on the image signal obtained by receiving the reflected light of visible light. In this example, visible light is used as the observation light.

[0112] Figure 5 It is a diagram showing an example of irradiation with special light and visible light.

[0113] If visible light is used as the observation light (as Figure 5 shown), the surgical area is irradiated with special light and visible light from the tip of the endoscope 11, and imaging is performed in this state. As described above, the special light image obtained by receiving the reflected light of special light is used for SLAM, and the visible light image obtained by receiving the reflected light of visible light is used as the observation image.

[0114] Functional configuration of CCU

[0115] Figure 6 It is a block diagram showing another example of the functional configuration of the CCU 13.

[0116] In Figure 6 the configuration shown, the same configurations as those described with reference to Figure 3 are denoted by the same reference numerals. Overlapping descriptions will be omitted as appropriate.

[0117] Figure 6 The configuration of the information processing unit 71 shown in Figure 3 is different from the configuration shown in

[0118] In addition to the special light irradiation unit 61 and the image sensor 62 for special light, the imaging unit 51 further includes a visible light irradiation unit 63 and an image sensor 64 for visible light. The visible light irradiation unit 63 irradiates the surgical area with visible light. The image sensor 64 for visible light images the surgical area during the visible light irradiation. For example, the irradiation with special light by the special light irradiation unit 61 and the irradiation with visible light by the visible light irradiation unit 63 are performed simultaneously. The special light irradiation by the special light irradiation unit 61 and the visible light irradiation by the visible light irradiation unit 63 may be performed in a time-division manner.

[0119] As Figure 7 shown, the prism 102 and the dichroic mirror 103 are disposed in front of the image sensor 62 for special light and the image sensor 64 for visible light. Among the light passing through the lens 101 of the lens provided in the endoscope 11, the special light is reflected by the dichroic mirror 103 and guided to the image sensor 62 for special light. In addition, among the light passing through the lens 101, the visible light passes through the dichroic mirror 103 and is guided to the image sensor 64 for visible light.

[0120] As Figure 6 shown, the image signal output from the image sensor 62 for special light is input to the special light development processing unit 81, and the image signal output from the image sensor 64 for visible light is input to the visible light development processing unit 91.

[0121] The visible light development processing unit 91 generates a visible light image based on the original signal provided from the image sensor 64 for visible light, and outputs the visible light image as an observation image indicating the state of the surgical area. The data of the visible light image output from the visible light development processing unit 91 is provided to the display controller 87 and the alignment processing unit 92.

[0122] The alignment processing unit 92 performs alignment in which the position of each pixel of the special light image provided from the special light development processing unit 81 is electronically aligned with the position of each pixel of the visible light image provided from the visible light development processing unit 91. The alignment processing unit 92 outputs the special light image obtained after alignment to the three-dimensional map generation unit 83.

[0123] The display controller 87 synthesizes the three-dimensional map converted by the screen position calculation unit 85 with the visible light image, the data of which is provided from the visible light development processing unit 91, and causes the display device 15 to display the synthesized visible light image.

[0124] Operation of the CCU

[0125] Here, with reference to Figure 8 the flowchart of Figure 6Surgical area image display processing of the configured CCU 13.

[0126] In step S21, the special light irradiation unit 61 of the imaging unit 51 irradiates the surgical area with special light. In addition, the visible light irradiation unit 63 irradiates the surgical area with visible light.

[0127] In step S22, the image sensor 62 for special light images the surgical area during the irradiation with special light. In addition, the image sensor 64 for visible light images the surgical area during the irradiation with visible light.

[0128] In step S23, the special light development processing unit 81 of the information processing unit 71 generates a special light image based on the original signal provided from the image sensor 62 for special light.

[0129] In step S24, the visible light development processing unit 91 generates an observation image (visible light image) based on the original signal provided from the image sensor 64 for visible light.

[0130] In step S25, the alignment processing unit 92 performs alignment based on the special light image provided from the special light development processing unit 81 and the visible light image provided as the observation image from the visible light development processing unit 91.

[0131] In step S26, the three-dimensional map generation unit 83 analyzes the special light image provided from the alignment processing unit 92 to generate a three-dimensional map.

[0132] In step S27, the three-dimensional map generation unit 83 estimates the position and orientation of the endoscope 11 based on the three-dimensional map.

[0133] In step S28, the on-screen position calculation unit 85 transforms the orientation of the three-dimensional map based on the estimation results of the position and orientation by the three-dimensional map generation unit 83.

[0134] In step S29, the display controller 87 synthesizes the transformed three-dimensional map provided from the on-screen position calculation unit 85 and the observation image provided from the visible light development processing unit 91.

[0135] In step S30, the display controller 87 causes the display device 15 to display the synthesized observation image.

[0136] As described above, providing the visible light irradiation unit 63 and the image sensor 64 for visible light to the imaging unit 51 allows the use of the visible light image as the observation image.

[0137] <<Combined SLAM>>

[0138] The generation of a three-dimensional map and the estimation of position and pose can be performed by combining and utilizing special light SLAM (SLAM using special light images) and visible light SLAM (SLAM using visible light images). The SLAM that combines and utilizes special light SLAM and visible light SLAM is combined SLAM.

[0139] Figure 9 FIG. is a diagram showing an example of combined SLAM.

[0140] Figure 9 In the upper row of, the left image is a visible light image, and the right image in the upper row is a special light image. Figure 9 The visible light image and the special light image shown are both images obtained by imaging the same range. In addition, each of the rhombuses shown in the image is a feature point detected by analyzing the image. Depending on the wavelength of the light used for each imaging, the details appearing in the image are different, and the feature points are set at different positions accordingly.

[0141] In Figure 9 the visible light image of, the regions A1 and A2 surrounded by the dashed line are regions where specular reflection is caused by the emitted visible light. Regions A1 and A2 appear as bright spots in the visible light image. In this example, many feature points are set at the positions of the respective bright spots.

[0142] Meanwhile, in Figure 9 the special light image of, the region A11 surrounded by the dashed line is a region where deep vascular structures appear. In this example, many feature points are set near the edges of the vascular structures.

[0143] As Figure 9 indicated by the arrow in, combined SLAM is SLAM that uses feature points detected from a visible light image and feature points detected from a special light image. The position and pose of the endoscope 11 when capturing a visible light image are the same as the position and pose of the endoscope 11 when capturing a special light image, and thus feature points detected from two or more images can be easily superimposed on the same coordinate system.

[0144] The feature points detected from the visible light image and the feature points detected from the special light image are different from each other in position, and thus combined SLAM allows for the generation of a more robust three-dimensional map.

[0145] In combined SLAM, not all feature points detected from the visible light image and feature points detected from the special light image are used, but rather feature points with high reliability are used.

[0146] A three-dimensional map can be generated based on feature points with high reliability, or the position and pose can be estimated based on feature points with high reliability among the feature points constituting the three-dimensional map.

[0147] In the latter case, for example, a three-dimensional map including feature points on a visible light image and a three-dimensional map including feature points on a special light image are generated, and the two three-dimensional maps are synthesized to generate a three-dimensional map of the combined SLAM. Among the feature points constituting the three-dimensional map of the combined SLAM, feature points with high reliability are used to estimate the position and pose.

[0148] Functional configuration of the CCU

[0149] Figure 10 is a block diagram showing another functional configuration example of the CCU 13.

[0150] In Figure 10 the configuration shown, the same configuration as the configuration described in the reference Figure 6 is denoted by the same reference numerals. Overlapping descriptions will be appropriately omitted.

[0151] The three-dimensional map generation unit 83 acquires the special light image obtained after alignment provided from the alignment processing unit 92 and the visible light image provided from the visible light development processing unit 91. The three-dimensional map generation unit 83 detects the feature points on the special light image and the feature points on the visible light image, and generates a three-dimensional map based on the detected feature points.

[0152] For example, the three-dimensional map generation unit 83 calculates the reliability of the feature points on the special light image and the reliability of the feature points on the visible light image, and generates a three-dimensional map including the feature points with high reliability.

[0153] The calculation of the reliability is as described below, for example: Calculate the eigenvalue difference between the feature points detected from a certain frame and the corresponding feature points detected from the previous frame; Based on this difference, when the eigenvalue difference becomes small, a higher value is set for the reliability, and when the eigenvalue difference becomes large, a lower value is set for the reliability.

[0154] For example, in the case where both the feature points detected from the special light image and the feature points detected from the visible light image are included within a predetermined range (in the case where the two feature points are in close positions), the three-dimensional map generation unit 83 selects the feature points with higher reliability as the feature points constituting the three-dimensional map. For the method of selecting the feature points constituting the three-dimensional map, various methods described later can be adopted.

[0155] The three-dimensional map generation unit 83 estimates the position and pose of the endoscope 11 based on the three-dimensional map including the selected feature points as described above, and outputs the position / pose information to the on-screen position calculation unit 85.

[0156] The on-screen position calculation unit 85 calculates the direction of the three-dimensional map based on the position, posture, etc. of the endoscope 11 indicated by the position / pose information provided by the three-dimensional map generation unit 83. The on-screen position calculation unit 85 converts the three-dimensional map according to the calculated direction so that the three-dimensional map is visible, and outputs the converted three-dimensional map to the display controller 87.

[0157] The display controller 87 synthesizes the three-dimensional map converted by the on-screen position calculation unit 85 with the visible light image for which data is provided by the visible light imaging processing unit 91, and then causes the display device 15 to display the synthesized visible light image.

[0158] In the case where a three-dimensional map including the feature points of the visible light image and a three-dimensional map including the feature points of the special light image are generated, the display controller 87 can display the three-dimensional map including the feature points of the visible light image. For example, such a display can be performed in which the information on the blood vessels detected from the special light image is superimposed on the three-dimensional map including the feature points of the visible light image.

[0159] Operation of CCU

[0160] Here, the surgical area image display process of the CCU 13 having the Figure 11 configuration will be described with reference to the Figure 10 flowchart.

[0161] The processing from step S51 to step S55 is similar to the processing from Figure 8 step S21 to step S25. The special light image obtained after alignment and the visible light image captured during illumination with visible light are provided to the three-dimensional map generation unit 83.

[0162] In step S56, the three-dimensional map generation unit 83 performs three-dimensional map generation / position and pose estimation processing. The three-dimensional map generation / position and pose estimation processing performed here is processing for combining SLAM. Details of the three-dimensional map generation / position and pose estimation processing will be described later.

[0163] The processing from step S57 to step S59 is similar to the processing from Figure 8 step S28 to step S30. The synthesis of the three-dimensional map is performed using the estimation results of the position and pose obtained by the three-dimensional map generation / position and pose estimation processing, and the visible light image obtained after the synthesis of the three-dimensional map is displayed.

[0164] Three-dimensional map generation / position and pose estimation processing

[0165] Figure 12 is a diagram showing an example of the feature points for the three-dimensional map generation / position and pose estimation processing.

[0166] AsFigure 12 As shown, in the three-dimensional map generation / position and posture estimation process, a three-dimensional map is generated so that the feature points X detected from the visible light image at a specific time t n t and the feature points Y detected from the special light image m The three-dimensional graph includes the feature points Z 1 t, …, Z p t, these feature points are retained as feature points with high reliability.

[0167] Figure 13 is a diagram showing examples of feature points detected from a visible light image and feature points detected from a special light image.

[0168] Figure 13 The upper row shows feature points detected from the visible light image. Figure 13 The lower row shows feature points detected from the special light image.

[0169] As shown in the visible light image on the right, the feature point X detected from the visible light image of the frame at time t (the visible light image of the tth frame) n t The eigenvalues ​​of n t In addition, as shown in the visible light image on the left, the feature point X detected from the visible light image of the frame at time t-1 (the visible light image of the (t-1)th frame) n t-1 The eigenvalues ​​of n t-1 express.

[0170] According to the eigenvalue x n t and the eigenvalue x n t-1 Calculate each feature point X detected from the visible light image n t The reliability of the eigenvalue x n t-1 is the corresponding feature point X of the frame before the frame n t-1 , as shown by connecting them with horizontal lines.

[0171] Similarly, as shown in the special light image on the right, the feature point Y detected from the special light image of the tth frame m t The eigenvalues ​​of m t In addition, as shown in the special light image on the left, the feature point Y detected from the special light image of the (t-1)th frame mt-1 The eigenvalue is represented by eigenvalue y m t-1 .

[0172] Based on eigenvalue y m t and eigenvalue y m t-1 calculate the reliability of each feature point Y detected from the special light image, where this eigenvalue y m t is the eigenvalue of the corresponding feature point Y in the previous frame m t-1 as shown by connecting them to each other with a horizontal line. m t-1

[0173] Figure 14 is a diagram showing the feature points superimposed on the special light image.

[0174] When feature point X n t-1 and feature point Y m t-1 are superimposed on the special light image of the (t - 1)th frame, the image is as shown on the left side of Figure 14 .

[0175] Meanwhile, when feature point X n t and feature point Y m t are superimposed on the special light image of the tth frame, the image is as shown on the right side of Figure 14 . In the special light image of the tth frame shown on the right side of Figure 14 , not only feature point X n t and feature point Y m t are shown, but also the feature points with high reliability among feature point X n t-1 and feature point Y m t-1 are shown.

[0176] For example, the feature point at position P1 is the feature point Y with high reliability m t-1 , and the feature point at position P2 is the feature point X with high reliability n t-1 . Based on the feature points shown on the special light image on the right side of Figure 14 , a 3D diagram at time t is generated.

[0177] Reference will be made to the flowchart in Figure 15 for the description in Figure 11 ​The 3D map generation / position and orientation estimation process executed in step S56.

[0178] As Figure 15 shown, for example, the processing for visible light images from step S101 to S104 and the processing for special light images from step S105 to S108 are executed in parallel.

[0179] In step S101, Figure 10 the 3D map generation unit 83 acquires the visible light image provided by the visible light development processing unit 91.

[0180] In step S102, the 3D map generation unit 83 extracts N feature points from each of the images of the t-th frame and the (t - 1)-th frame, and obtains the set of eigenvalue {x 1 t ,…,x N t} and {x 1 t-1 ,…,x N t-1} of each image.

[0181] In step S103, the 3D map generation unit 83 compares the eigenvalue x n t of the feature point X n t in the t-th frame with the set of extracted eigenvalues {x 1 t-1 ,...,x N t-1} in the (t - 1)-th frame, and searches for the corresponding feature point with the minimum distance. The eigenvalue of the corresponding feature point is x n t-1 .

[0182] For the distances dx n t (x n t ,x n t-1 ), dy m t (y m t ,y m t-1 ), the Euclidean distance (L2 norm), Hamming distance, etc. can be used. The distance dx n t (x n t ,x n t-1 ) is the eigenvalue x n tThe distance from the eigenvalue x n t-1 In addition, the distance dy m t (y m t , y m t-1 ) is the distance between the eigenvalue y m t and the eigenvalue y m t-1 .

[0183] In step S104, the 3D map generation unit 83 determines the reliability of the feature point X based on the smallness of the distance dx n t from the eigenvalue x of the feature point X in the t-th frame n t and the eigenvalue x of the corresponding feature point in the (t - 1)-th frame n t-1 (x n t , x n t , x n t-1 ). n t For the reliability, the reciprocal value of the distance can be used, or a value obtained by combining the distance and a function such as logarithm can be used. In addition, in order to adjust the influence degrees of normal light and special light, a predetermined coefficient can be set for the reliability of each light.

[0184] The above processing is performed for each frame of the visible light image.

[0185] Meanwhile, in step S105, the 3D map generation unit 83 acquires the special light image obtained after alignment provided by the alignment processing unit 92.

[0186] In step S106, the 3D map generation unit 83 extracts M feature points from each of the images in the t-th frame and the (t - 1)-th frame, and obtains the eigenvalue sets {y

[0187] ,..., y 1 t ,..., y M t}, {y 1 t-1 ,..., y M t-1} of the respective images.

[0188] In step S107, the 3D map generation unit 83 compares the feature point Y in the t-th frame m tThe eigenvalue y m t and the distance from the set of extracted eigenvalues {y 1 t-1 ,..., y M t-1} of the (t - 1)-th frame, and search for the corresponding feature point with the minimum distance. The eigenvalue of the corresponding feature point is y m t-1 .

[0189] In step S108, the 3D map generation unit 83 determines the reliability of the feature point Y m t of the t-th frame according to the smallness of the distance dy m t between the eigenvalue y m t-1 of the feature point Y of the t-th frame and the eigenvalue y m t (y m t , y m t-1 ) of the corresponding feature point of the (t - 1)-th frame. m t

[0190] Perform the above processing for each frame of the special light image.

[0191] In step S109, the 3D map generation unit 83 arranges the feature point X n t in the visible light image and the feature point Y m t in the special light image on the same plane of the t-th frame and the (t - 1)-th frame respectively.

[0192] In step S110, the 3D map generation unit 83 extracts the feature point set {Z 1 t ,..., X N t , Y 1 t ,..., Y M t} from all the feature points {X 1 t ,..., Z P t}, where the feature point set is a set of the top P feature points with higher reliability.

[0193] In step S111, the 3D map generation unit 83 is based on the feature point set {Z 1 t ,..., Z​P t}, and the positions of the corresponding feature points in the t-th frame and the (t-1)-th frame among the feature points constituting the three-dimensional map are used to estimate the position of the endoscope 11 and the set of feature points {Z 1 t ,..., Z P t} at the time of imaging of the t-th frame and the (t-1)-th frame. The estimation of the position of the endoscope 11 and the three-dimensional positions of {Z 1 t ,..., Z P t} is performed when performing bundle adjustment.

[0194] In this case, by using, for example, the Random Sample Consensus (RANSAC) method, values regarded as outliers are removed from the set of feature points {Z 1 t ,..., Z P t}. The set of feature points from which outliers have been removed becomes the set of feature points {Z 1 t ,..., Z P ’ t}.

[0195] In step S112, when a predetermined condition is satisfied, the three-dimensional map generation unit 83 adds the set of feature points {Z 1 t ,..., Z P’ t} to the three-dimensional map. For example, the set of feature points is added based on a condition. For example, the set of feature points {Z 1 t ,..., Z P’ t} has sufficient reliability compared to past feature points.

[0196] The above processing is repeated, thereby performing three-dimensional map generation / position and pose estimation processing.

[0197] The flowchart of the reference Figure 16 describes other three-dimensional map generation / position and pose estimation processing.

[0198] In the method of extracting the set of feature points {Z 1 t ,..., Z P t}, Figure 16 The processing shown is different from the processing described in the reference Figure 15 In other words, the processing from step S121 to S129 is different from that from Figure 15The processing of steps S101 to S109 is similar processing.

[0199] In step S130, the three-dimensional map generation unit 83 selects the top Q feature points with relatively high reliability among all the feature points {X 1 t ,..., X N t , Y 1 t ,..., Y M t}. In addition, the three-dimensional map generation unit 83 counts the number of feature points of the visible light image and the number of feature points of the special light image included in the Q feature points, and selects the feature point set {X 1 t ,..., X N t} or {Y 1 t ,..., Y M t} that includes more feature points. The three-dimensional map generation unit 83 extracts the top P feature points with relatively high reliability from the selected feature point set {X 1 t ,..., X N t} or {Y 1 t ,..., Y M t}, and sets it as the feature point set {Z 1 t ,..., Z P t}.

[0200] As described above, feature points can be selected by various methods and used as feature points for constructing a three-dimensional map at time t.

[0201] As described above, the combined SLAM that combines visible light SLAM and special light SLAM performs the generation of a three-dimensional map and the estimation of position and pose, which allows for an improvement in robustness. Compared with the case of using the SLAM result by a single light source, using the SLAM result by multiple light sources allows for an improvement in accuracy.

[0202] According to the situation, it is possible to switch between the feature points obtained by visible light SLAM and the feature points obtained by special light SLAM, and the feature points obtained by combined SLAM can be used.

[0203] The three-dimensional map obtained by visible light SLAM and the three-dimensional map obtained by special light SLAM can be aligned based on the estimation results of the position and posture of the endoscope 11 by combined SLAM, thereby generating a three-dimensional map. It should be noted that the alignment of the three-dimensional maps obtained by visible light SLAM and special light SLAM can be performed using machine learning. For example, the eigenvalues of visible light SLAM and the eigenvalues of special light SLAM paired with the eigenvalues of visible light SLAM are input as pre-labeled learning data into a machine learning model of a multi-layer neural network to generate parameters. Subsequently, the eigenvalues obtained by visible light SLAM and the eigenvalues obtained by special light SLAM can be input into an alignment algorithm, which is set based on the machine learning model and the parameters, thereby performing alignment.

[0204] The above series of processes enables the generation of a three-dimensional map that hardly changes during the execution of treatment during surgery in special light SLAM using the edges of blood vessel structures in deep parts etc. as feature points, and continuously obtaining highly reliable SLAM results.

[0205] In special light SLAM using special light with high transmittance, even when it is difficult to obtain information on visible light due to fog, mist, etc., SLAM results using the latest information can be continuously obtained.

[0206] In special light SLAM using polarized light, feature points in the specular reflection region can be detected from the special light image, and SLAM results can be obtained.

[0207] <<Variant Example>>

[0208] Combined SLAM can be performed using visible light and multiple types of special light. In this case, multiple types of special light with different wavelength bands are used, and imaging is performed during the irradiation of each type of special light to generate various types of special light images. Combined SLAM is performed based on the feature points detected from the visible light image and the feature points detected from each special light image.

[0209] As described above, the number of special lights to be used in combined SLAM can be set to multiple.

[0210] <<Application Example>>

[0211] <System Configuration>

[0212] Next, with reference to Figure 17 an example will be described in which a surgical video microscope device including an arm is used as an application example of a surgical support system according to an embodiment of the present technology.

[0213] Figure 17An example of a microscopic surgery system using a surgical video microscope device as an observation medical device for observing the inside of a patient's body is shown.

[0214] Figure 17 A state is shown in which a doctor 520 as an operator (user) performs surgery on an object to be operated on (patient) 540 lying on an operating table 530 by using a surgical tool 21 (such as a scalpel, forceps, and tweezers).

[0215] It should be noted that in the following description, an operation is a general term for various types of medical treatments such as surgery and examination, and this operation is performed by a doctor as the user 520 on a patient 540 as the object to be operated on. In addition, although Figure 17 the example shows a case of surgery as an example of an operation, the operation using the video microscope device for surgery 510 is not limited to surgery and can be any other operation.

[0216] A surgical video microscope device 510 according to an embodiment of the present technology is provided beside the operating table 530.

[0217] The surgical video microscope device 510 includes: a base portion 511 as a base, an arm portion 512 extending from the base portion 511, and an imaging unit 515 as a tip unit connected to the tip of the arm portion 512.

[0218] The arm portion 512 includes: a plurality of joints 513a, joint 513b, and joint 513c, a plurality of linkages 514a and 514b coupled to each other through the joints 513a and 513b, and an imaging unit 515 provided at the tip of the arm portion 512.

[0219] For simplicity of description, in the example of Figure 17 , the arm portion 512 includes: three joints 513a to 513c and two linkages 514a and 514b. In fact, while considering the degrees of freedom of the position and posture of the arm portion 512 and the imaging unit 515, the number and shape of the joints 513a to 513c and the linkages 514a and 514b, the directions of the drive shafts of the joints 513a to 513c, etc. can be appropriately set to achieve the desired degrees of freedom.

[0220] The joints 513a to 513c have a function of rotatably coupling the linkages 514a and 514b to each other. When the joints 513a to 513c are rotationally driven, the drive of the arm portion 512 is controlled.

[0221] The imaging unit 515 is connected as a tip unit to the tip of the arm portion 512.

[0222] The imaging unit 515 is a unit including an optical system that obtains an optical image of an object and thus obtains an image of the object to be imaged, and is, for example, a camera configured to be able to capture moving images and still images. As Figure 17 shown, the surgical video microscope device 510 controls the position and posture of the arm 512 and the imaging unit 515 so that the imaging unit 515 provided at the tip of the arm 512 images the state of the operation site of the object to be operated on 540.

[0223] It should be noted that the configuration of the imaging unit 515 connected to the tip of the arm 512 as the tip unit is not particularly limited. For example, the imaging unit 515 can be configured as an endoscope or a microscope. In addition, the imaging unit 515 can be configured to be separable from the arm 512.

[0224] For example, such a configuration can allow the imaging unit 515 (as the tip unit) corresponding to the application to be appropriately connected to the tip of the arm 512. It should be noted that the case where the imaging unit 515 is applied as the tip unit is mainly described here, but needless to say, the tip unit to be connected to the tip of the arm 512 is not necessarily limited to the imaging unit 515.

[0225] In addition, a display device 550 such as a monitor or a display is installed at a position facing the user 520. For example, after various types of image processing are performed on the image by an image processing device built in or externally provided to the surgical video microscope device 510, the image of the operation site obtained by the imaging unit 515 is displayed as an electrical image on the display screen of the display device 550.

[0226] Such a configuration allows the user 520 to perform various types of treatments (e.g., surgeries) while viewing the electrical image of the operation site displayed on the display screen of the display device 550.

[0227] Here, in Figure 17 the example of, the imaging unit 515 includes, for example, the imaging unit 51 described in reference Figure 3 etc. In addition, the image processing device that performs various types of image processing on the image of the operation site obtained by the imaging unit 515 corresponds to an example of the information processing unit 71 described in reference Figure 3 etc.

[0228] <Hardware Configuration>

[0229] Next, an example of the hardware configuration of the information processing device constituting the surgical support system according to the embodiment of the present technology will be described in detail with reference to Figure 18 .

[0230] Figure 18It is a block diagram showing an example of the hardware configuration of an information processing device 900 that constitutes a surgical support system according to an embodiment of the present technology.

[0231] As Figure 18 shown, the information processing device 900 includes: a CPU 901, a ROM 903, and a RAM 905. Additionally, the information processing device 900 includes: a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, and a storage device 919. It should be noted that the information processing device 900 may include: a drive 921, a connection port 923, and a communication device 925.

[0232] The CPU 901 serves as an arithmetic processing device and a control device, and controls all or part of the operations in the information processing device 900 according to various programs recorded in the ROM 903, the RAM 905, or the storage device 919 or on the removable recording medium 927.

[0233] The ROM 903 stores programs, arithmetic parameters, etc. used by the CPU 901. The RAM 905 temporarily stores programs to be used by the CPU 901, parameters that are appropriately changed during program execution, etc. These components are connected to each other through a host bus 907 configured by an internal bus such as a CPU bus. It should be noted that the configuration in the information processing unit 71 described with reference to Figure 3 etc. is implemented by the CPU 901, for example.

[0234] The host bus 907 is connected to the external bus 911 (such as a Peripheral Component Interconnect / Interface (PCI) bus) via the bridge 909. The input device 915, the output device 917, the storage device 919, the drive 921, the connection port 923, and the communication device 925 are connected to the external bus 911 through the interface 913.

[0235] The input device 915 is an operating device for user operation (such as a mouse, a keyboard, a touch panel, a button, a switch, a lever, and a pedal). In addition, the input device 915 can be, for example, a remote control device (so-called remote controller) that uses infrared rays or other radio waves. For example, the input device 915 can be an externally connected device 929 corresponding to the operation of the information processing device 900, such as a mobile phone, a smartphone, or a tablet terminal.

[0236] The input device 915 includes, for example, an input control circuit that generates an input signal based on the information input by the user using the above operating device and outputs the input signal to the CPU 901.

[0237] By operating the input device 915, the user can input various types of data regarding the information processing device 900 and give instructions for operations to be processed.

[0238] The output device 917 includes devices capable of visually or auditorily notifying the user of the acquired information. Specifically, the output device 917 is configured as a display device such as a cathode ray tube (CRT) display device, a liquid crystal display device, a plasma display device, an electroluminescence (EL) display device, or a lamp; a sound output device such as a speaker and headphones; a printer device, etc.

[0239] The output device 917 outputs, for example, the results obtained from various types of processing executed by the information processing device 900. Specifically, the display device displays the results obtained from various types of processing executed by the information processing device 900 in the form of text or images. At the same time, the sound output device converts an audio signal including reproduced sound data or acoustic data into an analog signal for output.

[0240] The storage device 919 is a device for storing data, and the storage device 919 is configured as an example of the storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs to be executed by the CPU 901, various types of data, etc.

[0241] The drive 921 is a reader / writer for a recording medium, and the drive 921 is built in or externally provided to the information processing device 900. The drive 921 reads the information recorded on a removable recording medium 927 (such as a mounted disk, an optical disk, a magneto-optical disk, or a semiconductor memory) and outputs the information to the RAM 905. In addition, the drive 921 is also capable of writing records on the removable recording medium 927 (such as a mounted disk, an optical disk, a magneto-optical disk, or a semiconductor memory).

[0242] The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, or a Blu-ray (registered trademark) medium. In addition, the removable recording medium 927 can be a compact flash (CF) (registered trademark), a flash memory, a secure digital (SD) memory card, etc. Further, the removable recording medium 927 can be, for example, an integrated circuit (IC) card or an electronic device including a non-contact IC chip.

[0243] The connection port 923 is a port for directly connecting an external connection device 929 to the information processing device 900. Examples of the connection port 923 include: a Universal Serial Bus (USB) port, an IEEE 1394 port, and a Small Computer System Interface (SCSI) port. Other examples of the connection port 923 include: an RS-232C port, an optical audio terminal, and a High-Definition Multimedia Interface (registered trademark) (HDMI) port. When the external connection device 929 is connected to the connection port 923, the information processing device 900 directly obtains various types of data from the external connection device 929 or provides various types of data to the external connection device 929.

[0244] The communication device 925 is, for example, a communication interface including a communication device for connecting to a communication network 931. The communication device 925 is, for example, a communication card for a wired or wireless Local Area Network (LAN), Bluetooth (registered trademark), or Wireless USB (WUSB). In addition, the communication device 925 may be a router for optical communication, a router for Asymmetric Digital Subscriber Line (ADSL), or a modem for various communications.

[0245] For example, the communication device 925 can send signals to and receive signals from the Internet or other communication devices according to a predetermined protocol such as TCP / IP. In addition, the communication network 931 connected to the communication device 925 may be configured by a network connected in a wired manner or a wireless manner. The communication network 931 may be, for example, the Internet or a home LAN, or may be a communication network for infrared communication, radio communication, or satellite communication.

[0246] Figure 18 The constituent elements of the information processing device 900 can be constituted by using general-purpose components, or can be constituted by hardware specific to the functions of the constituent elements. Therefore, when implementing an embodiment of the present technology, the hardware configuration to be used can be appropriately changed according to the technical level.

[0247] In addition, a computer program for implementing the functions of the information processing device 900 constituting the surgical support system according to an embodiment of the present technology can be generated, and the computer program can be implemented in a personal computer or the like. In addition, a computer-readable recording medium can be provided, and such a computer program is stored in the medium. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. In addition, the computer program can be distributed, for example, via a network without using a recording medium.

[0248] It should be noted that the program to be executed by a computer can be a program that is processed in chronological order along the order described in this specification, or can be a program that is processed in parallel, or a program that is processed at a necessary timing (such as when a call is executed).

[0249] <Others>

[0250] In this specification, a system refers to a collection of multiple constituent elements (devices, apparatuses, modules (components), etc.), regardless of whether all the constituent elements are included in the same housing. Thus, multiple devices housed in separate housings and connected to each other via a network are a system, and a single device including multiple modules in one housing is also a system.

[0251] It should be noted that the effects disclosed in this specification are merely exemplary and not restrictive, and any other effects may be produced.

[0252] Embodiments of the present technology are not limited to the above embodiments, and various modifications can be made without departing from the gist of the present technology.

[0253] For example, the present technology may have a cloud computing configuration in which multiple devices share a function and cooperate via a network to perform processing. In addition, the present technology may have a configuration in which a server calculates via a network coupled to a medical imaging device. In addition, the present technology may have a configuration of a converter that converts a signal output from a medical imaging device into an IP (Internet Protocol) signal. In other words, part or all of the functions of the CCU can be placed on a server or a converter.

[0254] In addition, each step described in the above flowchart can be executed by one device, or shared and executed by multiple devices.

[0255] In addition, in a case where one step includes multiple processing steps, the multiple processing steps in one step can be executed by one device, or shared and executed by multiple devices.

[0256] <Combination Examples of Configurations>

[0257] The present technology may have the following configurations.

[0258] (1) A medical observation system, comprising:

[0259] An imaging unit that captures an image of a surgical area;

[0260] A light source unit that irradiates the surgical area with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band; and

[0261] A controller that generates three-dimensional information based on a special light image captured by the imaging unit during irradiation with the special light.

[0262] (2) The medical observation system according to (1), wherein

[0263] The controller:

[0264] Generate three-dimensional information that indicates the shape of the space including the surgical area, and

[0265] When a special light image is captured, estimate the position and orientation of the imaging unit based on the generated three-dimensional information.

[0266] (3) The medical observation system according to (1) or (2), further comprising:

[0267] A development processing unit that generates an observation image based on a signal provided from the imaging unit after the imaging unit captures an image of the surgical area during illumination with special light; and

[0268] A display controller that causes the observation image corresponding to the estimation result of the position and orientation of the imaging unit to be displayed.

[0269] (4) The medical observation system according to (1) or (2), wherein

[0270] The imaging unit includes:

[0271] An observation light imaging device that is an imaging device for observation light, and

[0272] A special light imaging device that is an imaging device for special light.

[0273] (5) The medical observation system according to (4), further comprising:

[0274] A development processing unit that generates an observation image based on a signal provided from the imaging unit after the observation light imaging device captures an image of the surgical area during illumination with observation light; and

[0275] A display controller that causes the observation image corresponding to the estimation result of the position and orientation of the imaging unit to be displayed.

[0276] (6) The medical observation system according to (5), wherein

[0277] The controller generates three-dimensional information based on the special light image captured by the special light imaging device during illumination with special light and the observation light image captured by the observation light imaging device during illumination with observation light.

[0278] (7) The medical observation system according to (6), wherein

[0279] The controller generates three-dimensional information that includes feature points in the special light image and feature points in the observation light image.

[0280] (8) The medical observation system according to (7), wherein

[0281] Controller:

[0282] Based on the eigenvalues of each feature point, calculate the reliability of the feature points in the special light image and the reliability of the feature points in the observation light image, and

[0283] Generate three-dimensional information by using a predetermined number of feature points with high reliability as the feature points of the three-dimensional information.

[0284] (9) The medical observation system according to any one of (1) to (8), wherein,

[0285] The wavelength band of the special light is longer than the predetermined wavelength band of the observation light.

[0286] (10) The medical observation system according to any one of (1) to (8), wherein,

[0287] The observation light includes light to be reflected on the surface of the surgical area, and

[0288] The special light includes light to be reflected on the deep structures of the surgical area.

[0289] (11) The medical observation system according to any one of (1) to (8), wherein,

[0290] The observation light includes light to be reflected on the surface of the surgical area, and

[0291] The special light includes light to be reflected on the blood vessels on the surface of the surgical area.

[0292] (12) The medical observation system according to any one of (1) to (8), wherein,

[0293] The special light includes light obtained through a polarization filter.

[0294] (13) The medical observation system according to any one of (1) to (8), wherein,

[0295] The special light includes light having a predetermined pattern, and the surgical area is irradiated with the light having the predetermined pattern.

[0296] (14) The medical observation system according to any one of (1) to (8), wherein,

[0297] The special light includes light obtained by pulse-modulating light of a predetermined wavelength.

[0298] (15) A medical observation method for a medical observation system, the method comprising:

[0299] Capturing an image of the surgical area by an imaging unit;

[0300] The light source unit irradiates the surgical area with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band; and

[0301] The controller generates three-dimensional information based on a special light image captured by the imaging unit during the irradiation with the special light.

[0302] (16) An information processing device, comprising:

[0303] A controller that generates three-dimensional information based on a special light image obtained by capturing an image of the surgical area during irradiation with special light from a light source unit that irradiates the surgical area with observation light in a predetermined wavelength band or special light in a wavelength band different from the predetermined wavelength band.

[0304] This technology may also have the following configuration.

[0305] (1) A medical imaging system, comprising:

[0306] A light source configured to irradiate the surgical area with observation light in a first wavelength band or special light in a second wavelength band different from the first wavelength band;

[0307] An image capturing device configured to generate a special light image based on the reflected special light, which is the special light reflected from at least a part of the surgical area and received by the image capturing device; and

[0308] A control processing circuit configured to generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the special light image.

[0309] (2) The medical imaging system according to (1), wherein:

[0310] The reflected special light includes the special light reflected from a part of the surgical area located below the outer surface of the surgical area;

[0311] The image capturing device is configured to generate a special light image based on the reflected special light, which is the special light reflected from a part of the surgical area located below the outer surface of the surgical area, and the special light image includes information about the part of the surgical area located below the outer surface of the surgical area; and

[0312] The control processing circuit is further configured to generate three-dimensional information based on the information about the part of the surgical area located below the outer surface of the surgical area.

[0313] (3) The medical imaging system according to (1), wherein:

[0314] The outer surface of the surgical area is arranged between the special light image capturing device and the part of the surgical area located below the outer surface of the surgical area along the optical axis of the reflected special light received by the image capturing device.

[0315] (4) The medical imaging system according to (1), wherein:

[0316] The image capture device is included within the endoscope; and

[0317] The control processing circuit is further configured to estimate the position and posture of a part of the endoscope relative to the surgical area based on the generated three-dimensional information.

[0318] (5) The medical imaging system according to (4), further comprising:

[0319] A stored image memory configured to store a first three-dimensional map of a part of the surgical area obtained before the start of the surgery; and

[0320] A display control circuit, wherein,

[0321] The control processing circuit is further configured to generate a second three-dimensional map of the part of the surgical area based on the first three-dimensional map and the estimated position and posture of the part of the endoscope, and

[0322] The display control circuit is configured to control the display of the two-dimensional image based on the second three-dimensional map.

[0323] (6) The medical imaging system according to (1), wherein:

[0324] The light of the first band is visible light; and

[0325] The light of the second band is infrared light.

[0326] (7) The medical imaging system according to (1), wherein:

[0327] The three-dimensional information about the surgical area includes information about the three-dimensional shape of the space including the surgical area within the patient's body.

[0328] (8) The medical imaging system according to (6), further comprising:

[0329] The image capture device generates a visible light image based on the reflected visible light, which is the visible light reflected from at least a part of the surgical area and received by the visible light image capture device; and

[0330] The control processing circuit is further configured to perform alignment processing to adjust the special light image so that the adjusted special light image is aligned with the visible light image.

[0331] (9) The medical imaging system according to (8), wherein:

[0332] The control processing circuit is further configured to determine the reliability of the feature points in each of the special light image and the visible light image; and

[0333] The control processing circuit is further configured to generate three-dimensional information based on the reliability of the feature points in each of the determined special light image and visible light image.

[0334] (10) The medical imaging system according to (9), wherein:

[0335] The control processing circuit is further configured to generate three-dimensional information by retaining only the feature points with relatively high reliability.

[0336] (11) The medical imaging system according to (10), wherein:

[0337] The control processing circuit is further configured to generate a three-dimensional map based on the three-dimensional information.

[0338] (12) The medical imaging system according to (1), wherein the control processing circuit is configured to:

[0339] generate three-dimensional information indicating the shape of the space including the surgical area, and

[0340] estimate the position and pose of the imaging capture device at the time of capturing the special light image based on the generated three-dimensional information.

[0341] (13) The medical imaging system according to (1), wherein the control processing circuit is further configured to:

[0342] after the image capture device captures a special light image of the surgical area during irradiation with special light, generate an observation image based on a signal provided by the image capture device; and

[0343] cause the observation image corresponding to the estimation result of the position and pose of the image capture device to be displayed.

[0344] (14) The medical imaging system according to (1), wherein the control processing circuit is further configured to:

[0345] after the image capture device captures an image of the surgical area during irradiation with observation light, generate an observation image based on a signal provided by the image capture device; and

[0346] cause the observation image corresponding to the estimation result of the position and pose of the image capture device to be displayed.

[0347] (15) The medical imaging system according to (14), wherein the control processing circuit is further configured to:

[0348] generate three-dimensional information based on the special light image captured by the special light imaging device during irradiation with special light and the observation light image captured by the observation light imaging device during irradiation with observation light.

[0349] (16) The medical imaging system according to (15), wherein the control processing circuit is further configured to:

[0350] Generate three-dimensional information including feature points in the special light image and feature points in the observation light image.

[0351] (17) The medical imaging system according to (16), wherein the control processing circuit is further configured to:

[0352] Calculate the reliability of the feature points in the special light image and the reliability of the feature points in the observation light image based on the feature values of the respective feature points, and

[0353] Generate three-dimensional information by using a predetermined number of feature points with high reliability as the feature points of the three-dimensional information.

[0354] (18) The medical imaging system according to (1), wherein the control processing circuit includes an IP converter coupled to the image capture device.

[0355] (19) A medical imaging processing method, comprising:

[0356] Obtain first image data captured when irradiating light of a first wavelength band and second image data captured when irradiating light of a second wavelength band, wherein the first wavelength band is different from the second wavelength band, and

[0357] Generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the second image data.

[0358] (20) A medical information processing device, comprising:

[0359] A control processing circuit configured to:

[0360] Obtain first image data captured when irradiating light of a first wavelength band and second image data captured when irradiating light of a second wavelength band, wherein the first wavelength band is different from the second wavelength band, and

[0361] Generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the second image data.

[0362] Those skilled in the art should understand that various modifications, combinations, sub-combinations, and changes can be made according to design requirements and other factors, as long as they are within the scope of the appended claims or their equivalents.

[0363] List of reference numerals

[0364] 1 Medical observation system (medical imaging system)

[0365] 11 Endoscope (medical imaging device)

[0366] 13 CCU (Medical treatment device)

[0367] 15 Display device

[0368] 17 Light source device

[0369] 51 Imaging unit

[0370] 61 Special light irradiation unit

[0371] 62 Image sensor for special light

[0372] 63 Visible light irradiation unit

[0373] 64 Image sensor for visible light

[0374] 71 Information processing unit

[0375] 81 Special light development processing unit

[0376] 82 Development processing unit for special light observation

[0377] 83 3D map generation unit

[0378] 84 3D map storage unit

[0379] 85 On-screen position calculation unit

[0380] 86 3D information storage unit

[0381] 87 Display controller

[0382] 91 Visible light development processing unit

[0383] 92 Alignment processing unit.

Claims

1. A medical imaging system, comprising: a light source configured to irradiate a surgical area with observation light in a first band or special light in a second band different from the first band, wherein the special light is pulsed modulated light or polarized light generated by using a polarization light filter; an image capture device configured to generate a special light image based on the reflected special light, the reflected special light being special light reflected from at least a part of the surgical area and received by the image capture device; and a control processing circuit configured to generate three-dimensional information including three-dimensional coordinate information about the surgical area based on the special light image; wherein the light in the first band is visible light and the light in the second band is infrared light, the image capture device generates a visible light image based on the reflected visible light, the reflected visible light being visible light reflected from at least a part of the surgical area and received by the visible image capture device, the control processing circuit is further configured to: determine the reliability of feature points in each of the special light image and the visible light image, and generate the three-dimensional information based on the determined reliability of the feature points in each of the special light image and the visible light image, the reliability of the feature points is based on the eigenvalue difference between the feature points detected from a certain frame and the corresponding feature points detected from the previous frame, and the smaller the eigenvalue difference, the higher the reliability, and the eigenvalue difference is the distance between the eigenvalue of the feature point detected from the certain frame and the eigenvalue of the corresponding feature point detected from the previous frame.

2. The medical imaging system according to claim 1, wherein: the reflected special light includes special light reflected from a part of the surgical area located below the outer surface of the surgical area; the image capture device is configured to generate the special light image based on the reflected special light that is special light reflected from a part of the surgical area located below the outer surface of the surgical area, the special light image including information about a part of the surgical area located below the outer surface of the surgical area; and the control processing circuit is further configured to generate the three-dimensional information based on the information about a part of the surgical area located below the outer surface of the surgical area.

3. The medical imaging system according to claim 1, wherein: the outer surface of the surgical area is arranged between the special light image capture device and a part of the surgical area located below the outer surface of the surgical area along the optical axis of the reflected special light received by the image capture device.

4. The medical imaging system according to claim 1, wherein: the image capture device is included inside an endoscope; and the control processing circuit is further configured to estimate the position and posture of a part of the endoscope relative to the surgical area based on the generated three-dimensional information.

5. The medical imaging system according to claim 4, further comprising: a stored image memory configured to store a first three-dimensional map of a part of the surgical area obtained before the start of a surgical operation; and a display control circuit, wherein the control processing circuit is further configured to generate a second three-dimensional map of a part of the surgical area based on the first three-dimensional map and the estimated position and pose of the part of the endoscope, and the display control circuit is configured to control the display of the two-dimensional image based on the second three-dimensional map.

6. The medical imaging system according to claim 1, wherein: the three-dimensional information about the surgical area includes information about the three-dimensional shape of the space containing the surgical area in the patient's body.

7. The medical imaging system according to claim 1, wherein: the control processing circuit is further configured to perform alignment processing to adjust the special light image so that the adjusted special light image is aligned with the visible light image.

8. The medical imaging system according to claim 1, wherein: the control processing circuit is further configured to generate the three-dimensional information by only retaining feature points with relatively high reliability.

9. The medical imaging system according to claim 8, wherein: the control processing circuit is further configured to generate a three-dimensional map based on the three-dimensional information.

10. The medical imaging system according to claim 1, wherein, The control processing circuit is configured to: generate the three-dimensional information indicating the shape of the space including the surgical area, and estimate the position and pose of the image capture device when the special light image is captured based on the generated three-dimensional information.

11. The medical imaging system according to claim 1, wherein, The control processing circuit is further configured to: after the image capture device captures the special light image of the surgical area during irradiation with the special light, generate an observation image based on a signal provided by the image capture device; and cause the observation image to be displayed, the observation image corresponding to the estimation result of the position and pose of the image capture device.

12. The medical imaging system according to claim 1, wherein, The control processing circuit is further configured to: after the image capture device captures an image of the surgical area during irradiation with observation light, generate an observation image based on a signal provided by the image capture device; and cause the observation image to be displayed, the observation image corresponding to the estimation result of the position and pose of the image capture device.

13. The medical imaging system according to claim 12, wherein, The control processing circuit is further configured to: generate the three-dimensional information based on the special light image captured by the special light imaging device during irradiation with the special light and the observation light image captured by the observation light imaging device during irradiation with the observation light.

14. The medical imaging system according to claim 13, wherein, The control processing circuit is further configured to: generate the three-dimensional information including the feature points in the special light image and the feature points in the observation light image.

15. The medical imaging system according to claim 14, wherein, The control processing circuit is further configured to: calculate the reliability of the feature points in the special light image and the reliability of the feature points in the observation light image based on the feature values of the respective feature points, and generate the three-dimensional information by using a predetermined number of feature points with high reliability as the feature points of the three-dimensional information.

16. The medical imaging system according to claim 1, wherein, The control processing circuit includes an IP converter coupled to the image capture device.

17. A medical imaging processing method, comprising: Obtain first image data captured when irradiating light of a first band and second image data captured when irradiating light of a second band, where the first band is different from the second band, the light of the second band is pulsed modulated light or polarized light generated by using a polarization filter, and Generate three-dimensional information including three-dimensional coordinate information about a surgical area based on the second image data, where the light of the first band is visible light and the light of the second band is infrared light, The method further includes: Obtain a visible light image generated based on reflected visible light, where the reflected visible light is visible light reflected from at least a part of the surgical area and received by a visible image capturing device, Determine the reliability of feature points in each of the second image data and the visible light image, and generate the three-dimensional information based on the determined reliability of the feature points in each of the second image data and the visible light image, The reliability of the feature points is based on the eigenvalue difference between the feature points detected from a certain frame and the corresponding feature points detected from the previous frame, and the smaller the eigenvalue difference, the higher the reliability, and The eigenvalue difference is the distance between the eigenvalue of the feature point detected from the certain frame and the eigenvalue of the corresponding feature point detected from the previous frame.

18. A medical information processing device, comprising: A control processing circuit configured to: Obtain first image data captured when irradiating light of a first band and second image data captured when irradiating light of a second band, where the first band is different from the second band, the light of the second band is pulsed modulated light or polarized light generated by using a polarization filter, and Generate three-dimensional information including three-dimensional coordinate information about a surgical area based on the second image data; where the light of the first band is visible light and the light of the second band is infrared light, The control processing circuit is further configured to: Obtain a visible light image generated based on reflected visible light, where the reflected visible light is visible light reflected from at least a part of the surgical area and received by a visible image capturing device, Determine the reliability of feature points in each of the second image data and the visible light image, and generate the three-dimensional information based on the determined reliability of the feature points in each of the second image data and the visible light image, The reliability of the feature points is based on the eigenvalue difference between the feature points detected from a certain frame and the corresponding feature points detected from the previous frame, and the smaller the eigenvalue difference, the higher the reliability, and The eigenvalue difference is the distance between the eigenvalue of the feature point detected from the certain frame and the eigenvalue of the corresponding feature point detected from the previous frame.

Images