Medical observation system, method and medical observation device

Abstract

This technology relates to a medical observation system, method, and medical observation device capable of maintaining the accuracy of three-dimensional information even when changes occur in the optical system. The medical observation system acquires surgical field data acquired by the medical observation device, detects changes in the optical system of the medical observation device, and upon detecting changes in the optical system, estimates parameters representing the state of the optical system after the changes occur. Using the estimation results, the system sets conditions for generating three-dimensional information based on the surgical field data. This technology can be applied to surgical support systems.

Description

Technical Field

[0001] The present technology relates to a medical observation system, a method, and a medical observation apparatus, and more particularly, to a medical observation system, a method, and a medical observation apparatus capable of maintaining the accuracy of three-dimensional information even when an optical system changes. Background Art

[0002] In surgical operations using a medical observation apparatus such as an endoscope or a microscope, it has been proposed to generate three-dimensional information based on an image of a surgical field, and use the three-dimensional information for image processing or display processing of the image of the surgical field.

[0003] For example, Patent Document 1 proposes a technology that generates three-dimensional information by SLAM and displays it on a screen.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-225700 Summary of the Invention

[0007] Problems to be solved by the present invention

[0008] Incidentally, the optical system of a medical observation device may change during surgery. For example, when focusing the medical observation device, the position of the focus lens in the optical system shifts. In particular, in endoscopic surgery, the endoscope's mirror may be replaced during surgery, resulting in changes to the optical system.

[0009] When the optical system of a medical observation device changes, the assumed parameters are different. Therefore, the accuracy of the 3D information generated before the change differs from that generated after the change. However, during surgery, it is difficult to reconstruct 3D information from the outset.

[0010] The present technology has been proposed in view of such circumstances, and its purpose is to maintain the accuracy of three-dimensional information even when the optical system changes.

[0011] Solution to the problem

[0012] According to one aspect of the present technology, a medical observation system is provided, comprising: an acquisition unit configured to acquire surgical field data acquired by a medical observation device; a detection unit configured to detect a change in an optical system of the medical observation device; an estimation unit configured to estimate parameters determined based on the optical system after the change, when the detection unit detects the change in the optical system; and a setting unit configured to set generation conditions for three-dimensional information based on the surgical field data, by using an estimation result of the estimation unit.

[0013] According to another aspect of the present technology, a medical observation device is provided, including: an imaging unit configured to image a surgical field and generate surgical field data; and an output unit configured to output the surgical field data, the medical observation device being used in a medical observation system, the medical observation system detecting changes in an optical system of the imaging unit, and when the changes in the optical system are detected, estimating parameters determined based on the optical system after the changes, and setting generation conditions for three-dimensional information based on the surgical field data by using the estimation results.

[0014] In the present technology, surgical field data acquired by a medical observation device is acquired, changes in the optical system of the medical observation device are detected, and when changes in the optical system are detected, parameters determined according to the optical system after the changes are estimated, and generation conditions for three-dimensional information based on the surgical field data are set by using the estimation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram showing a configuration example of a surgery support system according to a first embodiment of the present technology;

[0016] Figure 2 is a block diagram showing a functional configuration example of a surgery support system according to a first embodiment of the present technology;

[0017] Figure 3 is a diagram showing an example of a mask area;

[0018] Figure 4 It shows Figure 2 Flowchart of the three-dimensional information generation process of the surgical assistance system;

[0019] Figure 5 is a diagram showing another configuration example of the surgery support system;

[0020] Figure 6 is a diagram illustrating a configuration example of an information processing device configuring a surgery support system according to a second embodiment of the present technology;

[0021] Figure 7is a block diagram showing a functional configuration example of a surgery support system according to a second embodiment of the present technology;

[0022] Figure 8 It shows Figure 7 Flowchart of the three-dimensional information generation process of the surgical assistance system;

[0023] Figure 9 is a diagram illustrating another configuration example of the surgery support system according to an embodiment of the present technology;

[0024] Figure 10 is a block diagram showing a hardware configuration example of an information processing apparatus. DETAILED DESCRIPTION

[0025] Hereinafter, a mode for implementing the present technology will be described. The description will be made in the following order.

[0026] 1. First embodiment (for use during surgery)

[0027] 2. Second Implementation (for Training)

[0028] 3. Application Examples

[0029] 4. Hardware Configuration

[0030] 5. Others

[0031] 1. First embodiment (for use during surgery)

[0032] <Configuration Example of Surgery Support System (Example of Endoscope Operator Holding Endoscope)>

[0033] Figure 1 : is a diagram showing a configuration example of a surgery support system according to a first embodiment of the present technology.

[0034] For example, Figure 1 An example of an endoscopic surgery system used in an endoscopic surgery of the abdomen is shown, and the endoscopic surgery is performed in a medical field instead of a related-art laparotomy.

[0035] exist Figure 1 In the surgical support system 1, during endoscopic abdominal surgery, rather than incising the abdominal wall to open the abdomen as in the related art, the abdominal wall is pierced at several locations using opening instruments called trocars 25a and 25b. Then, a laparoscope (hereinafter also referred to as an endoscope) 11, a medical observation device for observing the inside of the patient's body, an energy therapy tool 22, forceps 23, and the like are inserted into the body through holes provided in the trocars 25a and 25b.

[0036] While observing the image of the affected part (tumor, etc.) U in the patient's body captured in real time by the endoscope 11, the operator performs treatment, for example, resection of the affected part U with the energy treatment tool 22, etc. The endoscope 11, the energy treatment tool 22, and the forceps 23 are held by an operator, a robot, etc. Note that the operator is referred to as a medical staff member who participates in the operation performed in the operating room, and the operator includes, for example, a doctor who monitors the operation from a place other than the operating room, in addition to the operating surgeon, assistants, endoscope operators, and nurses. Figure 1 In the example shown, the endoscope 11 is held by, for example, an endoscope operator. The endoscope 11 includes an endoscope that is inserted into a patient's body and a camera including an imaging element that receives light guided by the endoscope and forms an image of the light. Note that the endoscope can be rigid or flexible. Furthermore, the endoscope and the imaging element can be integrated.

[0037] In an operating room where such endoscopic surgery is performed, a cart 31 on which devices for endoscopic surgery are mounted, a bed 33 on which a patient lies, a foot switch 35, etc. are installed. For example, devices such as a camera control unit (CCU) 13, a light source device 17, a treatment tool device 21, an air insufflation device 24, a display device 15, a recorder 26, and a printer 27 are placed on the cart 31 as medical devices.

[0038] The image signal of the affected part U, imaged by the observation optical system of the endoscope 11, is transmitted to the CCU 13 via a camera cable serving as a signal transmission cable. In addition to being connected to the endoscope 11 via the camera cable, the CCU 13 can also be connected to the endoscope 11 via a wireless communication path. The CCU 13 processes the image signal output from the endoscope 11 and outputs the processed image signal to the display device 15. In this configuration, an image of the surgical field of the affected part U is displayed on the display device 15.

[0039] Note that the CCU 13 can output the image signal obtained after signal processing to the recorder 26, causing the recorder 26 to record the surgical field image of the affected part U as image data (e.g., moving image data). In addition, the CCU 13 can output the image signal obtained after signal processing to the printer 27, causing the printer 27 to print the surgical field image of the affected part U.

[0040] The light source device 17 is connected to the endoscope 11 via a light guide cable, and can switch light of various wavelengths to radiate the light to the affected part U. The light radiated from the light source device 17 can be used as, for example, auxiliary light.

[0041] The treatment tool device 21 corresponds to, for example, a high-frequency output device that outputs a high-frequency current to the energy treatment tool 22 that cuts off the affected part U by using electrical heating.

[0042] The insufflation device 24 includes an air supply device and an air suction device, and supplies air to, for example, the abdominal area in the patient's body.

[0043] The foot switch 35 controls the CCU 13 , the treatment tool device 21 , and the like by using a foot operation of an operator, an assistant, and the like as a trigger signal.

[0044] <Functional Configuration Example of Surgery Support System>

[0045] (Configuration around CCU 13)

[0046] Figure 2 is a block diagram showing a functional configuration example of a surgery support system.

[0047] Figure 2 The surgery assisting system 100 in the embodiment includes an imaging unit 101 , an information processing unit 102 and a display unit 103 .

[0048] The imaging unit 101 corresponds to Figure 1 The endoscope 11 in the image processing unit 101. The imaging unit 101 images the surgical field according to the operation of the endoscope operator and outputs the image signal obtained by imaging to the information processing unit 102. The imaging unit 101 is a medical observation device that outputs the surgical field data obtained by imaging the surgical field. As a medical observation device, a microscope can be used instead of the endoscope. Note that a circuit for performing imaging processing and processing the generated image signal (for example, a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), or FPGA (Field Programmable Gate Array)) is stacked in the medical observation device.

[0049] The information processing unit 102 corresponds to Figure 1 The information processing unit 102 acquires the image signal supplied from the imaging unit 101, performs signal processing on the image signal, and outputs a signal of a surgical field image generated by performing the signal processing to the display unit 103. Note that the information processing unit 102 may be configured by a device different from the CCU 13.

[0050] The display unit 103 corresponds to Figure 1 The display device 15 in the display unit 103 displays the surgical field image based on the image signal supplied from the information processing unit 102.

[0051] (Detailed Configuration of Information Processing Unit 102)

[0052] The information processing unit 102 includes an optical system change detection unit 111 , a parameter estimation unit 112 , a three-dimensional information generation unit 113 , and a display information generation unit 114 .

[0053] At least a portion of the information processing unit 102 includes Figure 1 The information processing unit 102 is implemented by a circuit such as the CPU of the CCU 13 in the image processing unit 101, which executes a predetermined program. The image signal output from the imaging unit 101 is input to the optical system change detection unit 111, the three-dimensional information generation unit 113, and the display information generation unit 114. Note that at least part of the functions of the information processing unit 102 can be implemented by an FPGA.

[0054] Optical system change detection unit 111

[0055] The optical system change detection unit 111 detects a change in the optical system that occurs during surgery in the imaging unit 101. For example, when the imaging unit 101 performs adjustment of the optical system (e.g., adjustment of zoom (angle of view) (movement of the zoom lens) or adjustment of focus (movement of the focus lens)), or when an endoscope is replaced while it is included in the imaging unit 101, the optical system changes.

[0056] A case where information of the optical system can be obtained electronically from the imaging unit 101

[0057] For example, since some optical members of the optical system included in the imaging unit 101 move in the adjustment of zoom and the adjustment of focus based on the output of the CCU 13, information indicating a change in the optical system (for example, information indicating the position of the zoom lens and the position of the focus lens) is stored in the CCU 13. In this case, the optical system change detection unit 111 detects the change in the optical system based on the information indicating the change in the optical system stored in the CCU 13.

[0058] Furthermore, there are cases where the imaging unit 101 includes a detachable mirror, and a storage unit storing information indicating the type of mirror is provided in the mirror. In this case, the circuit included in the imaging unit 101 may acquire the mirror information and output the mirror information to the CCU 13. In this case, the optical system change detection unit 111 detects a change in the optical system based on the information acquired from the imaging unit 101.

[0059] Case where information of the optical system cannot be obtained electronically from the imaging unit 101

[0060] In this case, the optical system change detection unit 111 detects a change in the optical system based on the image signal obtained from the imaging unit 101 .

[0061] For example, the optical system change detection unit 111 repeatedly detects the occlusion area in the image signals sequentially supplied from the imaging unit 101. Figure 3As shown, in the surgical field image generated from the image signal, the masked area is a vignetting area formed around the effective area where the surgical field appears. When the mirror or the like is replaced, the vignetting changes, thereby changing the masked area and the diameter of the circular effective area.

[0062] The optical system change detection unit 111 detects a change in the optical system occurring in the imaging unit 101 by detecting such a change in the mask area.

[0063] Furthermore, the optical system change detection unit 111 detects changes in the optical system occurring in the imaging unit 101 by using the singular values ​​of the camera matrix (the fundamental matrix of the frame). A change in focal length can be detected based on the singular values ​​of the camera matrix. Therefore, changes in the optical system, such as movement of the focus lens or movement of the zoom lens, can be detected.

[0064] Here, the detection method using the singular values ​​of the camera matrix is ​​a method for detecting changes in focal length using the ratio of the singular values ​​of the fundamental matrix, taking advantage of the fact that the non-zero singular values ​​of the fundamental matrix calculated at two viewpoints are the same when the focal length is the same. This method is described, for example, in "Stability of 3D Restoration of Input Image Groups with Unknown Focal Lengths" by Kazuki Nozawa, CVIM-182, Vol. 2012, No. 19.

[0065] Specifically, the following processes (a) to (d) are performed in the optical system change detecting unit 111 .

[0066] (a) The optical system change detection unit 111 records a key frame, which is used as a reference for generating three-dimensional information in SLAM.

[0067] (b) The optical system change detecting unit 111 sequentially calculates the basic matrix E by using key frames.

[0068] (c) The optical system change detection unit 111 calculates non-zero singular values ​​of the basic matrix E.

[0069] Here, the basic matrix E is a 3×3 matrix. When the singular value decomposition (following equation (1)) is performed on E, the singular value of the third row of the diagonal matrix Σ is zero, and the diagonal matrix Σ in frame i is i As shown in the following equation (2).

[0070] [Mathematical formula 1]

[0071] E=U∑V T …(1)

[0072] [Mathematical formula 2]

[0073]

[0074] Here, σ i2 <σ i1 .

[0075] (d) The optical system change detection unit 111 detects a change in focal length by comparing the ratio of non-zero singular values ​​at each time.

[0076] That is, in the case where calculations are performed based on images captured at the same focal length, the singular values ​​in the first row are equal to the singular values ​​in the second row. From this property, in σ i2 / σ i1 When it is less than 1, it means that the focal length has changed. Therefore, as shown in the following equation (3), by comparing the ratio of the singular value of the diagonal matrix Σ in frame i to the threshold th, the change of the optical system can be detected.

[0077] [Mathematical formula 3]

[0078]

[0079] As described above, the optical system change detection unit 111 outputs the detection result obtained by detecting the change in the optical system to the parameter estimation unit 112. The detection result output to the parameter estimation unit 112 also includes information about the optical system of the imaging unit 101. Note that the method of detecting the change in the optical system is not limited to the above method, and other methods may be employed.

[0080] Parameter estimation unit 112

[0081] Figure 2 The parameter estimation unit 112 in the image estimates parameters used as generation conditions for three-dimensional information based on the surgical field image. These parameters are parameters determined according to the optical system and are information indicating, for example, the focal length, image center, magnification, and lens distortion coefficient. The information configuring the parameters only needs to include at least one parameter determined according to the optical system and only needs to include, for example, at least one of the focal length, image center, magnification, or distortion coefficient. Note that the parameters determined according to the optical system include parameters determined according to the settings of the optical system in the imaging unit 101. For example, even for the same endoscope, the image center can be slightly changed by removing the endoscope.

[0082] A case where information of the optical system can be obtained electronically from the imaging unit 101

[0083] In this case, the parameter estimation unit 112 refers to a table indicating the relationship between the information of the optical system and the parameters, and obtains parameters corresponding to the information of the optical system obtained from the imaging unit 101. A table generated in advance and indicating the relationship between the information of the optical system and the parameters is supplied to the parameter estimation unit 112.

[0084] Case where information of the optical system cannot be obtained electronically from the imaging unit 101

[0085] In this case, the parameter estimation unit 112 estimates a parameter matrix based on the image signal obtained from the imaging unit 101 as a parameter.

[0086] For example, an estimation method employing self-calibration is used, which is capable of estimating a parameter matrix without using a calibration pattern. Self-calibration is described, for example, in O.D. Faugeras, "Camera Self-Calibration: Theory and Experiments," European Conference on Computer Vision, 1992, pp. 321-334. Parameter estimation unit 112 calculates information serving as a reliability indicator for the estimated parameter matrix.

[0087] Parameter estimation unit 112 determines whether to set new conditions for generating three-dimensional information using the estimated parameter, that is, whether to update the parameters used as the conditions for generating three-dimensional information using the estimated parameter. If it is determined that the parameters used as the conditions for generating three-dimensional information are to be updated, the parameters used as the conditions for generating three-dimensional information are updated.

[0088] As a method of determining whether to update the parameters, there are an automatic determination method in which the parameter estimation unit 112 performs determination by itself and a manual determination method in which the user causes determination to be performed.

[0089] In the case of using the automatic determination method, the parameter estimation unit 112 determines whether to update the parameters based on a threshold value determination using a reliability index of the parameter matrix obtained after estimation. For example, if the reliability index of the parameter matrix obtained after estimation is higher than a preset threshold, it is determined that the parameters are updated, and if the reliability index is lower than the threshold, it is determined that the parameters are not updated.

[0090] In the case of using the manual determination method, the parameter estimation unit 112 presents the estimation result on the display unit 103 and determines whether to update the parameter according to the selection of the user who has seen the estimation result.

[0091] Three-dimensional information generating unit 113

[0092] The three-dimensional information generation unit 113 generates three-dimensional information based on each frame of the surgical field image represented by the image signal provided by the imaging unit 101, using parameters serving as the generation conditions for the three-dimensional information. The three-dimensional information is generated based on the surgical field image using the aforementioned parameters. The three-dimensional information includes a three-dimensional map representing the three-dimensional structure of the object (in an organ or body cavity) appearing in the surgical field image, as well as position and orientation information representing the imaging unit 101's own position and orientation.

[0093] Algorithms used for generating three-dimensional information include Visual SLAM, which only takes in surgical field images, and RGB-D-SLAM, which measures depth information using a ToF sensor, lidar, or the like and takes in both the surgical field image and depth information.

[0094] When the optical system change detection unit 111 detects a change in the optical system, the three-dimensional information generation unit 113 stops generating three-dimensional information until the parameter estimation unit 112 estimates new parameters. When the new parameters are estimated, the three-dimensional information generation unit 113 resumes generating three-dimensional information using the new parameters.

[0095] Furthermore, when the optical system change detection unit 111 detects a change in the optical system, the three-dimensional information generation unit 113 stores the three-dimensional information by distinguishing between the three-dimensional information before the optical system change and the three-dimensional information after the optical system change, without stopping the generation of the three-dimensional information. When imaging the same location as that imaged before the optical system change after estimating new parameters, the three-dimensional information generation unit 113 updates the three-dimensional information by replacing the three-dimensional information of the location (the three-dimensional information before the optical system change) with the three-dimensional information generated using the new parameters (the three-dimensional information after the optical system change).

[0096] The three-dimensional information generating unit 113 outputs the three-dimensional information generated in this manner to the display information generating unit 114 .

[0097] Display information generating unit 114

[0098] The display information generation unit 114 causes the display unit 103 to display a surgical field image based on the image signal supplied from the imaging unit 101 .

[0099] Furthermore, the display information generating unit 114 causes the display unit 103 to display a three-dimensional map based on the three-dimensional information supplied from the three-dimensional information generating unit 113. The three-dimensional map can be displayed by a display method that changes color or the like before and after updating parameters.

[0100] Furthermore, the display information generating unit 114 displays the detection result of the change of the optical system in the optical system change detecting unit 111. At this time, information indicating that the mirror in the imaging unit 101 has been replaced may be displayed, or information after replacement, for example, the type of the mirror may be displayed.

[0101] Furthermore, the display information generating unit 114 may cause the display unit 103 to display the new parameter set as the generation condition of the three-dimensional information.

[0102] <Operation Example of Surgery Support System>

[0103] Figure 4 3D information generation processing in the surgery support system 100 is shown in FIG.

[0104] In step S111 , the three-dimensional information generating unit 113 generates three-dimensional information by using parameters based on the surgical field image represented by the image signal obtained from the imaging unit 101 .

[0105] In step S112 , the three-dimensional information generating unit 113 updates the three-dimensional information generated so far by using the newly generated three-dimensional information.

[0106] In step S113 , the display information generating unit 114 causes the display unit 103 to display a three-dimensional map based on the three-dimensional information supplied from the three-dimensional information generating unit 113 .

[0107] In step S114 , the optical system change detection unit 111 determines whether a change in the optical system is detected.

[0108] In a case where it is determined in step S114 that a change in the optical system is detected, the parameter estimation unit 112 estimates the parameters in step S115. The generation of three-dimensional information stops until the parameters are updated.

[0109] In step S116, the parameter estimation unit 112 determines whether to update the parameters used as the generation conditions of the three-dimensional information by using the estimated parameters. The determination here is performed based on the reliability index of the parameter estimation result as described above.

[0110] In the event that it is determined in step S116 that the parameters have not been updated, the process returns to step S115 and the parameter estimation is repeated.

[0111] On the other hand, if the updated parameters are determined in step S116, the parameter estimation unit 112 updates the parameters used as the generation conditions of the three-dimensional information according to the new parameters in step S117. The parameters updated by the parameter estimation unit 112 are provided to the three-dimensional information generation unit 113.

[0112] In the three-dimensional information generating unit 113 , for example, until a change in the optical system is detected, generation of three-dimensional information is continued by adjusting the scale of the three-dimensional map using new parameters so as to be compatible with the three-dimensional map.

[0113] In step S118, the three-dimensional information generating unit 113 determines whether to end the three-dimensional information generating process. If it is determined in step S118 that the three-dimensional information generating process has not ended, or if it is determined in step S114 that no change in the optical system has been detected, the process returns to step S111 and the processes in step S111 and subsequent steps are repeated.

[0114] On the other hand, in a case where it is determined in step S118 that the three-dimensional information generation process is ended, the process of the surgery support system 100 is ended.

[0115] In the above-described processing, when a change occurs in the optical system of the imaging unit 101 , the parameters serving as the generation conditions of the three-dimensional information are updated, and the generation of the three-dimensional information is continued by using the updated parameters.

[0116] In order to generate accurate 3D information in Visual-SLAM, parameters including focal length, image center, and distortion coefficient need to be set to appropriate values. In general Visual-SLAM, the parameters are obtained through camera calibration, and during surgery (during the operation), the pre-acquired parameters are used as fixed values ​​to generate 3D information.

[0117] On the other hand, during surgery, optical system adjustments, such as zooming or replacing the endoscope itself, can be made, thus changing parameters. The scale and other factors of the generated 3D information can change between before and after the optical system changes, or errors can occur. To utilize the altered 3D information, simply readjust the parameters. However, since manual camera calibration requires removing the endoscope, performing this during surgery is impractical.

[0118] As described above, by continuing to generate three-dimensional information using updated parameters, even when the optical system of the imaging unit 101 changes during surgery, the accuracy of the three-dimensional information can be maintained without recalibrating the parameters or the like.

[0119] <Another Configuration Example of a Surgery Support System (Example of a Robotic Arm Holding an Endoscope)>

[0120] Figure 5 is a diagram illustrating another configuration example of the surgery support system.

[0121] exist Figure 5 In the configuration shown, the corresponding Figure 1The parts of the configuration in are denoted by the same reference numerals. Overlapping descriptions will be omitted as appropriate. Figure 5 The configuration of the surgical assistance system 200 shown in FIG. Figure 1 The configuration shown in FIG. 1 is different in that a robot arm device 212 including a robot arm 211 and a cart 213 is provided, on which various devices for endoscopic surgery are mounted.

[0122] The robot device 212 holds the endoscope 11 with the robot arm 211. The position and direction information of the endoscope 11 acquired by the robot device 212 is provided to the CCU 13 ( Figure 2 The information processing unit 102 in FIG.

[0123] In a case where the endoscope 11 is held by the robot arm 211 , position and direction information of the endoscope 11 supplied from the robot arm device 212 is used to detect changes in the optical system and estimate parameters.

[0124] <Functional Configuration Example of Surgery Support System>

[0125] Figure 5 Functional configuration and reference of surgical assistance system 200 in Figure 2 The configuration described is the same. Figure 2 , a method of detecting a change in the optical system, a method of using position and direction information of the imaging unit 101 (endoscope 11 ), and a method of estimating parameters will be described with respect to the surgery support system 200 .

[0126] The position and direction information of the imaging unit 101 supplied from the robot device 212 is input to the optical system variation detecting unit 111 and the three-dimensional information generating unit 113 .

[0127] Optical system change detection unit 111

[0128] The optical system change detection unit 111 detects a change in the optical system based on the trajectory of the self position of the imaging unit 101 supplied from the robot device 212 .

[0129] According to SLAM, when the optical system changes in a manner similar to a three-dimensional map, the self-position of the imaging unit 101 estimated by the three-dimensional information generating unit 113 results in an error. The optical system change detecting unit 111 compares the actual trajectory of the self-position of the imaging unit 101 obtained from the robot device 212 with the trajectory of the self-position estimated by the three-dimensional information generating unit 113, and detects that the optical system has changed if the error in the trajectory is large.

[0130] Furthermore, in general, when the viewing angle changes, it is difficult to distinguish between zooming in and out and movement of the imaging unit 101 along the optical axis. However, when the imaging unit 101 is held by the robotic arm 211, the presence or absence of movement of the imaging unit 101 can be detected. Therefore, changes in the optical system can be detected by using the viewing angle. That is, even when the viewing angle of the surgical field image changes, changes in the optical system can be detected even when the imaging unit 101 does not move.

[0131] Note that even in the case where the imaging unit 101 is held by an endoscope, when the presence or absence of movement of the imaging unit 101 can be detected by a sensor, etc., changes in the optical system can be detected by using changes in the viewing angle and the presence or absence of movement of the imaging unit 101 in a manner similar to the case of the robotic arm 211.

[0132] As a method of detecting a change in the optical system, for example, there is a method of detecting a change by recording feature points between frames and tracking a change in the feature points close to the outer peripheral side of the surgical field image.

[0133] Parameter estimation unit 112

[0134] The parameter estimation unit 112 estimates parameters by using the position and orientation information of the imaging unit 101 obtained from the robot arm device 212. Parameter estimation based on information obtained from the robot arm is disclosed in, for example, "The Advantage of Mounting a Camera onto a Robot Arm by Radu Horaud," Central European Workshop on Geometric Modeling and Invariants in Computer Vision, 1995, pp. 206-213.

[0135] Figure 5 The operation of the surgical assisting system 200 is basically the same as that of the reference Figure 4 The operations described are the same.

[0136] As described above, according to the first embodiment, even when the optical system is changed during surgery, the accuracy of the changed three-dimensional information can be maintained, and the three-dimensional information generated before the change and the three-dimensional information generated after the change can be continuously used.

[0137] 2. Second Implementation Method (for Training)

[0138] <Configuration Example of Surgery Support System>

[0139] Figure 6 : is a block diagram illustrating an example of the hardware configuration of an information processing device 300 configuring a surgery support system according to a second embodiment of the present technology.

[0140] include Figure 6 The surgical support system of the information processing device 300 is, for example, a system that displays images during surgery, and the images are recorded in the Figure 1 The surgical assistance system including the information processing device 300 is used to train an operator or a student. The surgical assistance system including the information processing device 300 may also be referred to as an endoscopic surgery training system.

[0141] like Figure 6 As shown, the information processing apparatus 300 includes, for example, a computer and the like.

[0142] The CPU 301 , the ROM 302 , and the RAM 303 are connected to one another via a bus 304 .

[0143] An input and output interface 305 is also connected to the bus 304 . An input unit 306 including a keyboard, a mouse, and the like, and an output unit 307 including a display, a speaker, and the like are connected to the input and output interface 305 .

[0144] Furthermore, a storage unit 308 including a hard disk, a nonvolatile memory, and the like, a communication unit 309 including a network interface and the like, and a drive 310 that drives a removable medium 311 are connected to the input and output interface 305 .

[0145] The second embodiment differs from the first embodiment in that, when the surgery support system is used for training, it is not necessary to estimate parameters immediately. The processing can be performed after the entire surgical field image is read once.

[0146] In the second embodiment, a three-dimensional map optimized for the entire recorded image (integrated three-dimensional map) is generated once, and then SLAM including camera orientation estimation is operated to display the three-dimensional map.

[0147] <Functional Configuration Example of Surgery Support System>

[0148] (Overall configuration)

[0149] Figure 7 is a block diagram showing a functional configuration example of a surgery support system.

[0150] Figure 7 The surgery support system 350 includes an image storage unit 351 , an information processing unit 352 , and a display unit 353 .

[0151] The image storage unit 351 corresponds to Figure 6 The storage unit 308 in the image storage unit 351 stores the image data obtained by the endoscope 11 ( Figure 1 ) captured surgical field images.

[0152] The information processing unit 352 is composed of Figure 6The information processing unit 352 performs signal processing on the surgical field image stored in the image storage unit 351 and supplies the surgical field image obtained by performing the signal processing to the display unit 353 .

[0153] The display unit 353 corresponds to the configuration Figure 6 The display unit 353 displays the surgical field image based on the image signal supplied from the information processing unit 352 .

[0154] (Detailed Configuration of Information Processing Unit 352)

[0155] The information processing unit 352 includes an optical system change detection unit 361, a three-dimensional map generation unit 362, a three-dimensional map storage unit 363, a three-dimensional information generation unit 364, and a display information generation unit 365. At least a part of the information processing unit 352 is executed by Figure 6 The description overlapping with the above description will be omitted as appropriate.

[0156] Optical system change detection unit 361

[0157] The optical system change detection unit 361 detects changes in the optical system by referring to the entire surgical field image stored in the image storage unit 351. Figure 2 The optical system change detection unit 111 performs the detection of the optical system change.

[0158] The optical system change detecting unit 361 sets a section of frames having the same parameters, that is, a section of frames in which the optical system has not changed, as a part.

[0159] The optical system change detection unit 361 estimates the parameters of each part. Figure 2 The parameter estimation unit 112 in FIG. 36 is used to perform parameter estimation. The optical system change detection unit 361 outputs the parameters of each part to the three-dimensional map generation unit 362.

[0160] 3D map generation unit 362

[0161] The three-dimensional map generating unit 362 generates a three-dimensional map of each part by using the parameters supplied from the optical system variation detecting unit 361. The three-dimensional map generated by the three-dimensional map generating unit 362 is a three-dimensional map of objects appearing in the surgical field images of the plurality of frames constituting the part.

[0162] In the generation of 3D maps, for example, in addition to Visual SLAM or RGB-D-SLAM, multi-view stereo or SfM, which can generate 3D maps from multiple viewpoints, can also be used. For example, multi-view stereo is described in "Multi-View Stereo: A Tutorial. Foundations and. Trends R in Computer Graphics and Vision, vol. 9, no. 1-2, 2013, pp. 1-148" and "Evaluation of multi-view 3D reconstruction software, CAIP2015: Computer Analysis of Images and Patterns, pp. 450-461".

[0163] The three-dimensional map generation unit 362 outputs the three-dimensional map of each section to the three-dimensional map storage unit 363 .

[0164] Three-dimensional map storage unit 363

[0165] The three-dimensional map storage unit 363 stores the three-dimensional map of each section generated by the three-dimensional map generation unit 362 .

[0166] Three-dimensional information generation unit 364

[0167] The three-dimensional information generating unit 364 integrates the three-dimensional map of each section stored in the three-dimensional map storing unit 363 and generates a three-dimensional map integrated by all sections.

[0168] Since the three-dimensional map generated for each part has different parameters for each part, the scale and position are different, making it difficult to integrate the three-dimensional maps as they are and use the integrated three-dimensional map for SLAM processing. Therefore, in the three-dimensional information generation unit 364, the scale and position relationship of each part are corrected, and the three-dimensional maps are integrated while optimizing the scale, etc.

[0169] Specifically, the three-dimensional information generating unit 364 estimates the scale of the three-dimensional map of another portion with respect to the three-dimensional map used as a reference in each portion so that the scales of the three-dimensional maps of all portions are integrated.

[0170] Each point in the three-dimensional map generated in each section holds a vector, called a feature quantity, representing the characteristics of that point. The three-dimensional information generation unit 364 can identify overlapping sections in the three-dimensional maps by searching for points that share the same feature quantity in different three-dimensional maps. Within the overlapping sections, the unit 364 uses the least squares method to identify the scale and positional relationship that minimizes the residual error.

[0171] Note that the points holding the feature amounts include feature points of the surgical field image, feature points of the three-dimensional map, and the like.

[0172] Feature points of the surgical field image are, for example, SIFT, SURF, ORB, AKAZE, and the like.

[0173] The characteristic points of the three-dimensional map are, for example, SHOT, PFH, and PPF.

[0174] In addition, in a case where each point of the generated three-dimensional map does not maintain feature quantities and identification of corresponding points (i.e., identification of overlapping parts) cannot be performed in the three-dimensional map, ICP can also be used to perform alignment of two point groups while simultaneously estimating the correspondence.

[0175] The three-dimensional information generation unit 364 generates three-dimensional information according to SLAM, including estimating the camera's own position and orientation by using the integrated three-dimensional map.

[0176] Three-dimensional information generating section 364 outputs the generated three-dimensional information to display information generating section 365 .

[0177] In the above description, an example of a case where a three-dimensional map of each part is generated in the three-dimensional map generation unit 362 has been described, but three-dimensional information (position and direction information and three-dimensional map) of each part may be generated in the three-dimensional map generation unit 362. The three-dimensional information of each part is stored in the three-dimensional map storage unit 363.

[0178] At this time, in the three-dimensional information generating unit 364 , pieces of three-dimensional information for each portion are integrated, SLAM processing including estimation of the camera's own position and orientation is performed by using the integrated three-dimensional information, and three-dimensional information is generated again.

[0179] Display information generation unit 365

[0180] The display information generating unit 365 is similar to Figure 2 The display information generating unit 114 causes the display unit 353 to display the surgical field image based on the image signal read from the image storage unit 351 .

[0181] Furthermore, the display information generating unit 365 causes the display unit 353 to display the integrated three-dimensional map based on the three-dimensional information supplied from the three-dimensional information generating unit 364 .

[0182] <Operation Example of Surgery Support System>

[0183] Figure 8 3D information generation processing in the surgery support system 350 is shown in FIG.

[0184] In step S311 , the optical system change detection unit 361 reads the surgical field image represented by the image signal obtained from the image storage unit 351 .

[0185] In step S312 , the optical system change detecting unit 361 refers to the entire surgical field image and sets a segment of frames having the same parameters, ie, a segment of frames without optical system change, as a part based on the detection result of the change in the optical system.

[0186] In step S313 , the optical system variation detecting unit 361 estimates the parameters of each section.

[0187] In step S314 , the three-dimensional map generation unit 362 generates a three-dimensional map for each section.

[0188] In step S315 , the three-dimensional map storage unit 363 stores the three-dimensional map of each section generated by the three-dimensional map generation unit 362 .

[0189] In step S316, the three-dimensional information generation unit 364 integrates the three-dimensional maps of each portion stored in the three-dimensional map storage unit 363 and generates an integrated three-dimensional map. The three-dimensional information generation unit 364 generates three-dimensional information according to SLAM, including estimating the camera's own position and orientation by using the integrated three-dimensional map.

[0190] In step S317 , the display information generating unit 365 causes the display unit 353 to display a three-dimensional map based on the three-dimensional information supplied from the three-dimensional information generating unit 364 .

[0191] When the three-dimensional map is displayed on the display unit 353 in step S317 , the processing of the surgery support system 350 ends.

[0192] In the above-described processing, parameters serving as generation conditions of three-dimensional information are updated for each portion set according to a change in the optical system of the imaging unit 101 , and the three-dimensional map generated for each portion is integrated.

[0193] As described above, according to the second embodiment, when the surgery support system is used for post-operative training or the like, even if the optical system changes during surgery, it is possible to prevent errors from occurring in three-dimensional information.

[0194] 《3. Application Examples》

[0195] Next, we will refer to Figure 9 An example is described of a case where a surgical video microscope apparatus including an arm is used as an application example of the surgery support system according to an embodiment of the present technology.

[0196] Figure 9An example of a microsurgery system using a surgical video microscope apparatus as an observation medical device for observing the inside of a patient's body is shown.

[0197] Figure 9 A state is shown in which a doctor as an operator (user) 520 performs surgery on a surgical object (patient) 540 on an operating table 530 by using, for example, a surgical instrument 521 (eg, a scalpel, forceps, or pliers).

[0198] Note that in the following description, surgery is a general term for various medical treatments (eg, surgery and examinations) performed by a doctor who is a user 520 on a patient who is a surgery target 540. Figure 9 In the example of FIG. 5 , the state of surgery is shown as an example of surgery, but surgery using the surgical video microscope device 510 is not limited to surgery and may be other various surgeries.

[0199] A surgical video microscope device 510 according to an embodiment of the present technology is provided next to an operating table 530 .

[0200] The surgical video microscope device 510 includes a base unit 511 as a base, an arm unit 512 extending from the base unit 511 , and an imaging unit 515 as a distal end unit connected to a distal end of the arm unit 512 .

[0201] The arm unit 512 includes a plurality of joints 513 a , 513 b , and 513 c , a plurality of links 514 a and 514 b connected by the joints 513 a and 513 b , and an imaging unit 515 provided at a distal end of the arm unit 512 .

[0202] exist Figure 9 In the example shown in FIG, for simplicity, the arm unit 512 includes three joints 513a to 513c and two links 514a and 514b. In practice, the number and shape of the joints 513a to 513c and the links 514a and 514b, the directions of the drive axes of the joints 513a to 513c, and the like can be appropriately set to achieve the desired degrees of freedom, taking into account the degrees of freedom of position and orientation of the arm unit 512 and the imaging unit 515.

[0203] The joints 513 a to 513 c have a function of rotatably connecting the links 514 a and 514 b to each other, and driving of the arm unit 512 is controlled by driving the rotation of the joints 513 a to 513 c .

[0204] The imaging unit 515 is connected to the distal end of the arm unit 512 as a distal unit.

[0205] The imaging unit 515 is a unit that acquires an image to be captured by including an optical system that acquires an optical image of a subject, and is configured as, for example, a camera or the like that can capture a moving image or a still image. Figure 9 As shown, the positions and directions of the arm unit 512 and the imaging unit 515 are controlled by the surgical video microscope device 510 , so that the imaging unit 515 disposed at the distal end of the arm unit 512 images the state of the surgical site of the surgical target 540 .

[0206] Note that the configuration of the imaging unit 515 as a distal unit connected to the distal end of the arm unit 512 is not particularly limited, and for example, the imaging unit 515 may be configured as an endoscope or a microscope.

[0207] With this configuration, for example, the imaging unit 515 corresponding to the application can be appropriately connected to the distal end of the arm unit 512 as the distal unit. Note that here, the description will focus on the case where the imaging unit 515 is used as the distal unit, but it goes without saying that the distal unit connected to the distal end of the arm unit 512 is not necessarily limited to the imaging unit 515.

[0208] In addition, a display device 550 (e.g., a monitor or display) is installed at a position facing the user 520. The image of the surgical site acquired by the imaging unit 515 is subjected to various image processing by, for example, an image processing device built into or externally attached to the surgical video microscope device 510, and then displayed as an electronic image on the display screen of the display device 550.

[0209] With this configuration, the user 520 can perform various treatments (eg, surgery, etc.) while observing the electronic image of the surgical site displayed on the display screen of the display device 550 .

[0210] Here, in Figure 9 In the example, the imaging unit 515 includes, for example, a reference Figure 2 The imaging unit 101 described above. In addition, the image processing device that performs various image processing on the image of the surgical site acquired by the imaging unit 515 corresponds to the image processing device described in reference to FIG. Figure 2 In a similar manner, the display device 550 corresponds to the example of the information processing unit 102 described in reference Figure 2 An example of the display unit 103 is described.

[0211] In addition, Figure 9 In the example of FIG. 1 , the arm unit 512 corresponds to the reference Figure 5 In a similar manner, the surgical video microscope device 510 includes a reference to Figure 5 The robotic arm device 212 is described.

[0212] 《4. Hardware Configuration》

[0213] Next, we will refer to Figure 10 An example of the hardware configuration of the information processing device configuring the surgery support system according to the embodiment of the present technology is described in detail.

[0214] Figure 10 : is a block diagram illustrating an example of the hardware configuration of the information processing device 900 configuring the surgery support system according to the embodiment of the present technology.

[0215] like Figure 10 As shown, the information processing device 900 includes a CPU 901, a ROM 903, and a RAM 905. In addition, the information processing device 900 includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, and a storage device 919. Note that the information processing device 900 may include a drive 921, a connection port 923, and a communication device 925.

[0216] The CPU 901 functions as an arithmetic processing device and a control device, and controls the entire operation or part of the operation in the information processing device 900 according to various programs recorded in the ROM 903 , the RAM 905 , the storage device 919 , or the removable recording medium 927 .

[0217] The ROM 903 stores programs used by the CPU 901, operating parameters, and the like. The RAM 905 mainly stores programs used by the CPU 901, parameters that change appropriately during program execution, and the like. These are connected to each other via the host bus 907 including an internal bus (e.g., a CPU bus). Figure 2 Each configuration of the described information processing unit 102 is realized by, for example, the CPU 901 .

[0218] The host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909. An input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925 are connected to the external bus 911 via an interface 913.

[0219] For example, the input device 915 is an operating device operated by the user, such as a mouse, keyboard, touchpad, button, switch, lever, and pedal. In addition, the input device 915 may be, for example, a remote control device (so-called remote controller) using infrared or other radio waves. The input device 915 may be, for example, an external connection device 929 such as a mobile phone, smartphone, or tablet terminal, which corresponds to the operation of the information processing device 900.

[0220] The input device 915 includes, for example, an input control circuit that generates an input signal based on information input by a user using the above-described operation device and outputs the input signal to the CPU 901 .

[0221] By operating the input device 915 , the user can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations.

[0222] The output device 917 includes a device capable of visually or aurally notifying the user of acquired information. Specifically, examples of the output device 917 include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, electroluminescent display devices, and lamps, audio output devices such as speakers and headphones, printers, and the like.

[0223] The output device 917 outputs, for example, the results obtained by various processes performed by the information processing device 900. Specifically, the display device displays the results obtained by various processes performed by the information processing device 900 as text or images. On the other hand, the audio output device converts an audio signal including reproduced audio data, acoustic data, etc. into an analog signal and outputs the analog signal. Note that, with reference to FIG. Figure 2 The depicted display unit 103 is realized by, for example, the output device 917 .

[0224] The storage device 919 is a data storage device that is an example of a storage unit of the information processing device 900. The storage device 919 includes, for example, a magnetic storage unit device such as an HDD (hard disk drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 919 stores programs executed by the CPU 901, various data, and the like.

[0225] The drive 921 is a recording medium reader / writer, and is built-in or externally attached to the information processing apparatus 900. The drive 921 reads information recorded on a mounted removable recording medium 927 (e.g., a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory), and outputs the information to the RAM 905. Furthermore, the drive 921 can also write records on the mounted removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0226] The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, or the like. Furthermore, the removable recording medium 927 may be a Compact Flash (registered trademark) (CF: CompactFlask), a flash memory, an SD (Secure Digital) memory card, or the like. Furthermore, the removable recording medium 927 may be, for example, an IC (Integrated Circuit) card on which a contactless IC chip is mounted, an electronic device, or the like.

[0227] The connection port 923 is a port for directly connecting an external connection device 929 to the information processing device 900. Examples of the connection port 923 include a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, and the like. Other examples of the connection port 923 include an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface) (registered trademark) port, and the like. By connecting the external connection device 929 to the connection port 923, the information processing device 900 directly obtains various data from the external connection device 929 or provides various data to the external connection device 929.

[0228] The communication device 925 is, for example, a communication interface including a communication device for connecting to a communication network (network) 931. The communication device 925 is, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). In addition, the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like.

[0229] For example, the communication device 925 can transmit and receive signals to and from the Internet and other communication devices according to a predetermined protocol (e.g., TCP / IP). In addition, the communication network 931 connected to the communication device 925 may include a network connected in a wired or wireless manner, etc. The communication network 931 may be, for example, the Internet, a home LAN, or a communication network in which infrared communication, radio wave communication, or satellite communication is performed.

[0230] Described above Figure 5 The information processing device 300 and Figure 10 Each component of the information processing device 900 can be configured by using a general-purpose member, or can be configured by hardware dedicated to the function of each component. Therefore, when implementing the embodiment of the present technology, the hardware configuration to be used can be appropriately changed according to the technical level.

[0231] Furthermore, a computer program for implementing each function of the information processing device 300 and the information processing device 900 constituting the surgical support system according to an embodiment of the present technology can be generated and installed on a personal computer or the like. Furthermore, a computer-readable recording medium storing such a computer program can be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, and the like. Furthermore, the computer program can be distributed via, for example, a network without using a recording medium.

[0232] Note that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing (for example, upon calling).

[0233] 5. Others

[0234] As described above, this technology acquires surgical field data captured by a medical observation device and detects changes in the device's optical system. If changes in the optical system are detected, parameters determined based on the changed optical system are estimated, and the estimation results are used to set the conditions for generating three-dimensional information based on the surgical field data. Consequently, the accuracy of the three-dimensional information can be maintained even when changes in the optical system occur.

[0235] Even if the optical system changes during surgery, the accuracy of the three-dimensional information can be maintained without the need to recalibrate the parameters.

[0236] Furthermore, in the case where the surgery support system is used for post-operative training or the like, even when a change in the optical system occurs in the surgical field image, it is possible to prevent errors from occurring in the three-dimensional information.

[0237] Note that in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0238] Furthermore, the effects described in this specification are merely examples, and are not limited thereto, and other effects may be provided.

[0239] The embodiment of the present technology is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present technology.

[0240] For example, the present technology may have a cloud computing configuration in which one function is shared and processing is performed cooperatively by a plurality of devices via a network.

[0241] Furthermore, each step described in the above flowchart may be performed by one device, or may be shared and performed by a plurality of devices.

[0242] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be shared and executed by a plurality of devices.

[0243] <Configuration combination example>

[0244] The present technology may also have the following configurations.

[0245] (1) A medical observation system comprising:

[0246] an acquisition unit configured to acquire surgical field data acquired by a medical observation device;

[0247] a detection unit configured to detect a change in an optical system of the medical observation device;

[0248] an estimating unit configured to estimate a parameter determined based on the optical system after the change, when the detecting unit detects a change in the optical system; and

[0249] A setting unit is configured to set a generation condition of three-dimensional information based on surgical field data by using the estimation result of the estimation unit.

[0250] (2) The medical observation system according to (1), wherein

[0251] The detection unit detects a change in the optical system by detecting a change in a surgical field image represented by the surgical field data.

[0252] (3) The medical observation system according to (1), wherein

[0253] The detection unit detects a change in the optical system by detecting a change in the focus of a surgical field image represented by the surgical field data.

[0254] (4) The medical observation system according to (1), wherein

[0255] The detection unit detects a change in the optical system by using a position trajectory of the medical observation device held by the robot arm.

[0256] (5) The medical observation system according to (1), wherein

[0257] The detection unit detects that there is a change in the optical system when the viewing angle of the surgical field image represented by the surgical field data changes and the medical observation device held by the robot arm does not move.

[0258] (6) The medical observation system according to any one of (1) to (5), wherein

[0259] The detection unit divides the surgical field data into each portion as a segment of a frame including a plurality of surgical field images based on the change of the optical system, and

[0260] The estimation unit estimates a parameter of each part.

[0261] (7) The medical observation system according to any one of (1) to (5), wherein

[0262] The estimation unit estimates parameters corresponding to the information of the optical system acquired from the medical observation apparatus based on a table obtained in advance that indicates a relationship between the information of the optical system and the parameters.

[0263] (8) The medical observation system according to any one of (1) to (5), wherein

[0264] The estimation unit estimates parameters based on surgical field data.

[0265] (9) The medical observation system according to (8), wherein

[0266] The estimation unit estimates parameters from a surgical field image represented by surgical field data and generates a reliability index of a parameter matrix.

[0267] (10) The medical observation system according to any one of (1) to (9), further comprising:

[0268] The three-dimensional information generating unit is configured to generate three-dimensional information by using the parameters estimated by the estimating unit.

[0269] (11) The medical observation system according to (10), wherein

[0270] The three-dimensional information generating unit stops generating the three-dimensional information if the detecting unit detects a change in the optical system, and resumes generating the three-dimensional information by using the estimated parameters if the parameters are estimated by the estimating unit.

[0271] (12) The medical observation system according to any one of (1) to (11), further comprising:

[0272] A display control unit is configured to control display of a surgical field image or three-dimensional information represented by the surgical field data.

[0273] (13) The medical observation system according to (12), wherein

[0274] The display control unit displays a detection result of a change in the optical system obtained by the detection unit.

[0275] (14) The medical observation system according to (13), wherein

[0276] The display control unit displays information indicating that the endoscope of the medical observation apparatus is replaced as a detection result.

[0277] (15) The medical observation system according to (13), wherein

[0278] The display control unit displays information related to the endoscope of the medical observation apparatus as a detection result.

[0279] (16) The medical observation system according to (13), wherein

[0280] The display control unit displays the three-dimensional information before the change and the three-dimensional information after the change.

[0281] (17) A medical observation method comprising causing a medical observation system to:

[0282] Acquiring surgical field data acquired by a medical observation device;

[0283] Detecting changes in the optical systems of medical observation devices;

[0284] In the event that a change in the optical system is detected, estimating parameters determined based on the optical system after the change; and

[0285] By using the estimation result, generation conditions of three-dimensional information based on the surgical field data are set.

[0286] (18) A medical observation device comprising:

[0287] an imaging unit configured to image a surgical field and generate surgical field data; and

[0288] An output unit configured to output surgical field data,

[0289] The medical observation device is used in a medical observation system, which detects changes in the optical system of an imaging unit, estimates parameters determined according to the optical system after the change, and sets generation conditions of three-dimensional information based on surgical field data by using the estimation results.

[0290] Explanation of symbols

[0291] 1. Surgical Assistance System

[0292] 11 Endoscope

[0293] 13 CCU

[0294] 15 Display device

[0295] 100 Surgical Assistance Systems

[0296] 101 Camera

[0297] 102 Information Processing Unit

[0298] 103 Display unit

[0299] 111 Optical system change detection unit

[0300] 112 Parameter Estimation Unit

[0301] 113 Three-dimensional information generation unit

[0302] 114 Display information generation unit

[0303] 200 Surgical Assistance System

[0304] 211 Robotic Arm

[0305] 212 Robotic Arm Device

[0306] 300 Surgical Assistance System

[0307] 301 CPU

[0308] 307 Output Unit

[0309] 308 storage unit

[0310] 350 Surgical Assistance System

[0311] 351 Image Storage Unit

[0312] 352 Information Processing Unit

[0313] 353 Display Unit

[0314] 361 Optical System Change Detection Unit

[0315] 362 3D map generation unit

[0316] 363 Three-dimensional map storage unit

[0317] 364 Three-dimensional information generation unit

[0318] 365 Display information generation unit.

Claims

1. A medical observation system comprising: an acquisition unit configured to acquire surgical field data acquired by a medical observation device; a detection unit configured to detect a change in an optical system of the medical observation device in response to a change in the viewing angle of the surgical field image represented by the surgical field data while the medical observation device held by the robot arm does not move; an estimating unit configured to estimate a parameter determined based on the optical system after the change, if the detecting unit detects a change in the optical system; as well as a setting unit configured to set a generation condition of three-dimensional information based on the surgical field data by using the estimation result of the estimation unit, wherein the estimation unit estimates parameters corresponding to the information of the optical system acquired from the medical observation apparatus based on a table obtained in advance indicating a relationship between the information of the optical system and the parameters, The estimation unit determines whether to update the parameters according to a threshold value determination using a reliability index of the parameter matrix obtained after the estimation.

2. The medical observation system according to claim 1, wherein: The detection unit detects a change in the optical system by detecting a change in a surgical field image represented by the surgical field data.

3. The medical observation system according to claim 1, wherein: The detection unit detects a change in the optical system by detecting a change in the focus of a surgical field image represented by the surgical field data.

4. The medical observation system according to claim 1, wherein: The detection unit detects a change in the optical system by using a position trajectory of the medical observation device held by a robot arm.

5. The medical observation system according to claim 1, wherein: The detection unit divides the surgical field data into each portion as a segment of a frame including a plurality of surgical field images based on the change of the optical system, and The estimation unit estimates a parameter of each part.

6. The medical observation system according to claim 1, further comprising: The three-dimensional information generating unit is configured to generate three-dimensional information by using the parameters estimated by the estimating unit.

7. The medical observation system according to claim 6, wherein: The three-dimensional information generating unit stops generating the three-dimensional information if the detecting unit detects a change in the optical system, and resumes generating the three-dimensional information by using the estimated parameters if the parameters are estimated by the estimating unit.

8. The medical observation system according to claim 1, further comprising: A display control unit is configured to control display of a surgical field image or three-dimensional information represented by the surgical field data.

9. The medical observation system according to claim 8, wherein: The display control unit displays a detection result of the change of the optical system obtained by the detection unit.

10. The medical observation system according to claim 9, wherein: The display control unit displays information indicating that the endoscope of the medical observation apparatus is replaced as a detection result.

11. The medical observation system according to claim 9, wherein: The display control unit displays information related to the endoscope of the medical observation apparatus as a detection result.

12. The medical observation system according to claim 9, wherein: The display control unit displays the three-dimensional information before the change and the three-dimensional information after the change.

13. A medical observation method, comprising causing a medical observation system to: Acquiring surgical field data acquired by a medical observation device; detecting a change in the optical system of the medical observation device in response to a change in the viewing angle of the surgical field image represented by the surgical field data while the medical observation device held by the robot arm does not move; In the case where a change in the optical system is detected, estimating parameters determined based on the optical system after the change; setting generation conditions of three-dimensional information based on the surgical field data by using the estimation result; as well as estimating parameters corresponding to the information of the optical system acquired from the medical observation apparatus based on a table obtained in advance indicating a relationship between the information of the optical system and parameters, The method further includes determining whether to update the parameters according to a threshold value of a reliability indicator using the parameter matrix obtained after the estimation.

14. A medical observation device comprising: an imaging unit configured to image a surgical field and generate surgical field data; as well as an output unit configured to output the surgical field data, The medical observation device is used in a medical observation system, which detects changes in the optical system of an imaging unit. When the medical observation device held by a robotic arm does not move in response to a change in the viewing angle of the surgical field image represented by the surgical field data, the medical observation system estimates parameters determined according to the optical system after the change, and sets generation conditions for three-dimensional information based on the surgical field data by using the estimation result, estimates parameters corresponding to the information of the optical system acquired from the medical observation device based on a pre-obtained table representing the relationship between the information and parameters of the optical system, and determines whether to update the parameters based on a threshold determination of a reliability index of a parameter matrix obtained after the estimation.

Images