Surgical assistance device, surgical method, non-transitory computer readable medium, and surgical assistance system

Abstract

A surgical assisting device includes a circuit configured to generate a first image to be displayed in a first display area and including a first surgical field image of a surgical object and a first visual annotation image superimposed on a predetermined area in the surgical field of the first surgical field image corresponding to a body area of ​​the surgical object. The circuit is also configured to generate a second image to be displayed in a second display area and including a second surgical field image and a second visual annotation image superimposed on an estimated area in the second surgical field image corresponding to the body area of ​​the surgical object.

Description

Technical Field

[0001] The present disclosure relates to a surgery support system and a display method, and in particular, to a surgery support system and a display method capable of improving communication between operators.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of Japanese Priority Patent Application JP 2018-052776 filed on Mar. 20, 2018, the entire contents of which are hereby incorporated by reference. Background Art

[0004] At some surgical sites, an advising doctor performs surgical assistance, such as presenting the area to be operated on, in order to assist less experienced surgeons. In addition, in some cases, multiple surgeons perform surgery while consulting each other. Therefore, a medical device having the function of explicitly sending instructions from one operator to another operator is needed.

[0005] For example, Patent Document 1 discloses a three-dimensional observation device capable of writing annotations into a three-dimensional surgical image.

[0006] Reference List

[0007] Patent Literature

[0008] Patent Document 1: WO 2013 / 179905 A Summary of the invention

[0009] Technical issues

[0010] However, in a conventional configuration, via a single operative field image, a consulting doctor presents a region to be operated on, or a surgeon confirms the presented region to be operated on.

[0011] In this case, there is a possibility that the action of writing annotations to the surgical field image by the first user interferes with the operator (i.e., the second user) who is performing the operation while confirming the surgical field image by blurring the surgical field image, and the operators (i.e., the first user and the second user) cannot successfully communicate with each other.

[0012] The present disclosure has been made in view of the above circumstances, and according to the embodiments of the present disclosure, communication between operators can be improved.

[0013] Solution to the problem

[0014] A surgical assisting device includes a circuit configured to generate a first image to be displayed in a first display area and including a first surgical field image of a surgical object and a first visual annotation image superimposed on a predetermined area in the surgical field of the first surgical field image corresponding to a body area of ​​the surgical object. The circuit is also configured to generate a second image to be displayed in a second display area and including a second surgical field image and a second visual annotation image superimposed on an estimated area in the second surgical field image corresponding to the body area of ​​the surgical object.

[0015] A surgical method includes: generating a first image, which will be displayed in a first display area and includes a first surgical field image of a surgical object and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical object, and generating a second image, which will be displayed in a second display area and includes a second surgical field image and a second visual annotation image superimposed on an estimated area of ​​the second surgical field image corresponding to the body area of ​​the surgical object.

[0016] A non-transitory computer-readable medium storing instructions which, when executed by a computer, cause the computer to perform the following steps: generating a first image which will be displayed in a first display area and includes a first surgical field image of a surgical object and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical object; and generating a second image which will be displayed in a second display area and includes a second surgical field image and a second visual annotation image superimposed on an estimated area of ​​the second surgical field image corresponding to a body area of ​​the surgical object.

[0017] A surgical assistance system comprises: an endoscope including an imager for capturing a second surgical field image of a surgical object; a circuit configured to generate a first image, which will be displayed in a first display area and includes the first surgical field image of the surgical object and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical object, and to generate a second image, which will be displayed in a second display area and includes the second surgical field image and a second visual annotation image superimposed on an estimated area of ​​the second surgical field image corresponding to the body area of ​​the surgical object; and a surgical tool, which moves inside the surgical object under the control of a medical staff based on the second visual annotation image in the second image to perform surgical treatment on the surgical object.

[0018] Advantageous Effects of the Invention

[0019] According to an embodiment of the present disclosure, communication between operators can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] [ Figure 1 ] Figure 1 A configuration example of a surgery assisting system according to an embodiment of the present disclosure is shown.

[0021] [ Figure 2 ] Figure 2 is a diagram for explaining the display of comments.

[0022] [ Figure 3 ] Figure 3 is a block diagram showing a functional configuration example of a surgery support system.

[0023] [ Figure 4 ] Figure 4 is a block diagram illustrating a detailed configuration example of a sensing unit and an image data generator.

[0024] [ Figure 5 ] Figure 5 is a block diagram illustrating a detailed configuration example of a sensing unit and an image data generator.

[0025] [ Figure 6 ] Figure 6 is a block diagram illustrating a detailed configuration example of a sensing unit and an image data generator.

[0026] [ Figure 7 ] Figure 7 This is a flowchart illustrating comment writing processing.

[0027] [ Figure 8 ] Figure 8 This is a flowchart illustrating comment writing processing.

[0028] [ Fig. 9 ] Fig. 9 A display example of a surgical field image during annotation writing processing is shown.

[0029] [ Fig.10 ] Fig.10 A display example of a surgical field image during annotation writing processing is shown.

[0030] [ Fig.11 ] Fig.11 A display example of a surgical field image during annotation writing processing is shown.

[0031] [ Fig.12 ] Fig.12 A display example of a surgical field image during annotation writing processing is shown.

[0032] [ Fig.13 ] Fig.13 A display example of a surgical field image during annotation writing processing is shown.

[0033] [ Fig.14 ] Fig.14 A display example of a surgical field image during annotation writing processing is shown.

[0034] [ Fig.15 ] Fig.15 A display example of a surgical field image during annotation writing processing is shown.

[0035] [ Fig.16 ] Fig.16 2 is a diagram for explaining a display mode of a surgical field image.

[0036] [ Fig.17 ] Fig.17 2 is a diagram for explaining a display mode of a surgical field image.

[0037] [ Fig.18 ] Fig.18 2 is a diagram for explaining a display mode of a surgical field image.

[0038] [ Fig.19 ] Fig.19 2 is a diagram for explaining a display mode of a surgical field image.

[0039] [ Fig. 20 ] Fig. 20 An example of writing annotations to past surgical field images is shown.

[0040] [ Fig.21 ] Fig.21 An example of a masked display is shown.

[0041] [ Fig. 22 ] Fig. 22 An example of a preview display of a three-dimensional image is shown.

[0042] [ Fig.23 ] Fig.23 is a diagram illustrating a method for writing a comment.

[0043] [ Fig.24 ] Fig.24 A display example of comments is shown.

[0044] [ Fig.25 ] Fig.25 A display example of comments is shown.

[0045] [ Fig.26 ] Fig.26 A display example of comments is shown.

[0046] [ Fig. 27 ] Fig. 27 Another configuration example of the surgery assisting system according to the embodiment is shown.

[0047] [ Fig.28 ] Fig.28 is a block diagram showing a hardware configuration example of an information processing device. DETAILED DESCRIPTION

[0048] Embodiments of the present disclosure (hereinafter referred to as embodiments) are described below. Note that the description will be made in the order described below.

[0049] 1. System Configuration

[0050] 2. Comment writing process

[0051] 3. Display mode of surgical field images

[0052] 4. Variant

[0053] 5. Application

[0054] 6. Hardware Configuration

[0055] 7. Summary

[0056] <1. System Configuration>

[0057] (Configuration example of surgical support system)

[0058] Figure 1 A configuration example of a surgery assisting system according to an embodiment of the present disclosure is shown.

[0059] Figure 1 An example of an endoscopic surgery system is shown, which is used, for example, for endoscopic surgery of the abdomen performed at a medical site instead of laparotomy in the related art.

[0060] exist Figure 1 In the surgical assisting system 1 shown, in endoscopic surgery of the abdomen, as in the prior art, a plurality of perforating tools called trocars 25a and 25b are attached through the abdominal wall instead of cutting the abdominal wall and performing a laparotomy. Then, a laparoscope (hereinafter also referred to as an endoscope or imager) 11 used as a medical device for observing the inside of the patient's body, an energy treatment tool 22, a forceps 23, etc. are inserted into the body through the holes provided in the trocars 25a and 25b.

[0061] The operator performs treatment such as resection of the affected part U (e.g., a tumor) inside the patient's body by using the energy treatment tool 22, etc., while observing in real time the image of the affected part U captured by performing video imaging using the endoscope 11. The endoscope 11, the energy treatment tool 22, and the forceps 23 are held by an operator, a robot, etc. Note that the operator refers to a medical staff member who participates in the operation performed in the operating room, and in addition to the surgeon, assistant, endoscopist, and nurse of the operation, the operator also includes, for example, a doctor who monitors the operation in a place different from the operating room, etc.

[0062] In an operating room where an endoscopic surgery as described above is performed, a trolley 31 on which devices for endoscopic surgery are installed, a bed 33 on which a patient lies, a foot switch 35, etc. are provided. In the trolley 31, for example, a camera control unit (CCU) 13 as medical equipment, a light source device 17, a device for treatment tools 21, a pneumoperitoneum device 24, a display device 15, a recorder 26, a printer 27, etc. are placed.

[0063] An image signal obtained by imaging the affected part U using the observation optical system of the endoscope 11 is sent to the CCU 13 via a camera cable. The CCU 13 may be connected to the endoscope 11 via a camera cable, or may be connected to the endoscope 11 via a radio communication path. The CCU 13 performs signal processing on the image signal that has been output from the endoscope 11, and outputs the image signal after the signal processing to the display device 15. By adopting the above configuration, an endoscopic image of the affected part U is displayed on the display device 15.

[0064] Note that the CCU 13 may output the image signal after signal processing to the recorder 26 so that the recorder 26 records the endoscopic image of the affected part U as image data (e.g., moving image data). In addition, the CCU 13 may output the image signal after signal processing to the printer 27 so that the printer 27 prints the endoscopic image of the affected part U.

[0065] The light source device 17 is connected to the endoscope 11 via a light guide cable, and can irradiate the affected part U with light of various wavelengths while switching the light. The light emitted from the light source device 17 can be used as auxiliary light, for example.

[0066] The device 21 for treatment means corresponds to, for example, a high-frequency output device that outputs a high-frequency current to the energy treatment tool 22 that cuts off the affected part U by using electric heat.

[0067] The pneumoperitoneum device 24 includes an air supply unit and an air suction unit, and supplies air to the abdominal region inside the patient's body (surgical object), for example.

[0068] The foot switch 35 controls the CCU 13 , the device for treatment tools 21 , and the like by using a foot operation of an operator, an assistant, and the like as a trigger signal.

[0069] (Display of comments)

[0070] The surgery assisting system according to the present embodiment realizes writing annotations on the surface of an organ or body cavity captured in an endoscopic image (hereinafter referred to as a surgical field image) that has been captured by the endoscope 11. The above-mentioned annotations refer to visual information used by an operator to present an area to be operated on, such as a body area of ​​a surgical object, to another operator without blurring the field of view of the other operator of the surgical field image.

[0071] Specifically, when the first user (operator) specifies that the first user wishes to write a comment in Figure 2 When the position on the surface of the organ captured in the surgical field image shown in the left hand portion of FIG. 1 is superimposed on the position (specified position) already specified by the first user on the surgical field image, as shown in FIG. Figure 2 shown in the center.

[0072] Even in the case where the position or posture of the endoscope 11 changes, the written annotation is displayed while tracking the designated position on the surgical field image, such as Figure 2 shown in the right hand portion of .

[0073] In the surgery support system according to the present embodiment, the position of the annotation (specified position) is retained based on the visual and stereoscopic texture of the surface of the organ or body cavity captured in the surgical field image and the positional relationship of the surface of the organ or body cavity, and the orientation and position (motion) of the endoscope 11 are estimated. Then, the display position is updated based on the result of the estimated motion so that the annotation remains superimposed at the same position on the organ (the estimated area of ​​the second surgical field image corresponding to the body area of ​​the surgical object).

[0074] The position (specified position) of the annotation is fixed regardless of the coordinates on the screen on which the surgical field image is displayed. Therefore, even in the case where the annotation is written to a still image obtained by stopping the surgical field image or a past surgical field image that has been recorded, the result of writing the annotation is reflected in the surgical field image currently displayed.

[0075] (Functional configuration example of surgical support system)

[0076] Figure 3 : is a block diagram showing a functional configuration example of a surgery support system capable of realizing the above-mentioned display of annotations.

[0077] Figure 3The illustrated surgery assisting system 100 is configured by a sensing unit 110 , an information processing device 130 , and a display device 150 .

[0078] The sensing unit 110 and Figure 1 The information processing device 130 corresponds to the endoscope 11 and provides an image signal obtained by performing imaging.

[0079] The information processing device 130 and Figure 1 The information processing device 130 corresponds to the CCU 13 , and performs signal processing on the image signal obtained from the sensing unit 110 , and provides the image signal to the display device 150 .

[0080] The display device 150 and Figure 1 The display device 150 corresponds to the display device 15 of the image processing apparatus 130, and displays the surgical field image based on the image signal obtained from the information processing apparatus 130. The details will be described later, but the display device 150 is provided with two display areas (a first display area 151 and a second display area 152), and displays one surgical field image in each of the two display areas.

[0081] The information processing device 130 includes an image data generator 131 , a display controller 132 , a designated position calculator 133 , and a three-dimensional surface mapping data memory 134 .

[0082] The image data generator 131 performs development processing on the image signal obtained from the sensing unit 110 to provide image data of a surgical field image including a plurality of frames to the display controller 132 .

[0083] Furthermore, the image data generator 131 estimates the movement of the endoscope 11 based on each frame configuring the surgical field image to generate position / posture information indicating the movement, and supplies the position / posture information to the display controller 132 and the specified position calculator 133 .

[0084] In addition, the image data generator 131 generates three-dimensional information indicating the three-dimensional structure of the object (the inside of an organ or a body cavity) captured in the surgical field image based on each frame configuring the surgical field image and the estimated movement of the endoscope 11. The generated three-dimensional information is stored in the three-dimensional surface mapping data storage 134 as three-dimensional surface mapping data.

[0085] Detailed configurations of the sensing unit 110 and the image data generator 131 will be described later.

[0086] The display controller 132 performs control to display a surgical field image in each of two display areas (a first display area 151 and a second display area 152 ) of the display device 150 based on the image data obtained from the image data generator 131 .

[0087] The display controller 132 includes a first superimposing unit 141 and a second superimposing unit 142 .

[0088] The first superimposing unit 141 superimposes the annotation on a position (designated position) designated by the user on the surgical field image displayed in the first display area 151. Specifically, the first superimposing unit 141 obtains the designated position on the surgical field image displayed in the first display area 151 based on the position / posture information obtained from the image data generator 131 and the designated position calculated by the designated position calculator 133, and the first superimposing unit 141 superimposes the annotation. The designated position corresponds to a body region of the surgical object.

[0089] On the other hand, the second superimposing unit 142 superimposes the annotation while tracking the designated position on the surgical field image displayed in the second display area 152. Specifically, the second superimposing unit 142 obtains the designated position on the surgical field image displayed in the second display area 152 based on the position / posture information obtained from the image data generator 131 and the designated position calculated by the designated position calculator 133, and the second superimposing unit 142 superimposes the annotation on the estimated area corresponding to the body area of ​​the surgical object.

[0090] When the designated position calculator 133 obtains information indicating that an annotation is to be written to the surgical field image displayed in the first display area 151 as operation information indicating the user's operation, the designated position calculator 133 calculates the designated position based on the position / posture information obtained from the image data generator 131 and the three-dimensional surface mapping data of the three-dimensional surface mapping data storage device 134.

[0091] Specifically, the designated position calculator 133 obtains which position on the three-dimensional surface mapping data corresponds to the position where the annotation has been written based on the current position and current posture of the endoscope 11 indicated by the position / posture information. The designated position obtained as described above is retained in the designated position calculator 133 and is provided to the first superimposition unit 141 and the second superimposition unit 142.

[0092] (Detailed Configuration Example 1 of Sensing Unit and Image Data Generator)

[0093] Here, reference Figure 4 A detailed configuration example of the sensing unit 110 and the image data generator 131 is described.

[0094] The sensing unit 110 includes an image sensor 211 , and supplies an image signal obtained by performing imaging to the information processing device 130 .

[0095] The image data generator 131 is configured by a development processing unit 221 , an initial three-dimensional data generator 222 , a camera position / posture estimation unit 223 , and a three-dimensional surface restoration unit 224 .

[0096] The development processing unit 221 performs development processing on the image signal obtained from the image sensor 211 to generate image data. The generated image data is supplied to each of the initial three-dimensional data generator 222, the camera position / posture estimation unit 223, and the three-dimensional surface restoration unit 224.

[0097] The initial three-dimensional data generator 222 obtains three-dimensional information indicating the three-dimensional structure of the object based on the correspondence between pixel positions having the same texture of the surface of the organ in the corresponding pieces of image data (frame data) provided in time series from the development processing unit 221. As the texture, the pixel value of the image data without change is used, or the feature amount extracted from the image data is used. The initial three-dimensional data generator 222 generates initial three-dimensional surface mapping data by using the obtained three-dimensional information.

[0098] The camera position / posture estimation unit 223 estimates the camera position / posture (estimates the movement of the endoscope 11) based on the correspondence between the pixel values ​​or feature quantities of each piece of image data (frame data) supplied in time series from the development processing unit 221 and the pixel values ​​or feature quantities of the temporally preceding image data. The camera position / posture estimation unit 223 generates position / posture information indicating the movement of the endoscope 11 based on the result of estimating the movement.

[0099] The three-dimensional surface restoration unit 224 restores three-dimensional information based on the pixel value or feature amount from the image data of the motion of the endoscope 11 which has been estimated by the camera position / posture estimation unit 223, and updates the three-dimensional surface mapping data which has been generated by the initial three-dimensional data generator 222. The updated three-dimensional surface mapping data is stored in the three-dimensional surface mapping data memory 134. In addition, the three-dimensional surface restoration unit 224 can read the three-dimensional surface mapping data stored in the three-dimensional surface mapping data memory 134 as three-dimensional information, and can update the three-dimensional surface mapping data.

[0100] Simultaneous localization and mapping (SLAM) techniques can be used to generate three-dimensional surface mapping data and estimate camera position / pose, as described above. For example, the basic principles of SLAM techniques using a monocular camera are described in Andrew J. Davison, "Real-Time Simultaneous Localization and Mapping with a Single Camera", Proceedings of the 9th IEEE International Conference on Computer Vision, Vol. 2, 2003, pp. 1403-1410. Note that SLAM techniques for visually estimating positions by using camera images are also specifically referred to as visual SLAM.

[0101] (Detailed Configuration Example 2 of Sensing Unit and Image Data Generator)

[0102] The sensing unit 110 and the image data generator 131 may also adopt Figure 5 Configuration shown.

[0103] Figure 5 The sensing unit 110 includes a depth sensor 232 in addition to the image sensor 231 , and provides depth data of an imaging range together with an image signal obtained by performing imaging to the image data generator 131 .

[0104] Figure 5 The image data generator 131 is configured by a development processing unit 241 , a three-dimensional surface restoration unit 242 , and a camera position / posture estimation unit 243 .

[0105] The development processing unit 241 performs development processing on the image signal obtained from the image sensor 231 to generate image data.

[0106] The three-dimensional surface restoration unit 242 generates three-dimensional surface mapping data by obtaining three-dimensional information about the object from the depth data obtained by the depth sensor 232 , or updates the three-dimensional surface mapping data by performing alignment in a three-dimensional space.

[0107] The camera position / posture estimation unit 243 compares the depth data currently obtained from the depth sensor 232 with the three-dimensional surface mapping data that has been generated or restored by the three-dimensional surface restoration unit 242 to estimate the camera position / posture (estimate the movement of the endoscope 11).

[0108] exist Figure 5 In the configuration, the motion of the endoscope 11 is estimated, and this causes the three-dimensional surface restoration unit 242 to search to perform spatial reduction of the alignment in three-dimensional space.

[0109] (Detailed Configuration Example 3 of Sensing Unit and Image Data Generator)

[0110] The sensing unit 110 and the image data generator 131 may also adopt Figure 6 Configuration shown.

[0111] Figure 6 The sensing unit 110 is configured as a stereo camera, includes a pair of image sensors 251 and 252 , and provides each of image signals obtained by performing imaging to the image data generator 131 .

[0112] Figure 6 The image data generator 131 is configured by a development processing unit 261 , a three-dimensional surface restoration unit 262 , and a camera position / posture estimation unit 263 .

[0113] The development processing unit 261 performs development processing on the image signal obtained from the image sensor 251 to generate image data.

[0114] Furthermore, the development processing unit 261 performs triangulation based on parallax using image signals obtained from the image sensors 251 and 252 to generate depth data of the imaging range, and supplies the depth data to the three-dimensional surface restoration unit 262 and the camera position / posture estimation unit 263 .

[0115] The three-dimensional surface restoration unit 262 generates three-dimensional surface mapping data by obtaining three-dimensional information about the object from the depth data obtained from the development processing unit 261 , or updates the three-dimensional surface mapping data by performing alignment in a three-dimensional space.

[0116] The camera position / posture estimation unit 263 compares the depth data currently obtained from the development processing unit 261 with the three-dimensional surface mapping data that has been generated or restored by the three-dimensional surface restoration unit 262 to estimate the camera position / posture (estimate the movement of the endoscope 11).

[0117] By employing the surgery support system 100 configured as described above, even in a case where the endoscope 11 has been moved, the annotation written on the surgical field image is displayed so that the annotation remains at the position where the annotation has been originally written.

[0118] (Insertion of 3D information)

[0119] In the surgery support system 100 , when the user writes an annotation, three-dimensional information may be inserted if the three-dimensional surface mapping data has a low density and there is no three-dimensional information corresponding to a position (specified position) where the annotation is to be superimposed.

[0120] For example, three-dimensional information about the designated position is estimated by using three-dimensional information corresponding to points around the designated position in the three-dimensional surface mapping data.

[0121] In addition, the surgical assisting system 100 has Figure 4 In the case of a configuration of , by using the camera position / posture estimated by the camera position / posture estimating unit 223 and the image data including a plurality of frames and corresponding to the camera position / posture, three-dimensional surface mapping data with high density can be obtained according to the multi-view stereo technique. In this case, three-dimensional information related to a specified position can be estimated by using the obtained three-dimensional surface mapping data with high density.

[0122] (Update of 3D information)

[0123] In the surgery support system 100 , when the object to be observed in the body cavity moves or changes in shape, the position of the annotation can be changed according to the change of the object to be observed.

[0124] For example, when updating three-dimensional surface mapping data by using three-dimensional information restored by the three-dimensional surface restoration unit 262, when the position of a specific area has significantly changed, the three-dimensional information related to the specific area is updated to the latest three-dimensional information that has been restored.

[0125] Furthermore, in the three-dimensional surface mapping data, by adding time information indicating the time of generation / update to the three-dimensional information related to each position, the reliability of the entire three-dimensional surface mapping data can be improved. By updating the three-dimensional surface mapping data in a manner that leaves the latest information with high reliability, even in the case where the object to be observed in the body cavity moves or changes in shape, annotations can be displayed while tracking the object to be observed.

[0126] <2. Comment writing process>

[0127] Next, refer to Figure 7 and Figure 8 The flowchart of FIG. 1 describes the flow of the annotation writing process of the surgery support system 100 according to the present embodiment.

[0128] For example, when a consultant presents the area to be operated on to assist a less experienced surgeon, Figure 7 and Figure 8 In this example, it is assumed that the consulting doctor presents the area to be operated by using the surgical field image displayed in the first display area 151 , and the surgeon performs the operation while observing the surgical field image displayed in the second display area 152 .

[0129] In step S11 , the display controller 132 determines whether a writing mode for enabling annotations to be written has been selected based on operation information indicating an operation of the user (specifically, the consulting doctor).

[0130] Fig. 9 A display example of the surgical field images displayed in the first display area 151 and the second display area 152 is shown.

[0131] In the first display area 151, the surgical field image captured by the sensing unit 110 is displayed in real time, and characters "Live" indicating this fact are displayed in the upper left portion of the screen of the first display area 151. Note that nothing is displayed in the second display area 152.

[0132] A writing mode button 311 and a freezing mode button 312 are displayed at the lower right portion of the first display area 151 having a touch panel function. The consulting doctor performs an operation to touch the writing mode button 311 displayed in the first display area 151, thereby selecting the writing mode.

[0133] In other words, the process of step S11 is repeated until the consulting doctor performs an operation of touching the writing mode button 311. When the operation of touching the writing mode button 311 is performed, the process proceeds to step S12.

[0134] In step S12 , the display controller 132 displays the surgical field image in each of the first display area 151 and the second display area 152 .

[0135] Specifically, Fig.10 As shown, similar to the first display area 151 , the surgical field image is also displayed in real time in the second display area 152 , and the characters “Live” indicating this fact are displayed in the upper left portion of the screen of the second display area 152 .

[0136] In step S13, the display controller 132 determines whether a freeze mode for freezing (stopping) the display in the first display area 151 has been selected based on the operation information indicating the operation of the user (consultant doctor). Specifically, it is determined whether the consultant doctor has performed an operation of touching the freeze mode button 312 displayed in the first display area 151.

[0137] Therefore, in step S13, when the consulting doctor performs an operation of touching the freeze mode button 312, the process proceeds to step S14.

[0138] In step S14 , the display controller 132 displays a still image of a single frame of the surgical field image displayed in real time in the first display area 151 instead of the surgical field image.

[0139] Specifically, Fig.11 As shown, in the first display area 151, a single frame of the surgical field image at the timing when the operation of touching the freeze mode button 312 is performed is displayed as a still image. In addition, the characters "pause" indicating that the still image is displayed are displayed at the upper left portion of the screen of the first display area 151. At this time, the display controller 132 holds the current position / posture information obtained from the image data generator 131 together with the still image at the timing when the operation of touching the freeze mode button 312 is performed.

[0140] In contrast, in step S13 , in a case where the consulting doctor does not perform an operation of touching the freeze mode button 312 , step S14 is skipped.

[0141] In step S15, the display controller 132 determines whether an annotation has been written to the surgical field image displayed in the first display area 151 based on the operation information indicating the operation of the user (consultant doctor). For example, the annotation is written by the consultant doctor performing an operation of tracing the area to be operated on the first display area 151 or another operation.

[0142] Here, when a still image is displayed in the first display area 151 , annotations are accepted to be written to the still image. In contrast, when a surgical field image (moving image) displayed in real time is displayed in the first display area 151 , annotations are accepted to be written to the moving image.

[0143] The process of step S15 is repeated until the comment has been written. When the comment is written, the process proceeds to step S16.

[0144] At this time, in response to the writing of the annotation, the designated position calculator 133 calculates the designated position based on the position / posture information obtained from the image data generator 131 and the three-dimensional surface mapping data stored in the three-dimensional surface mapping data memory 134 .

[0145] In step S16, the display controller 132 determines whether the position where the annotation has been written (specified position) is located in the area where the three-dimensional surface mapping data has been sufficiently constructed.

[0146] In step S16, in the case where it is determined that the position where the annotation has been written is located in the area where the three-dimensional surface mapping data has been sufficiently constructed, the process proceeds to step S17.

[0147] In step S17 , the first superimposing unit 141 superimposes the annotation on a designated position of the surgical field image displayed in the first display area 151 .

[0148] Then, in step S18 , the second superimposing unit 142 superimposes the annotation while tracking the designated position on the surgical field image displayed in the second display area 152 .

[0149] For example, in the case where the freeze mode has been selected, the annotation 331 is superimposed on the still image displayed in the first display area 151 at the designated position calculated by the designated position calculator 133, as shown in FIG. Fig.12 shown.

[0150] On the other hand, while tracking the designated position based on the position / posture information obtained from the image data generator 131 , an annotation 332 similar to the annotation 331 is superimposed on the designated position on the surgical field image displayed in real time in the second display area 152 .

[0151] exist Fig.12 In the example of FIG. 1 , in the surgical field image displayed in the second display area 152, the position and posture of the endoscope 11 have changed from the state in which the still image is displayed in the first display area 151, and the position and orientation of the organ captured in the image have changed. Even in this case, the annotation 332 is displayed while tracking the designated position on the surgical field image displayed in the second display area 152.

[0152] In addition, when the freeze mode has not been selected, the annotation 331 is superimposed on the designated position calculated by the designated position calculator 133 on the surgical field image displayed in real time in the first display area 151, as shown in FIG. Fig.13 shown.

[0153] Similarly, while tracking the designated position based on the position / posture information obtained from the image data generator 131 , an annotation 332 similar to the annotation 331 is superimposed on the designated position on the surgical field image displayed in real time in the second display area 152 .

[0154] Now return to step S16. In the case where it is determined that the position where the annotation has been written is not located in the area where the three-dimensional surface mapping data has been sufficiently constructed, the process proceeds to step S19.

[0155] In step S19 , the display controller 132 displays the still image on which the annotation has been superimposed in the first display area 151 .

[0156] Specifically, Fig.14 As shown, in a case where the freeze mode has been selected, the first superimposing unit 141 superimposes the annotation 331 on a designated position on the still image displayed in the first display area 151 .

[0157] In contrast, in the case where the freeze mode has not been selected, the display controller 132 displays a single frame at the timing when the annotation is written as a still image instead of the surgical field image displayed in the first display area 151. Then, the first superimposing unit 141 superimposes the annotation 331 on a designated position on the still image displayed in the first display area 151.

[0158] At this time, characters “SAVE” indicating that the still image on which the annotation 331 has been superimposed is retained are displayed in the upper left portion of the first display area 151 .

[0159] On the other hand, regardless of whether the freeze mode has been selected, the second superimposing unit 142 does not superimpose the annotation on the surgical field image displayed in the second display area 152 .

[0160] In the case where the three-dimensional surface mapping data has not been fully constructed in the area where the annotation has been written, the tracking accuracy of the annotation may be reduced. In view of this, by displaying a still image only when the annotation is written and not reflecting the annotation in the surgical field image displayed in real time, safety can be ensured without reducing the usability of the surgeon.

[0161] In the above-mentioned process, the writing of annotations by the consultant to the surgical field image displayed in the first display area 151 is reflected in the surgical field image displayed in the second display area 152 and being viewed by the surgeon. Therefore, the action of the consultant to write annotations to the surgical field image does not interfere with the surgeon who is performing the operation while confirming the surgical field image, and communication between operators can be improved.

[0162] Specifically, in the freeze mode, the consultant can freeze the surgical field image to easily write annotations, and the written annotations can be reflected in the surgical field image that the surgeon is watching. In other words, the consultant can write annotations at a more accurate position, and the surgeon can continuously observe the surgical field image that has reflected the annotations while correctly understanding the consultant's instructions.

[0163] Here, if Fig.15 As shown, in the surgical field image displayed in the second display area 152 and being viewed by the surgeon, the display / non-display (on / off state of superimposed display) of the annotation 332 can be switched according to the surgeon's operation. By doing so, the display of the annotation 332 itself can be suppressed from interfering with the progress of the surgery.

[0164] In addition, Fig.15 In addition to the example of FIG. 1 , in the surgical field image displayed in the first display area 151 , the on / off state of the superimposed display of the annotation 331 may be switched according to the operation of the consulting doctor.

[0165] Note that in the above description, the annotations written in the surgical field image displayed in the first display area 151 are reflected in the surgical field image displayed in the second display area 152, but a function opposite thereto may be provided. In other words, the annotations written in the surgical field image displayed in the second display area 152 may be reflected in the surgical field image displayed in the first display area 151.

[0166] By adopting the above-described configuration, for example, two surgeons can advance the operation while performing interactive communication such as mutual confirmation of the position to be operated on.

[0167] <3. Display Mode of Surgical Field Image>

[0168] Here, a display mode of the surgical field image displayed in the first display area 151 and the second display area 152 by the display controller 132 is described.

[0169] Fig.16 A first example of the display mode of the surgical field image is shown.

[0170] exist Fig.16 In the example of FIG. 3 , the first display area 351 and the second display area 352 are displayed side by side on the screen of a single display device 350 .

[0171] Fig.17 A second example of the display mode of the surgical field image is shown.

[0172] exist Fig.17 In the example of FIG. 3 , the first display area 351 and the second display area 352 are displayed in a picture-in-picture format on the screen of a single display device 350. Specifically, a second display area 352 smaller than the first display area 351 is set at the lower left portion of the first display area 351 displayed on the entire screen of the display device 350.

[0173] Fig.18 A third example of the display mode of the surgical field image is shown.

[0174] Fig.18 The example of indicates a state in which the first display area 351 and the second display area 352 are switched and displayed on the screen of the single display device 350 .

[0175] Fig.19 A fourth example of the display mode of the surgical field image is shown.

[0176] like Fig.19 As shown in the example of , the first display area 351 and the second display area 352 may be respectively displayed in separate display devices 350 - 1 and 350 - 2 connected to each other via a communication line.

[0177] By adopting Fig.19 With the configuration, the display devices 350-1 and 350-2 can be set at locations physically separated from each other, and this enables the consulting doctor to present annotations to a surgeon located in a remote place.

[0178] <4. Variants>

[0179] Modifications of the above-described embodiment are described below.

[0180] (Write annotations to past surgical field images)

[0181] In addition to writing annotations to a surgical field image displayed in real time or a still image of a single prescribed frame serving as a surgical field image, it is also acceptable to write annotations to past surgical field images within a prescribed period of time.

[0182] Fig. 20 An example of writing annotations to past surgical field images is shown.

[0183] exist Fig. 20 In the example, Fig.17 Similar to the example of , the first display area 351 and the second display area 352 are displayed on the screen of the single display device 350 in the form of picture-in-picture.

[0184] In addition, Fig. 20 In the example of FIG. 3 , for example, a search bar 370 is displayed on the right-hand side of the second display area 352, and the search bar 370 enables the time position of a prescribed time period (such as 20 seconds) in the past surgical field image to be specified. The right-hand end of the search bar 370 indicates the current time, and the left-hand end of the search bar 370 indicates 20 seconds before the current time. On the search bar 370, a slider 371 for specifying the time position of the past surgical field image and thumbnails 372 of frames in the specified time position in the surgical field image are displayed.

[0185] The user operates the slider 371 on the search bar 370 to designate a time position of 20 seconds past in the surgical field image. By doing so, a still image serving as a frame of the designated time position in the surgical field image is displayed in the first display area 351.

[0186] Then, the user writes annotations to the still image displayed in the first display area 351 , so that the annotations can be reflected in the current surgical field image displayed in real time in the second display area 352 .

[0187] As described above, annotations can also be written into past surgical field images.

[0188] (Mask Display)

[0189] In the above description, in the case where the position where the annotation has been written is not located in the area where the three-dimensional surface mapping data has been sufficiently constructed, the annotation is not reflected in the surgical field image displayed in real time.

[0190] The present disclosure is not limited thereto, and mask display may be performed in an area where three-dimensional surface mapping data has not been sufficiently constructed.

[0191] Fig.21 An example of a masked display is shown.

[0192] exist Fig.21 In the example, Fig.17 Similar to the example of , the first display area 351 and the second display area 352 are displayed on the screen of the display device 350 in the form of picture-in-picture. Fig.21 In the example of , it is assumed that the freeze mode has been selected, and a still image is displayed in the first display area 351 , and a real-time surgical field image is displayed in the second display area 352 .

[0193] Furthermore, a partial region of an organ captured in the still image displayed in the first display region 351 has been subjected to mask display 380. Writing of an annotation is not accepted in the region subjected to mask display 380.

[0194] By doing so, it is possible to suppress annotations from being written in a region where three-dimensional surface mapping data has not been sufficiently constructed, and therefore, a decrease in the tracking accuracy of the annotations can be avoided.

[0195] (Preview display of 3D image)

[0196] A preview of the three-dimensional image may be displayed so that, for example, the user can confirm whether an annotation that has been written by the user has been superimposed on an intended area.

[0197] Fig. 22 An example of a preview display of a three-dimensional image is shown.

[0198] exist Fig. 22 In the example of , the first display area 351 is displayed on the entire screen of the display device 350. Fig. 22 In the example of FIG. 35 , it is assumed that the freeze mode has been selected and a still image is displayed in the first display area 351 .

[0199] In addition, Fig. 22 In the example of FIG. 3 , at the right-hand end of the screen of the display device 350 , in addition to the write mode button 311 and the freeze mode button 312 , a three-dimensional image display button 391 is displayed.

[0200] When an annotation is written to the surgical field image displayed in the first display area 351 and an operation of touching the three-dimensional image display button 391 is performed, a three-dimensional image 392 based on three-dimensional information (three-dimensional surface mapping data) is displayed in the lower left portion of the first display area 351 .

[0201] In the three-dimensional image 392, the orientation and angle of the displayed object can be freely changed according to the user's operation. Fig. 22 In the example of FIG. 39 , in the three-dimensional image 392 , the annotation has been superimposed on a position corresponding to a specified position of the still image displayed in the first display area 351 (the position where the annotation has been written).

[0202] As described above, the three-dimensional image 392 on which the annotation has been superimposed is displayed so that the user can easily confirm whether the annotation that the user himself has written has been superimposed on the expected position.

[0203] (How to write comments)

[0204] In the above description, writing of an annotation (designation of a designated position) is achieved by performing an operation of touching the display device with a touch panel function, but writing of an annotation may be achieved by using another method.

[0205] For example, the coordinate position where the annotation is to be written on the screen may be input by using a mouse, or may be obtained by performing image recognition on an area designated by a surgical tool such as forceps captured on the screen.

[0206] Furthermore, an instruction to write an annotation may be issued by using voice, or an instruction to write an annotation may be issued by using the user's line of sight or orientation of the body.

[0207] Furthermore, the instruction to write a comment is not only issued by using a method for directly specifying a coordinate position on the screen.

[0208] For example, in the case where an instruction to write a comment of a designated area is issued, the boundary of the cancerous area 411 to be designated is automatically identified by surrounding the cancerous area 411 with the frame E1, as shown in FIG. Fig.23 Then, the annotation 412 indicating the cancerous region 411 is displayed, as shown in FIG. Fig.23 The right side is not.

[0209] Furthermore, annotations can be suppressed from being written by segmenting the three-dimensional surface mapping data into organs and masking the surface of each of the organs.

[0210] In the surgery support system according to the present embodiment, examples of organs with annotations written therein include various internal organs, blood vessels, tumors, lymphatic system, nerves, adipose tissue, abdominal wall, bleeding area, discolored area, etc. Therefore, for example, the holding position or puncture site of each of the above organs can also be displayed by using the writing of annotations.

[0211] (Example of displaying comments)

[0212] The annotation may be a point image indicating a point (designated position) designated by the user on the screen or a line image indicating a line used as a collection of points. Examples of the above-mentioned lines include a line segment connecting two points, a polyline configured by a plurality of line segments, a curve drawn by the user, a closed polyline, and a closed curve. In addition, the annotation may be a plane image indicating an area surrounded by a closed polyline or a closed curve. Examples of the plane image include a geometric pattern indicating an area having a volume in addition to the surface area of ​​an organ.

[0213] As described above, annotations can be displayed in different forms depending on the usage.

[0214] In addition, by using three-dimensional surface mapping data (three-dimensional information), annotations indicating the position and area of ​​the surface of the organ captured in the surgical field image and the position and area in the depth direction of the organ (stereoscopic area) can be written (superimposed). Three-dimensional information is associated with each of these positions and areas, and thus the distance between points or the area / volume of the area can be virtually calculated without directly touching the organ.

[0215] For example, in Fig.24 In the example of , a line image connecting point images 421 and 422 and a plane image 423 of the surface of an organ captured in the surgical field image are superimposed on the surgical field image as annotations. Fig.24 In the example of FIG. 4 , the distance between the point image 421 and the point image 422 is calculated to be 42 mm, and the area of ​​the plane image 423 is calculated to be 200 mm 2 .

[0216] Note that a closed curve 424 that surrounds the plane image 423 and is spaced apart from the boundary of the plane image 423 by a specified distance (eg, 12 mm) may be superimposed on the surgical field image as an annotation, as shown in FIG. Fig.24 shown.

[0217] In addition, Fig.25 In the example of FIG. 4 , a stereoscopic image 432 having a volume in the depth direction of a region 431 on the surface of an organ captured in the surgical field image and a point image 433 on the region 431 are superimposed on the surgical field image as annotations. Fig.25 In the example, the volume of the stereoscopic image 432 is calculated to be 32526 mm3 , and the distance between the point image 433 and the stereoscopic image 432 is calculated to be 25 mm.

[0218] Furthermore, by changing the color or density of a point image or a plane image according to the distance from the observation optical system of the endoscope 11, perspective can be given to the point image or the plane image displayed as an annotation, so that the user can intuitively grasp the three-dimensional position of the annotation.

[0219] In addition, an example has been described above in which a single user (e.g., a consultant doctor) writes annotations to a surgical field image displayed in a single display area (first display area 151). The present disclosure is not limited to this, and multiple consultant doctors can write annotations to a surgical field image displayed in a single display area. In addition, a display area can be set for each of multiple consultant doctors, and each of the multiple consultant doctors can write annotations to a surgical field image displayed in each of the multiple display areas.

[0220] In this case, for example, annotations having display forms different from each other can be superimposed by changing the color or shape according to each of the plurality of consultant doctors who have written the annotations (each user who has designated a designated position on the surgical field image). Fig.26 As shown, the annotation 331 - 1 that has been written by the first consulting doctor is displayed with a solid line, and the annotation 331 - 2 that has been written by the second consulting doctor is displayed with a dotted line.

[0221] By doing so, it is possible to discern who has written the annotation that has been superimposed on the surgical field image.

[0222] Furthermore, in the above-described example, from among comments written by a plurality of users, comments written by a specified user can be reset (deleted).

[0223] In addition, an image or three-dimensional information that has been obtained by another method (a device that takes medical images, such as a computed tomography (CT) device or a magnetic resonance imaging (MRI) device) can be superimposed on the surgical field image. By doing so, the annotation can be displayed as if it were written inside the organ.

[0224] In addition, by displaying various types of information related to an area in the position of the starting point of writing as an annotation, when the user himself or other users confirm the area at a later time, useful information can be presented to the user himself or other users, or can attract the attention of the user himself or other users.

[0225] (Write annotations into recorded surgical field images)

[0226] In the above description, annotations are written to the surgical field image captured by the sensing unit 110 and displayed in real time. The present disclosure is not limited to this, and the surgical field image captured by the sensing unit 110 can be recorded, and annotations can be written to the recorded surgical field image. The above form can be used in medical education venues, etc.

[0227] (Obtaining Absolute Three-Dimensional Information)

[0228] In this embodiment, relative three-dimensional information is obtained based on the camera position / posture obtained in motion estimation. However, absolute three-dimensional information using the outside of the body cavity of the patient to be observed as a reference can be obtained. Specifically, an optical and magnetic tracking system having an image recognition function and including a reference coordinate system outside the body cavity of the patient is set in the endoscope 11, and absolute three-dimensional information is obtained by using this coordinate system as a reference. This makes it possible to present the absolute three-dimensional position of the annotation.

[0229] (Use of machine learning)

[0230] In recent years, it has become possible to automatically detect tumors and the like by using AI (such as machine learning or deep learning), and the results of the above detection can be used as annotations. By doing so, annotations can be continuously superimposed and displayed in the position or area of ​​the tumor that has been detected in a specific frame of the surgical field image. In this case as well, by enabling the on / off state of the superimposed display of the annotation to be switched, the display of the annotation itself can be suppressed from hindering the progress of the surgery.

[0231] <5. Application>

[0232] Next, refer to Fig. 27 , an example of a case where a video microscope apparatus for surgery including an arm is used is described as another application of the surgery assisting system according to the present embodiment.

[0233] Fig. 27 An example of a microscope surgery system using a video microscope device for surgery used as a medical device for observation for observing the inside of a patient's body is shown.

[0234] Fig. 27 A state is shown in which a doctor serving as an operator (user) 520 performs surgery on an object to be operated (patient) 540 on an operating table 530 by using a tool 521 for surgery such as a scalpel, tweezers, or forceps.

[0235] Note that in the following description, it is assumed that operation is a general term for various types of medical treatments (such as surgery and examination) performed by a doctor serving as a user 520 on a patient serving as an object to be operated 540. In addition, Fig. 27The example of indicates the state of surgery as an example of operation. However, the operation using the video microscope device 510 for surgery is not limited to surgery, and may be various other operations.

[0236] On the side of the operating table 530, a video microscope device 510 for surgery according to the present embodiment is provided.

[0237] The video microscope apparatus 510 for surgery includes a base unit 511 serving as a base, an arm unit 512 extending from the base unit 511 , and an imaging unit 515 connected to an end of the arm unit 512 as an end unit.

[0238] The arm unit 512 includes a plurality of joint units 513 a , 513 b , and 513 c , a plurality of links 514 a and 514 b coupled through the joint units 513 a and 513 b , and an imaging unit 515 provided at an end of the arm unit 512 .

[0239] exist Fig. 27 In the example, for simplicity, the arm unit 512 includes three joint units 513a to 513c and two connecting rods 514a and 514b. In fact, considering the degree of freedom of the position and posture of the arm unit 512 and the imaging unit 515, the number and shape of the joint units 513a to 513c and the connecting rods 514a and 514b, the direction of the driving axis of the joint units 513a to 513c, etc. can be appropriately set to achieve the desired degree of freedom.

[0240] The joint units 513 a to 513 c have a function of rotatably coupling the links 514 a and 514 b to each other, and drive the rotation of the joint units 513 a to 513 c , thereby controlling the driving of the arm unit 512 .

[0241] The imaging unit 515 is connected to an end of the arm unit 512 as an end unit.

[0242] The imaging unit 515 is a unit that captures an image of an object to be imaged by including an optical system that captures an optical image of the object, and is configured as a camera capable of capturing moving images, still images, and the like, for example. Fig. 27 As shown, the posture and position of the arm unit 512 and the imaging unit 515 are controlled by the video microscope device 510 for surgery, so that the imaging unit 515 set at the end of the arm unit 512 images the state of the area to be operated of the object 540 to be operated.

[0243] Note that the configuration of the imaging unit 515 connected to the end of the arm unit 512 as an end unit is not particularly limited, and the imaging unit 515 may be configured as an endoscope or a microscope, for example. In addition, the imaging unit 515 may be configured to be attachable to or detachable from the arm unit 512.

[0244] By adopting the above configuration, for example, the imaging unit 515 according to the purpose can be appropriately connected as an end unit to the end of the arm unit 512. Note that the description here focuses on the case where the imaging unit 515 is adopted as the end unit, but it goes without saying that the end unit connected to the end of the arm unit 512 is not limited to the imaging unit 515.

[0245] In addition, a display device 550 such as a monitor or a display is provided at a position facing the user 520. For example, an image processing device incorporated in or externally attached to the video microscope apparatus 510 for surgery performs various types of image processing on an image of the area to be operated that has been captured by the imaging unit 515, and displays the image on a display screen of the display device 550 as an electronic image.

[0246] By adopting the above-described configuration, the user 520 can perform various types of processing (such as surgery) while viewing the electronic image of the area to be operated displayed on the display screen of the display device 550 .

[0247] Here, in Fig. 27 In the example of FIG. 5 , the imaging unit 515 includes, for example, a reference Figure 3 In addition, the image processing device that performs various types of image processing on the image of the area to be operated that has been captured by the imaging unit 515 is similar to the image processing device of the reference Figure 3 Similarly, the display device 550 corresponds to the example of the information processing device 130 described in reference to FIG. Figure 3 The examples of the display device 150 described above correspond to each other.

[0248] <6. Hardware Configuration>

[0249] Next, refer to Fig.28 An example of the hardware configuration of the information processing device configuring the surgery support system according to the present embodiment is described in detail.

[0250] Fig.28 is a block diagram showing an example of the hardware configuration of the information processing apparatus 900 configuring the surgery support system according to the present embodiment.

[0251] like Fig.28As shown, the information processing apparatus 900 includes a CPU 901, a ROM 903, and a RAM 905. In addition, the information processing apparatus 900 includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, and a storage device 919. Note that the information processing apparatus 900 may include a drive 921, a connection port 923, and a communication device 925.

[0252] The CPU 901 functions as an arithmetic processing device and a controller, and controls the whole or part of operations in the information processing device 900 according to various programs recorded in the ROM 903 , the RAM 905 , the storage device 919 , or the removable recording medium 927 .

[0253] The ROM 903 stores programs used by the CPU 901, operating parameters, and the like. The RAM 905 temporarily stores programs to be used by the CPU 901, parameters that change appropriately in the execution of the programs, and the like. The ROM 903 and the RAM 905 are connected to each other via a host bus 907 configured by an internal bus such as a CPU bus. Figure 3 The described respective configurations of the information processing apparatus 130 are realized by, for example, the CPU 901 .

[0254] The host bus 907 is connected to an external bus 911 such as a peripheral component interconnect / interface (PCI) bus via a bridge 909. An input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925 are connected to the external bus 911 via an interface 913.

[0255] The input device 915 is an operation unit operated by the user, such as a mouse, keyboard, touch panel, button, switch, joystick, or pedal. In addition, the input device 915 may be, for example, a remote control unit (referred to as a remote controller) using infrared or other radio waves, or may be an external connection device 929 corresponding to an operation performed on the information processing device 900, such as a mobile phone or a PDA.

[0256] The input device 915 is configured by, for example, an input control circuit or the like which generates an input signal based on information input by a user using the above-described operation unit and outputs the input signal to the CPU 901 .

[0257] The user operates the input device 915 so as to be able to input various types of data to the information processing device 900 or issue an instruction to the information processing device 900 to perform a processing operation.

[0258] The output device 917 is configured by a device capable of visually or auditorily reporting the obtained information to the user. Specifically, the output device 917 is configured as a CRT display device, a liquid crystal display device, a plasma display device, an EL display device, a display device such as a lamp, a sound output device such as a speaker or headphones, a printer device, etc.

[0259] The output device 917 outputs, for example, the results obtained by the information processing device 900 performing various types of processing. Specifically, the display device displays the results obtained by the information processing device 900 performing various types of processing in the form of text or images. On the other hand, the sound output device converts an audio signal configured by reproduced sound data, acoustic data, etc. into an analog signal and outputs the analog signal. Note that, referring to Figure 3 The described display device 150 is realized by the output device 917 , for example.

[0260] The storage device 919 is a device for data storage configured as an example of storage of the information processing device 900. The storage device 919 is configured by, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc. The storage device 919 stores programs to be executed by the CPU 901, various types of data, and the like.

[0261] The drive 921 is a reader / writer for a recording medium, and is incorporated into the information processing apparatus 900 or is externally attached to the information processing apparatus 900. The drive 921 reads information recorded in an attached removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 905. Furthermore, the drive 921 can write records to the attached removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0262] The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, etc. In addition, the removable recording medium 927 may be a CompactFlash (registered trademark) (CF), a flash memory, a secure digital (SD) memory card, etc. In addition, the removable recording medium 927 may be, for example, an integrated circuit (IC) card, an electronic device, etc., in which a contactless IC chip is mounted.

[0263] The connection port 923 is a port for directly connecting an external connection device 929 to the information processing apparatus 900. Examples of the connection port 923 include a universal serial bus (USB) port, an IEEE 1394 port, a small computer system interface (SCSI) port, etc. Other examples of the connection port 923 include an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) (registered trademark) port, etc. By connecting the external connection device 929 to the connection port 923, the information processing apparatus 900 directly obtains various types of data from the external connection device 929, or provides various types of data to the external connection device 929.

[0264] The communication device 925 is, for example, a communication interface configured by the communication device for connecting to a communication network (network) 931, etc. The communication device 925 is, for example, a wired or wireless local area network (LAN), Bluetooth (registered trademark), a communication card for wireless USB (WUSB), etc. In addition, the communication device 925 may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, etc.

[0265] For example, the communication device 925 can transmit / receive signals to / from, for example, the Internet or other communication devices according to a prescribed protocol such as TCP / IP. In addition, the communication network 931 connected to the communication device 925 can be configured by a network connected by wire or wirelessly, etc. The communication network 931 may be, for example, the Internet or an indoor LAN, or may be a communication network in which infrared communication, radio wave communication, or satellite communication is performed.

[0266] Each of the above components of the information processing device 900 may be configured using a general-purpose part, or may be configured by hardware dedicated to the function of each of the components. Therefore, the hardware configuration to be used can be appropriately changed according to the technical level when implementing the present embodiment.

[0267] In addition, a computer program for realizing each of the functions of the information processing device 900 configuring the surgical assisting system according to the present embodiment may be generated, and the computer program may be installed on a personal computer or the like. In addition, a computer-readable recording medium storing the above-mentioned computer program may also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the computer program may be distributed, for example, via a network without using a recording medium.

[0268] <7. Conclusion>

[0269] As described above, in the surgical assistance system according to an embodiment of the present disclosure, annotations are superimposed on a designated position already designated by a user on a surgical field image displayed in a first display area. Then, the annotations are superimposed while tracking the designated position on the surgical field image displayed in real time in a second display area.

[0270] By doing so, writing of annotations that has been performed by the consulting physician on the surgical field image displayed in the first display area is reflected in the surgical field image that is displayed in the second display area and is being viewed by the surgeon.

[0271] In the related art, endoscopic surgery or surgery using a microscope is surgery via a screen, and thus it is difficult to verbally convey detailed instructions related to the technique between operators.

[0272] In view of this, by adopting a surgery assisting system according to another embodiment of the present disclosure, consultant doctors, surgeons, assistants, and nursing staff in leading positions can collaborate with each other using annotated virtual markers.

[0273] Furthermore, by employing the surgery assisting system according to still another embodiment of the present disclosure, a mark may be placed on a surgical field image for educational purposes, or a mark may be placed on a recorded surgical field image at a later time.

[0274] In the above virtual markers, invasive markers are not placed directly on the organ, or instructions are not issued using forceps, etc., and therefore the organ is not damaged. In addition, when establishing a surgical plan, a strategy can be determined while placing markers, or in the case where a marker is unnecessary, a new marker can be placed.

[0275] Note that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present disclosure.

[0276] For example, the present disclosure may adopt a configuration of cloud computing in which a plurality of devices perform processing in cooperation with each other by sharing a single function via a network.

[0277] Furthermore, the processing of the respective steps described above with reference to the flowcharts may be performed by a single device, or may be shared and performed by a plurality of devices.

[0278] Furthermore, in the case where a single step includes a plurality of processes, the plurality of processes included in the single step may be executed by a single device, or may be shared and executed by a plurality of devices.

[0279] Furthermore, the present disclosure can adopt the configurations described below. (1)

[0281] A surgical assisting device, comprising:

[0282] circuit, is configured as

[0283] generating a first image to be displayed in a first display area and including a first surgical field image of a surgical subject and a first visual annotation image superimposed on a predetermined area corresponding to a body area of ​​the surgical subject in the surgical field of the first surgical field image, and

[0284] A second image is generated, which will be displayed in the second display area and includes a second surgical field image and a second visual annotation image superimposed on an estimated region of the second surgical field image corresponding to a body region of the surgical subject. (2)

[0286] The surgical assisting device according to (1), wherein the circuit is further configured to:

[0287] generating three-dimensional information indicating a three-dimensional structure of a surgical object;

[0288] estimating a position of an imager that captures a second surgical field image of the surgical object; and

[0289] A second image is generated by superimposing a second visual annotation image on a portion of the three-dimensional structure of the surgical object corresponding to the estimated region and estimated to correspond to the body region of the surgical object based on the estimated motion of the imager and the generated three-dimensional information. (3)

[0291] The surgical assisting device according to (2), wherein:

[0292] The generated three-dimensional information of the surgical object includes time information indicating the generation / update time of the corresponding three-dimensional information. (4)

[0294] The surgical assisting device according to (1), wherein:

[0295] The first visual annotation image is input by an input device configured to be used by a first user to input the first visual annotation image on a first display area without obscuring a second user's view of a second display area. (5)

[0297] The surgical assisting device according to (4), wherein:

[0298] an input device for inputting a first visual annotation image configured to be used by a first user to identify a body region of a surgical subject as a target for a surgical action to be performed on the surgical subject; and

[0299] The second visual annotation image in the second display area is configured to be used by a second user to control the performance of a surgical action on the body region of the surgical subject. (6)

[0301] The surgical assisting device according to (1), wherein:

[0302] The first display area is displayed on the first display device; and

[0303] The second display area is displayed on the second display device. (7)

[0305] The surgical assisting device according to (1), wherein:

[0306] The first display area is displayed in a first window area of ​​the display; and

[0307] The second display area is displayed in the second window area of ​​the display. (8)

[0309] The surgical assisting device according to (2), wherein:

[0310] The imager is included in an endoscope that captures a second surgical field image as a moving image of the interior of a surgical object. (9)

[0312] The surgical assisting device according to (8), wherein the circuit is further configured to:

[0313] generating a first image as a still image obtained from a moving image of the interior of the surgical object according to a freeze mode input from the user interface device; and

[0314] Information associating the first visual annotation image with a body region of a surgical subject is received according to a writing mode input from a user interface device. (10)

[0316] A surgical assistance device according to (2), wherein the circuit is further configured to update the second image during movement of the imager or movement of the body region of the surgical object to maintain correspondence between the second visual annotation image superimposed on the estimated area of ​​the second surgical field image and the body region of the surgical object. (11)

[0318] The surgical assisting device according to (1), wherein the circuit includes a programmable processor. (12)

[0320] The surgery assisting apparatus according to (1), wherein the second surgical field image includes a motion image of the surgical object indicating the motion of the surgical object in real time. (13)

[0322] The surgical assisting apparatus according to (12), wherein the circuit is further configured to obtain the first surgical field image from a moving image of the surgical object. (14)

[0324] The surgical assisting apparatus according to (1), wherein the circuit is further configured to receive information about the first visual annotation image from an input device operated using the first display area. (15)

[0326] A surgical assistance device according to (1), wherein the second surgical field of view image is a motion image indicating at least the movement of the surgical object, and the second visual annotation image moves with the movement of at least the surgical object to maintain the correspondence between the estimated area of ​​the second surgical field of view image and the body area of ​​the surgical object. (16)

[0328] The surgical assisting device according to (2), wherein the circuit is further configured to:

[0329] A second visual annotation image including an indication of a position in the depth direction of the second surgical field image is generated based on the generated three-dimensional information. (17)

[0331] The surgical assisting device according to (2), wherein the circuit is further configured to:

[0332] The mask display is performed on the area of ​​the first image where the generated three-dimensional information has not yet been fully constructed. (18)

[0334] The surgical assisting device according to (17), wherein the circuit is further configured to:

[0335] When the estimated region of the second surgical field image corresponding to the body region of the surgical subject includes a region of the first image where the generated three-dimensional information has not been sufficiently constructed, the second image is generated without superimposing at least a portion of the second visual annotation image. (19)

[0337] The surgical assisting device according to (18), wherein the circuit is further configured to:

[0338] In a case where the estimated region of the second surgical field image corresponding to the body region of the surgical subject includes a region of the first image where the generated three-dimensional information has not been sufficiently constructed, the first image is displayed as a still image instead of a moving image. (20)

[0340] A surgical procedure comprising:

[0341] generating a first image to be displayed in a first display area and including a first surgical field image of a surgical subject and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical subject; and

[0342] A second image is generated, which will be displayed in the second display area and includes a second surgical field image and a second visual annotation image superimposed on an estimated region of the second surgical field image corresponding to a body region of the surgical subject. (twenty one)

[0344] A non-transitory computer-readable medium storing instructions, which, when executed by a computer, cause the computer to perform the following steps, the steps comprising:

[0345] generating a first image to be displayed in a first display area and including a first surgical field image of a surgical subject and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical subject; and

[0346] A second image is generated, which will be displayed in the second display area and includes a second surgical field image and a second visual annotation image superimposed on an estimated region of the second surgical field image corresponding to a body region of the surgical subject. (twenty two)

[0348] A surgical assistance system, comprising:

[0349] an endoscope including an imager for capturing an image of a second surgical field of view of a surgical object;

[0350] circuit, is configured as

[0351] generating a first image to be displayed in a first display area and including a first surgical field image of a surgical subject and a first visual annotation image superimposed on a predetermined area of ​​the first surgical field image corresponding to a body area of ​​the surgical subject, and

[0352] generating a second image to be displayed in the second display area and including a second surgical field image and a second visual annotation image superimposed on an estimated region of the second surgical field image corresponding to a body region of the surgical subject; and

[0353] The surgical tool is moved inside the surgical object under the control of the medical staff based on the second visual annotation image in the second image to perform surgical treatment on the surgical object. (twenty three)

[0355] A surgical assistance system, comprising:

[0356] a display controller that performs control to display a surgical field image in two or more display areas;

[0357] a first superimposing unit that superimposes the visual information on a designated position designated by a user on the surgical field image displayed in a first display area of ​​the two or more display areas; and

[0358] The second superimposing unit superimposes the visual information while tracking a designated position on the surgical field image displayed in real time in a second display area different from the first display area. (twenty four)

[0360] A surgical assistance system according to (23), wherein:

[0361] The display controller displays a still image instead of the surgical field image displayed in real time in the first display area based on an instruction of a user, the still image being a single prescribed frame of the surgical field image, and

[0362] The first superimposing unit superimposes the visual information on a designated position designated by the user on the still image displayed in the first display area. (25)

[0364] A surgical assistance system according to (23), wherein:

[0365] The first superimposing unit superimposes the visual information on a designated position designated by the user on the surgical field image displayed in real time in the first display area. (26)

[0367] A surgical assistance system according to (23), wherein:

[0368] The display controller displays the surgical field image for a prescribed period of time in the past in the first display area, and

[0369] The first superimposing unit superimposes the visual information on a designated position designated by the user on the past surgical field image displayed in the first display area. (27)

[0371] A surgical assisting system according to any one of (23) to (26), wherein:

[0372] The display controller displays the first display area and the second display area side by side on the single display device. (28)

[0374] A surgical assisting system according to any one of (23) to (26), wherein:

[0375] The display controller displays the first display area and the second display area in a picture-in-picture format on a single display device. (29)

[0377] A surgical assisting system according to any one of (23) to (26), wherein:

[0378] The display controller switches and displays the first display area and the second display area on the single display device. (30)

[0380] A surgical assisting system according to any one of (23) to (26), wherein:

[0381] The display controller displays the first display area and the second display area on the separate display devices. (31)

[0383] A surgical assisting system according to any one of (23) to (30), wherein:

[0384] The display controller switches display and non-display of the visual information that has been superimposed on the surgical field image displayed at least in the second display area based on an instruction of a user. (32)

[0386] A surgical assisting system according to any one of (23) to (31), wherein:

[0387] The first superimposing unit and the second superimposing unit superimpose visual information in a display form that is different according to a user who has designated a designated position on the surgical field image. (33)

[0389] A surgical assisting system according to any one of (23) to (32), wherein:

[0390] The first superimposing unit and the second superimposing unit superimpose at least one of a point image, a line image, and a plane image as visual information according to an operation of a user. (34)

[0392] A surgical assistance system according to (33), wherein:

[0393] The first superimposing unit and the second superimposing unit superimpose visual information indicating the position in the depth direction of the surgical field image based on the three-dimensional information that has been constructed for the surgical field image. (35)

[0395] A surgical assisting system according to any one of (23) to (34), wherein:

[0396] The display controller performs mask display on a region in which three-dimensional information has not been sufficiently constructed in the surgical field image displayed in the first display region. (36)

[0398] A surgical assisting system according to any one of (23) to (34), wherein:

[0399] When the user designates a designated position on the surgical field image displayed in the first display area, the display controller displays a preview of a three-dimensional image in which visual information has been superimposed on a position corresponding to the designated position on the surgical field image. (37)

[0401] A surgical assisting system according to any one of (23) to (34), wherein:

[0402] When the designated position is designated on a region in which three-dimensional information has not been sufficiently constructed in the surgical field image displayed in the first display region, the second superimposing unit does not superimpose the visual information on the still image displayed in the second display region. (38)

[0404] A surgical assistance system according to (37), wherein:

[0405] When a designated position is designated in a region of the surgical field image displayed in the first display region where three-dimensional information has not yet been fully constructed,

[0406] The display controller displays a still image of a single prescribed frame serving as the surgical field image instead of the surgical field image displayed in real time in the first display area, and

[0407] The first superimposing unit superimposes the visual information on a designated position designated by the user on the still image displayed in the first display area. (39)

[0409] A surgical assisting system according to any one of (23) to (38), wherein:

[0410] The surgical field image is an image captured by a medical device for observation that observes the inside of a patient's body. (40)

[0412] A surgical assistance system according to (39), wherein:

[0413] The medical device used for observation is an endoscope that captures images by using the inside of a patient's body as an object. (41)

[0415] A surgical assistance system according to (39), wherein:

[0416] The medical device used for observation is a microscope including an optical system that captures an optical image of an object. (42)

[0418] A display method performed by a surgery assisting system, the display method comprising:

[0419] performing control to display a surgical field image in two or more display areas;

[0420] superimposing the visual information on a designated position designated by a user on the surgical field image displayed in a first display area of ​​the two or more display areas; and

[0421] Visual information is superimposed while tracking a designated position on a surgical field image displayed in real time in a second display area different from the first display area.

[0422] Reference Mark List

[0423] 1. Surgical Assistance System

[0424] 11. Endoscope

[0425] 13 CCU

[0426] 15 Display device

[0427] 100 Surgical Assistance Systems

[0428] 110 Sensing unit

[0429] 130 Information processing device

[0430] 131 Image Data Generator

[0431] 132 Display Controller

[0432] 133 Specified location calculator

[0433] 134 3D surface mapping data storage

[0434] 141 First superposition unit

[0435] 142 Second superposition unit

[0436] 150 Display device

[0437] 151 First display area

[0438] 152 Second display area.

Claims

1. A surgical assisting device, comprising: circuit, is configured as generating a first image which is displayed in a first display area and includes a still first surgical field image of a surgical object and a first visual annotation image superimposed on a designated position in the surgical field of the first surgical field image which is designated by a user and corresponds to a body region of the surgical object, and generating a second image that will be displayed in a second display area and includes a real-time second surgical field image and a second visual annotation image that is similar to the first visual annotation image and is superimposed on the second surgical field image while tracking the designated position corresponding to the body region of the surgical object in the second surgical field image, wherein the static first surgical field image is a single frame of the real-time second surgical field image, and Wherein, the circuit is further configured as follows: generating three-dimensional information indicating the three-dimensional structure of the surgical object, and The mask display is performed on the area of ​​the first image where the generated three-dimensional information has not been fully constructed.

2. The surgical assisting device according to claim 1, wherein: The circuit is further configured to: estimating a position of an imager that captures the second surgical field image of the surgical object; and The second image is generated by superimposing the second visual annotation image on a portion of the three-dimensional structure of the surgical object corresponding to an estimated region and estimated to correspond to the body region of the surgical object based on the estimated motion of the imager and the generated three-dimensional information.

3. The surgical assisting device according to claim 2, wherein: The generated three-dimensional information of the surgical object includes time information indicating a generation / update time of the corresponding three-dimensional information.

4. The surgical assisting device according to claim 1, wherein: The first visual annotation image is input by an input device configured to be used by a first user to input the first visual annotation image on the first display area without obscuring a second user's view of the second display area.

5. The surgical assisting device according to claim 4, wherein: the input device for inputting the first visual annotation image is configured to be used by the first user to identify the body region of the surgical subject as a target for a surgical action to be performed on the surgical subject; and The second visual annotation image in the second display area is configured to be used by the second user to control performance of the surgical action on the body region of the surgical subject.

6. The surgical assisting device according to claim 1, wherein: The first display area is displayed on a first display device; and The second display area is displayed on a second display device.

7. The surgical assisting device according to claim 1, wherein: The first display area is displayed in a first window area of ​​the display; and The second display area is displayed in a second window area of ​​the display.

8. The surgical assisting device according to claim 2, wherein: The imager is included in an endoscope that captures the second surgical field image as a moving image of the interior of the surgical object.

9. The surgical assisting device according to claim 8, wherein: The circuit is further configured to: generating the first image as a still image obtained from the moving image of the interior of the surgical object according to a freeze mode input from a user interface device; and Information associating the first visual annotation image with the body region of the surgical subject is received according to a writing mode input from the user interface device.

10. The surgical assisting device according to claim 2, wherein: The circuit is further configured to update the second image during movement of the imager or movement of the body region of the surgical subject to maintain correspondence between the second visual annotation image superimposed on the estimated area of ​​the second surgical field image and the body region of the surgical subject.

11. The surgical assisting device according to claim 1, wherein: The circuit includes a programmable processor.

12. The surgical assisting device according to claim 1, wherein: The second surgical field image includes a motion image of the surgical object indicating motion of the surgical object in real time.

13. The surgical assisting device according to claim 12, wherein: The circuit is further configured to obtain the first surgical field image from the motion image of the surgical object.

14. The surgical assisting device according to claim 1, wherein: The circuitry is further configured to receive information about the first visual annotation image from an input device operated using the first display area.

15. The surgical assisting device according to claim 1, wherein: The second surgical field image is a moving image indicating at least movement of the surgical object, and the second visual annotation image moves with the movement of at least the surgical object to maintain correspondence between an estimated region of the second surgical field image and the body region of the surgical object.

16. The surgical assisting device according to claim 2, wherein: The circuit is further configured to: The second visual annotation image including an indication of a position in the depth direction of the second surgical field image is generated based on the generated three-dimensional information.

17. The surgical assisting device according to claim 2, wherein: The circuit is further configured to: When the estimated region of the second surgical field image corresponding to the body region of the surgical object includes a region of the first image where the generated three-dimensional information has not been sufficiently constructed, the second image is generated without superimposing at least a portion of the second visual annotation image.

18. The surgical assisting device according to claim 17, wherein: The circuit is further configured to: In a case where the estimated region of the second surgical field image corresponding to the body region of the surgical object includes a region of the first image where generated three-dimensional information has not been sufficiently constructed, the first image is displayed as a still image rather than a moving image.

19. A surgical method comprising: generating a first image which will be displayed in a first display area and includes a still first surgical field image of a surgical object and a first visual annotation image superimposed on a designated position designated by a user and corresponding to a body region of the surgical object in the surgical field of the first surgical field image; and generating a second image that will be displayed in a second display area and includes a real-time second surgical field image and a second visual annotation image that is similar to the first visual annotation image and is superimposed on the second surgical field image while tracking the designated position corresponding to the body region of the surgical object in the second surgical field image, wherein the static first surgical field image is a single frame of the real-time second surgical field image, and Wherein, the surgical method further comprises: generating three-dimensional information indicating the three-dimensional structure of the surgical object, and The mask display is performed on the area of ​​the first image where the generated three-dimensional information has not been fully constructed.

20. A non-transitory computer readable medium storing instructions, which, when executed by a computer, cause the computer to perform the following steps, the steps comprising: generating a first image which will be displayed in a first display area and includes a still first surgical field image of a surgical object and a first visual annotation image superimposed on a designated position designated by a user and corresponding to a body region of the surgical object in the surgical field of the first surgical field image; and generating a second image that will be displayed in a second display area and includes a real-time second surgical field image and a second visual annotation image that is similar to the first visual annotation image and is superimposed on the second surgical field image while tracking the designated position corresponding to the body region of the surgical object in the second surgical field image, wherein the static first surgical field image is a single frame of the real-time second surgical field image, and Wherein, the steps further include: generating three-dimensional information indicating the three-dimensional structure of the surgical object, and The mask display is performed on the area of ​​the first image where the generated three-dimensional information has not been fully constructed.

21. A surgical assistance system, comprising: an endoscope including an imager for capturing an image of a second surgical field of view of a surgical object; circuit, is configured as generating a first image which is displayed in a first display area and includes a still first surgical field image of the surgical object and a first visual annotation image superimposed on a designated position in the surgical field of the first surgical field image which is designated by a user and corresponds to a body region of the surgical object, and generating a second image to be displayed in a second display area and including the second surgical field image in real time and a second visual annotation image that is similar to the first visual annotation image and is superimposed on the second surgical field image while tracking the designated position corresponding to the body region of the surgical object in the second surgical field image, wherein the stationary first surgical field image is a single frame of the second surgical field image in real time; and a surgical tool, based on the second visual annotation image in the second image, moved inside the surgical object under the control of a medical staff to perform surgical treatment on the surgical object, Wherein, the circuit is further configured as follows: generating three-dimensional information indicating the three-dimensional structure of the surgical object, and The mask display is performed on the area of ​​the first image where the generated three-dimensional information has not been fully constructed.

Images