Information processor, method and program

Abstract

To improve usability of a user interface for providing a position and a range of an organ within an operation image.SOLUTION: An information processor according to one embodiment of the present invention comprises: an operation image acquisition part that acquires an operation image from an operation image; a position and range estimation part that estimates a position and a range of a structure from the operation image; and a position and range display control part that controls display and non-display of the estimated position and range of the structure.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] Conventionally, there is known a technology for displaying the positions and extents of organs in an image of a patient undergoing surgery (hereinafter also referred to as a surgical image). By checking the displayed positions and extents of organs, a doctor can avoid high-risk actions (for example, actions that may cause postoperative complications) and determine whether the removal of a malignant tumor and the lymph nodes surrounding the tumor has been completed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7031925 Summary of the Invention [Problem to be solved by the invention]

[0004] However, doctors are unable to display the positions and extents of organs in surgical images in the manner they desire, and there is a need for a user interface that is easier to use.

[0005] Therefore, an object of the present invention is to improve the ease of use of a user interface that provides the location and extent of organs within a surgical image. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes a surgical image acquisition unit that acquires surgical images from surgical footage, a position and range estimation unit that estimates the position and range of a structure from the surgical images, and a position and range display control unit that controls the display or non-display of the estimated position and range of the structure. [Effects of the Invention]

[0007] The present invention provides an improved usability of a user interface that provides the location and extent of organs within a surgical image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overall configuration according to an embodiment of the present invention; [Figure 2] 1 is a functional block diagram of an information processing apparatus according to an embodiment of the present invention; [Figure 3] 10 is a flowchart of a control process for displaying and hiding the position and range of a structure according to an embodiment of the present invention. [Figure 4] 10 is an example of a screen according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams for explaining pre-processing of a surgical video according to an embodiment of the present invention. [Figure 6] 10 is an example of a screen according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram for explaining color shades (presets) of organs according to one embodiment of the present invention. [Figure 8] FIG. 10 is a diagram for explaining an area for setting the display of the position and range of a structure according to one embodiment of the present invention. [Figure 9] 10 is an example of a screen according to an embodiment of the present invention. [Figure 10] 10 is an example of a screen according to an embodiment of the present invention. [Figure 11] 10 is an example of a screen according to an embodiment of the present invention. [Figure 12] 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Terminology> As used herein, "surgery" refers to any type of surgery, including surgery performed solely by humans, surgery performed by humans using medical instruments such as endoscopes (e.g., laparoscopic surgery, thoracoscopic surgery, etc.), surgery performed by humans using robots, and surgery performed solely by robots. For example, surgery is a surgical procedure to remove cancerous tissue, including the primary tumor and lymph nodes, from tissue. Examples of surgery include esophagectomy, gastrectomy, colectomy, prostatectomy, pancreatectomy, liver resection, cholecystectomy, lung resection, and hysterectomy, as well as endoscopic submucosal dissection for early-stage cancer. In this specification, a "structure" refers to any object that constitutes the human body. The structure is at least one of an organ and a body fluid (for example, bile, intestinal fluid, or bleeding blood). While this specification mainly describes the case of an organ, the present invention can also be applied to estimating the position and range of a body fluid and controlling whether or not to display it. In this specification, an "organ" refers to any organ in the body. For example, an organ may be an organ whose shape changes due to a surgeon's risky behavior, an organ that appears in a surgical image due to a surgeon's risky behavior, or an organ that appears in a surgical image after an object to be removed (also called a dissection) during surgery (for example, a malignant tumor and lymph nodes surrounding the tumor, but may also be a benign tumor) is removed.

[0011] <System configuration> Figure 1 is a diagram showing the overall configuration according to one embodiment of the present invention. The surgery support system 1 can include an information processing device (e.g., surgery support device) 10, an imaging device 20, a display device 30, and a surgery support robot 12. A case will be described in which a doctor 11 operates the surgery support robot 12 to perform surgery. Each of these will be described below.

[0012] <<Information processing equipment>> The information processing device 10 is a device that acquires surgical images from surgical videos, estimates the positions and ranges of structures such as organs from the surgical images, and controls the display or non-display of the positions and ranges of the structures such as organs. The information processing device 10 can acquire images captured by the surgical support robot 12 from the surgical support robot 12, or acquire images captured by the imaging device 20 from the imaging device 20. The information processing device 10 can also display the positions and ranges of structures such as organs on a display device 30. The information processing device 10 is made up of one or more computers.

[0013] The information processing device 10 may be used not only during surgery as shown in FIG. 1, but also for education or confirmation after surgery (i.e., it estimates the position and range of structures such as organs from surgical images of previous surgeries, and controls the display or non-display of the position and range of the structures such as organs).

[0014] <<Imaging device>> The imaging device 20 is a device that captures an image of a site where surgery is being performed on the patient 13 (hereinafter also referred to as the surgical site). The image of the surgical site captured by the imaging device 20 is also referred to as a surgical image, and an image included in the surgical image (video) is also referred to as a surgical image (still image).

[0015] <<Display device>> The display device 30 is a monitor, display, or the like that displays information acquired from the information processing device 10. By looking at the information displayed on the display device 30, the doctor 11 can confirm the position and range of structures such as organs of the patient 13.

[0016] In addition, at least two of the information processing device 10, the imaging device 20, the display device 30, and the surgical support robot 12 may be implemented in a single device (for example, the surgical support robot 12 may have the functions of the imaging device 20).

[0017] <Function block> 2 is a functional block diagram of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 may include a surgical image acquisition unit 101, a position / range estimation unit 102, a position / range display control unit 103, a surgical procedure selection unit 104, a surgical procedure determination unit 105, a robot selection unit 106, a preprocessing unit 107, a risk estimation unit 108, a risk display control unit 109, a surgical progress estimation unit 110, a surgical progress display control unit 111, a blood loss estimation unit 112, a blood loss display control unit 113, an organ blood flow estimation unit 114, an organ blood flow display control unit 115, a resection completion estimation unit 116, a resection completion display control unit 117, a model storage unit 118, and a surgical video storage unit 119. By executing the programs, the information processing device 10 can function as a surgical image acquisition unit 101, a position / range estimation unit 102, a position / range display control unit 103, a surgical procedure selection unit 104, a surgical procedure determination unit 105, a robot selection unit 106, a preprocessing unit 107, a risk estimation unit 108, a risk display control unit 109, a surgical progress estimation unit 110, a surgical progress display control unit 111, a bleeding amount estimation unit 112, a bleeding amount display control unit 113, an organ blood flow estimation unit 114, an organ blood flow display control unit 115, a resection completion estimation unit 116, and a resection completion display control unit 117. Each of these units will be described below.

[0018] The surgical image acquisition unit 101 acquires surgical images from the surgical video.

[0019] [Surgery video] Here, the surgical video will be explained. The surgical image acquisition unit 101 may acquire in real time the video captured by the surgical support robot 12 or the imaging device 20 and acquire the surgical images from the surgical video, or may acquire a surgical video captured in the past and stored in the surgical video storage unit 119 and acquire the surgical images from the surgical video.

[0020] The position / range estimation unit 102 estimates the position and range of structures such as organs from at least one surgical image acquired by the surgical image acquisition unit 101. The position / range estimation unit 102 estimates the position and range of structures such as organs in the surgical image by inputting the surgical image acquired by the surgical image acquisition unit 101 into a trained model (estimation model) that has been machine-learned so that when a surgical image is input, the position and range of structures such as organs in the surgical image is output.

[0021] [Location and extent of organs and other structures] Here, the position and range of a structure such as an organ will be described. The position and range of a structure such as an organ may be the region of the structure such as an organ obtained by image segmentation, or may be a rectangle (bounding box) surrounding the structure such as an organ obtained by object detection within the image. Furthermore, the position / range estimation unit 102 may determine the position and range of the structure such as an organ by processing the output of the estimation model, for example, by referring to past frames.

[0022] For example, the position / range estimation unit 102 can estimate the position and range of structures such as organs from the surgical images acquired by the surgical image acquisition unit 101 using an estimation model according to the surgical procedure. It is assumed that the structures such as organs whose positions and ranges are estimated are determined for each surgical procedure (for example, for surgical procedure A, the positions and ranges of organs 1 and 2 are estimated, and for surgical procedure B, the positions and ranges of organs 2 and 3 are estimated). Therefore, a user (for example, doctor 11) can estimate the positions and ranges of structures such as organs to be confirmed for that surgical procedure simply by selecting the surgical procedure.

[0023] Furthermore, the position / range estimation unit 102 may use an estimation model according to the surgery assist robot 12 .

[0024] In addition, in specific cases or specific conditions (e.g., cases with a lot of fat, cases with a lot of liquids (body fluids) such as blood, bile, intestinal fluid, lymph, or a lot of mist), it is difficult to recognize structures such as organs. Therefore, the position / range estimation unit 102 can lower the threshold for recognizing structures such as organs (i.e., the threshold for recognizing structures such as organs in image segmentation and object detection within an image) depending on the amount of fat, blood, bile, intestinal fluid, lymph, or a lot of mist (i.e., when each amount is large). Therefore, the position and range of structures such as organs can be displayed at an early stage. In addition, whether a surgical image acquired by the surgical image acquisition unit 101 is a specific case or a specific condition may be estimated by inputting the surgical image acquired by the surgical image acquisition unit 101 into a trained model that has been machine-learned to output whether the surgical image is a specific case or a specific condition (e.g., whether there is a lot of fat, whether there is a lot of liquids such as blood, bile, intestinal fluid, lymph, or a lot of mist) when the surgical image is input.

[0025] The position / range display control unit 103 controls whether to display or hide the positions and ranges of structures such as organs estimated by the position / range estimation unit 102 .

[0026] [Displaying or hiding the location and extent of structures such as organs] The display and non-display of the positions and ranges of structures such as organs will be explained.

[0027] For example, the position / range display control unit 103 can control whether to display or hide the positions and ranges of structures such as organs according to the surgical procedure. Specifically, the position / range display control unit 103 can display or hide structures such as organs that are determined for each surgical procedure. Therefore, the user can select whether to display or hide the positions and ranges of structures such as organs that should be confirmed for the surgical procedure selected by the user.

[0028] For example, the position / range display control unit 103 can display structures such as organs in a predetermined color tone.

[0029] [Color] Here, the color tone will be explained. The color tone may be determined based on at least one of the following: how to overlay colors, the contrast of the base image (surgical image), the contrast of the mask (region of a structure such as an organ), the transparency of the mask (region of a structure such as an organ), and the blurring of the boundary of the mask (region of a structure such as an organ).

[0030] The position / range display control unit 103 can display structures such as organs in a color tone selected by the user. For example, the position / range display control unit 103 can accept a color tone preset selected by the user and display structures such as organs in the color tone of the preset (that is, the user selects a desired preset from a plurality of color tone presets (where each value of the color overlay, base image contrast, mask contrast, mask transparency, and mask boundary blurring is determined in advance)). Note that the user may also specify each value for determining the color tone (that is, each value of the color overlay, base image contrast, mask contrast, mask transparency, and mask boundary blurring).

[0031] The position / range display control unit 103 can determine a color tone corresponding to a specific case or a specific condition (e.g., the condition of the patient 13) and display structures such as organs in that color tone. For example, specific cases or conditions include cases with a lot of fat, cases with a lot of fluids such as blood, bile, intestinal fluid, and lymphatic fluid, and conditions with a lot of mist. Note that whether a surgical image acquired by the surgical image acquisition unit 101 is a specific case or a specific condition may be estimated by inputting the surgical image acquired by the surgical image acquisition unit 101 into a trained model that has been machine-learned to output whether a surgical image is a specific case or a specific condition (e.g., whether a surgical image is a lot of fat, whether a surgical image is a lot of fluids such as blood, bile, intestinal fluid, and lymphatic fluid, or whether a surgical image is a lot of mist).

[0032] For example, in cases where there is a lot of fat, or where there is a lot of liquid such as blood, bile, intestinal fluid, or lymphatic fluid, or in cases where there is a lot of mist, the following colors are preferable to make it easier for the user to confirm the position and extent of structures such as organs. - Color layering: A layering method that emphasizes the mask (Screen (inverting the color of the base image and multiplying it with the color of the mask)) Base image contrast, mask contrast: Increase Mask transparency: Set it to low (the mask will be more clearly visible) Mask edge feathering: Reduced (makes the mask appear more clearly)

[0033] The larger the area for setting the display of the positions and ranges of structures such as organs (for example, the area on the screen such as 3001 in FIG. 8), the easier it is to operate, but if it is always displayed, it will interfere with the surgery. Therefore, the user may be able to switch between display and non-display. In other words, the position / range display control unit 103 can display or hide the area for setting the display of the positions and ranges of structures such as organs in accordance with the user's instruction to display or not display.

[0034] If the area for setting the display of the position and range of structures such as organs (for example, an area on the screen such as 3001 in FIG. 8) is not transparent, the surgical video will be partially hidden during display. On the other hand, if it is transparent, it will be difficult to check the settings. For this reason, it may be semi-transparent (frosted glass-like). In other words, the position / range display control unit 103 can display the area for setting the display of the position and range of structures such as organs semi-transparently in accordance with the display specification from the user.

[0035] The surgical procedure selection unit 104 accepts a surgical procedure selected by the user.

[0036] The surgical procedure determination unit 105 determines the surgical procedure from the surgical video. For example, the surgical procedure determination unit 105 can determine the surgical procedure by inputting the surgical video into a trained model that has been machine-learned to output a surgical procedure when the surgical video is input.

[0037] Note that, if the surgical procedure selected by the user differs from the surgical procedure determined by the surgical procedure determination unit 105, the surgical procedure determined by the surgical procedure determination unit 105 may be used instead of the surgical procedure selected by the user (i.e., the position / range estimation unit 102 and the position / range display control unit 103 may use the surgical procedure determined by the surgical procedure determination unit 105 instead of the surgical procedure selected by the user). Alternatively, the fact that the two procedures differ may be displayed (i.e., the user may be notified), and the user may specify which one to use. This can prevent the user from making a mistake in selecting a surgical procedure.

[0038] The robot selection unit 106 accepts the type of surgery support robot selected by the user.

[0039] The pre-processing unit 107 performs pre-processing of the surgical video in accordance with the type of surgical support robot selected by the user. The pre-processing of the surgical video will be described in detail later with reference to FIG.

[0040] The risk estimation unit 108 estimates the risk of the surgeon's actions (for example, the degree to which postoperative complications (recurrent laryngeal nerve paralysis, etc.) will occur) from the surgical images acquired by the surgical image acquisition unit 101 and the positions and ranges of structures such as organs (for example, structures such as organs whose shapes change due to the surgeon's risky actions) estimated by the position / range estimation unit 102. For example, the risk estimation unit 108 can estimate the risk of the surgeon's actions by inputting the surgical images into a trained model that has been machine-learned so that the risk of the surgeon's actions is output when the "surgical images" and the "position and range of structures such as organs" are input.

[0041] The risk level display control unit 109 controls whether to display or hide the risk level of the surgeon's behavior estimated by the risk level estimation unit 108. For example, the risk level of the surgeon's behavior is displayed by a bar.

[0042] The surgical progress estimation unit 110 estimates the degree of progress of the surgery (for example, how much of the entire surgery has been completed, how much of the process has been completed, etc.). For example, the surgical progress estimation unit 110 can estimate the degree of progress of the surgery by inputting surgical images into a trained model that has been machine-learned so that when surgical images are input, the trained model outputs the degree of progress of the surgery.

[0043] The surgical progress display control unit 111 controls whether to display or hide the degree of surgical progress. For example, the degree of surgical progress is displayed as a bar.

[0044] The bleeding amount estimation unit 112 estimates the amount of bleeding. For example, the bleeding amount estimation unit 112 can estimate the amount of bleeding by inputting a surgical image into a trained model that has been machine-learned to output the amount of bleeding when a surgical image is input.

[0045] The bleeding amount display control unit 113 controls whether to display the bleeding amount. For example, the bleeding amount is displayed as a bar.

[0046] The organ blood flow degree estimation unit 114 estimates the degree of blood flow in an organ. For example, the organ blood flow degree estimation unit 114 can estimate the degree of blood flow in an organ by inputting a surgical image into a trained model that has been machine-learned so that the degree of blood flow in an organ is output when a surgical image is input.

[0047] The organ blood flow level display control unit 115 controls whether to display or hide the level of blood flow in an organ. For example, the level of blood flow in an organ is displayed as a bar.

[0048] The resection completion estimation unit 116 estimates the completion of resection of the resection object (for example, a malignant tumor and lymph nodes surrounding the tumor) from the surgical image acquired by the surgical image acquisition unit 101 and the position and range of the structure such as an organ estimated by the position and range estimation unit 102. For example, the resection completion estimation unit 116 can estimate the completion of resection of the resection object by inputting the surgical image into a trained model that has been machine-learned so that when the "surgical image" and the "position and range of the structure such as an organ" are input, whether or not the resection of the resection object is completed is output.

[0049] The resection completion display control unit 117 controls whether to display or hide the resection completion. If the next step has not been performed within a certain time period since the resection completion was estimated (specifically, if the user has not changed to the screen for the next step), the resection completion display control unit 117 issues a notification urging the user to proceed to the next step after the certain time period has elapsed. Note that the certain time period may be specified by the user.

[0050] The resection completion display control unit 117 may determine whether the next step has started (for example, by inputting a surgical image into a trained model that has been machine-learned to output which step it is when a surgical image is input, thereby determining whether the next step has started), and may notify the user if the next step has started but the screen has not changed to that of the next step, or may change to the screen of the next step without any notification.

[0051] The model storage unit 118 stores trained models used by the information processing device 10.

[0052] The surgery video storage unit 119 stores surgery videos that have been taken in the past.

[0053] <Processing method> FIG. 3 is a flowchart of a control process for displaying and hiding the position and range of a structure according to one embodiment of the present invention.

[0054] In step 1 (S1), the surgical image acquisition unit 101 of the information processing device 10 acquires a surgical image from a surgical video.

[0055] In step 2 (S2), the position / range estimation unit 102 of the information processing device 10 estimates the position and range of a structure such as an organ from the surgical image acquired in S1.

[0056] In step 3 (S3), the position / range display control unit 103 of the information processing device 10 controls the display and non-display of the positions and ranges of structures such as organs estimated in S2.

[0057] <User Interface> The screen displayed on the display device 30 will now be described.

[0058] 4 is an example of a screen for the user to set information about surgery. The screen for the user to set information about surgery includes an area 1001 for selecting a surgical procedure, an area 1002 for selecting the type of surgery support robot, an area 1003 for selecting a surgery video (specifically, an area for selecting whether the surgery is currently being performed (i.e., the video captured by the surgery support robot 12 or the image capture device 20 is acquired in real time and used) or a surgery performed in the past (i.e., the video of a surgery captured in the past and stored in the surgery video storage unit 119 is acquired and used)), and an area 1004 indicating the available recording time.

[0059] For example, the user can select a surgical procedure, a type of robot, and a surgical video from a plurality of options.

[0060] Pre-processing of the surgical video is performed according to the type of surgical support robot selected by the user in area 1002 for selecting the type of surgical support robot. As shown in FIG. 5, if the video captured by the surgical support robot is full-screen (i.e., video with no margins), the video is used as is. If the video captured by the surgical support robot is not full-screen (i.e., video with margins), pre-processing is performed to remove unnecessary margins (black areas on both sides in the example of FIG. 5). Note that regardless of the type of surgical support robot, it may be possible to determine from the image whether there is margin, and then perform processing to remove the margins.

[0061] Fig. 6 is an example of a screen displaying a surgical video. The surgical video selected on the screen of Fig. 4 is displayed, and the positions and ranges of structures such as organs estimated from the surgical image using an estimation model corresponding to the surgical procedure selected on the screen of Fig. 4 are displayed. As shown in Fig. 6, along with the surgical video, an area 2001 for selecting whether to display or hide the positions and ranges of structures such as organs (structures such as organs determined for each surgical procedure), an area 2002 for selecting the color tone for displaying structures such as organs, an area 2003 for selecting the blinking speed for displaying structures such as organs, and an area 2004 for selecting whether to display and record the positions and ranges of structures such as organs.

[0062] Depending on the surgical procedure, areas 2001 to 2004 in FIG. 6 may be displayed for each step (for example, "lymph node dissection around the right recurrent nerve" and "lymph node dissection around the left recurrent nerve").

[0063] Fig. 7 is a diagram for explaining the color shades (presets) of organs according to one embodiment of the present invention. In the area 2002 for selecting the color shade when displaying structures such as organs in Fig. 6, the user can select a desired preset (i.e., a color shade that does not look strange to the user) from among multiple color shade presets (each of which has a predetermined value for the color overlay method, the contrast of the base image, the contrast of the mask, the transparency of the mask, and the blurring of the mask boundary), thereby simplifying the adjustment of the color shade. As shown in Fig. 7, in each preset, the position and range of each organ are masked with a predetermined color shade, but the user may also be able to adjust the color shade.

[0064] 8 is a diagram illustrating an area for setting the display of the positions and ranges of structures. Area 3001 for setting the display of the positions and ranges of structures such as organs (i.e., area including 2001 to 2004 in FIG. 6) is semi-transparent (frosted glass), and may be able to be switched between display and non-display using a switching button 3002. Therefore, area 3001 for setting the display of the positions and ranges of structures such as organs can be prevented from getting in the way when the user views the surgical video.

[0065] As shown in Figure 9, when the resection of the resection object is completed, a message indicating that the resection of the resection object is complete is displayed (for example, "EXPOSURE" is displayed in white below the surgical image screen). Note that the message is not limited to the completion of the resection of the resection object, and any other completion message may be displayed, or an alert may be displayed (for example, "WIPE" when the camera becomes foggy, or "take out a specimen / gauze" when a specimen or gauze is placed in the abdominal cavity).

[0066] As shown in FIG. 10, the risk level of the surgeon's actions is displayed as a bar (also called a bar gauge). As the risk level increases, the bar scale can be increased and the color can be changed (although the color does not have to be changed). In cases where the bar is operating frequently (i.e., cases where there are many high-risk actions), the color display can be changed (for example, the entire bar can be displayed in red to notify that any actions that are more risky than this should be avoided). Note that the bar in FIG. 10 is horizontal, but it is not limited to this and can also be vertical.

[0067] Although FIG. 10 illustrates the display of the risk level of the surgeon's actions using a bar, various types of information may be displayed using a bar, as described below.

[0068] The progress of the surgery (for example, how much of the whole procedure has been completed, how much of the process has been completed, etc.) may be displayed using a bar.

[0069] During a surgical procedure, particularly during vascular anastomosis, the amount of bleeding may be displayed by a bar.

[0070] During intestinal anastomosis or intestinal resection, the degree of blood flow in the organ may be displayed by a bar.

[0071] When mist generated by an electric scalpel or an ultrasonic scalpel appears, the degree of mist may be displayed by a bar.

[0072] As shown in Figure 11, if the next step is not performed within a certain time after the completion of resection is estimated, a notification (on the right side of Figure 11) is sent after the certain time to prompt the user to proceed to the next step.

[0073] For example, in the case of esophageal cancer resection, when transitioning from right recurrent nerve lymph node dissection to left recurrent nerve lymph node dissection, the user must manually change from right recurrent nerve lymph node dissection mode (the screen displayed during right recurrent nerve lymph node dissection) to left recurrent nerve lymph node dissection mode (the screen displayed during left recurrent nerve lymph node dissection), but there is a possibility that the user may forget. Therefore, if the mode is not changed from right recurrent nerve lymph node dissection mode to left recurrent nerve lymph node dissection mode within a certain period of time after the right recurrent nerve lymph node dissection is completed (i.e., after resection completion (EXPOSURE) is displayed), a notification will be displayed after the certain period of time prompting the user to change to left recurrent nerve lymph node dissection mode.

[0074] <Effects> In one embodiment of the present invention, doctors and others can display the position and range of a structure (at least one of an organ and a body fluid) in a surgical image in a desired manner.

[0075] <Hardware configuration> 12 is a hardware configuration diagram of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 can include a control unit 1001, a main memory unit 1002, an auxiliary memory unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0076] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003.

[0077] The main memory unit 1002 includes a non-volatile memory (Read Only Memory (ROM)) and a volatile memory (Random Access Memory (RAM)). The ROM stores various programs, data, etc. required for the control unit 1001 to execute various programs installed in the auxiliary memory unit 1003. The RAM provides a working area into which the various programs installed in the auxiliary memory unit 1003 are expanded when executed by the control unit 1001.

[0078] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0079] The input unit 1004 is an input device through which the operator of the information processing device 10 inputs various instructions to the information processing device 10 .

[0080] The output unit 1005 is an output device that outputs the internal state of the information processing device 10 and the like.

[0081] The interface unit 1006 is a communication device for connecting to a network and communicating with other devices.

[0082] Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims. [Explanation of symbols]

[0083] 1. Surgical support system 10 Surgical support equipment 20 Imaging device 30 Display device 11 Doctor 12 Surgical support robot 13 patients 101 Surgical image acquisition unit 102 Position and range estimation unit 103 Position / range display control section 104 Surgical Procedure Selection Section 105 Surgical Procedure Judgment Department 106 Robot Selection Section 107 Pretreatment section 108 Risk Estimation Department 109 Risk level display control unit 110 Surgery progress estimation unit 111 Surgery progress display control unit 112 Bleeding amount estimation section 113 Bleeding volume display control unit 114 Organ Blood Flow Estimation Unit 115 Organ blood flow display control unit 116 Resection completion estimation section 117 Resection completion display control unit 118 Model Memory Unit 119 Surgical Video Storage Unit 1001 control section 1002 Main memory 1003 Auxiliary storage unit 1004 Input section 1005 Output section 1006 Interface section

Claims

1. a surgical image acquisition unit that acquires a surgical image from the surgical video; a position and range estimation unit that estimates the position and range of a structure from the surgical image; a position / range display control unit that controls whether to display or hide the estimated structure position and range; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the structure is at least one of an organ and a body fluid.

3. the position / range estimation unit estimates the position and range of the structure from the surgical image using an estimation model according to the surgical procedure; The information processing device according to claim 1 , wherein the position / range display control unit controls whether to display or hide the position and range of the structure in accordance with the surgical procedure.

4. The information processing device according to claim 3 , further comprising a surgical procedure selection unit that accepts the surgical procedure selected by a user.

5. The information processing device according to claim 3 , further comprising a surgical procedure determination unit that determines the surgical procedure from the surgical video.

6. A surgical procedure selection unit that accepts the surgical procedure selected by the user; A surgical procedure determination unit that determines the surgical procedure from the surgical video, 4. The information processing device according to claim 3, wherein when the surgical procedure selected by the user and the surgical procedure determined by the surgical procedure determination unit are different, the position / range estimation unit and the position / range display control unit use the surgical procedure determined by the surgical procedure determination unit instead of the surgical procedure selected by the user.

7. A surgical procedure selection unit that accepts the surgical procedure selected by the user; A surgical procedure determination unit that determines the surgical procedure from the surgical video, The information processing device according to claim 3 , wherein when the surgical procedure selected by the user differs from the surgical procedure determined by the surgical procedure determination unit, the user is notified of this.

8. a robot selection unit that accepts the type of surgical assist robot selected by a user; a pre-processing unit that performs pre-processing of the surgical video in accordance with the type of the surgical support robot selected by the user; The information processing device according to claim 1 , further comprising:

9. The information processing device according to claim 1 , wherein the position / range display control unit displays the structure in a color selected by a user.

10. The information processing device according to claim 1 , wherein the position / range display control unit determines a color tone corresponding to a specific case or a specific condition, and displays the structure in the determined color tone.

11. 11. The information processing device according to claim 9, wherein the color tone is determined based on at least one of a color overlay method, a contrast of a base image, a contrast of a mask, a transparency of the mask, and a blur of a boundary of the mask.

12. The information processing device according to claim 1 , wherein the area for setting the display of the position and range of the structure can be switched between display and non-display by a user.

13. The information processing device according to claim 1 , wherein the area for setting the display of the position and range of the structure is semi-transparent.

14. a risk estimation unit that estimates the risk of the surgeon's actions from the surgical image and the position and range of the structure; a risk level display control unit that controls whether to display or hide the risk level of the surgeon's action; Furthermore, The information processing device according to claim 1 , wherein the degree of risk of the operator's action is displayed by a bar.

15. a surgical progress estimation unit that estimates the degree of progress of the surgery; a surgical progress display control unit that controls whether to display or hide the degree of progress of the surgical procedure; Furthermore, The information processing device according to claim 1 , wherein the degree of progress of the surgery is displayed by a bar.

16. a bleeding amount estimation unit that estimates the amount of bleeding; a bleeding amount display control unit that controls whether to display or hide the bleeding amount; Furthermore, The information processing device according to claim 1 , wherein the amount of bleeding is displayed by a bar.

17. an organ blood flow degree estimation unit that estimates the degree of blood flow in the organ; an organ blood flow degree display control unit that controls whether to display or hide the degree of blood flow in the organ; Furthermore, The information processing device according to claim 2 , wherein the degree of blood flow in the organ is displayed by a bar.

18. a resection completion estimation unit that estimates the completion of resection from the surgical image and the position and range of the structure; Controlling whether to display or hide the completion of the resection; a resection completion display control unit that issues a notification urging the user to proceed to the next step after a certain time has elapsed since the completion of the resection has been estimated, if the user has not proceeded to the next step within the certain time period; The information processing device according to claim 1 , further comprising:

19. In the information processing device, acquiring a surgical image from the surgical video; estimating the location and extent of a structure from the surgical image; Controlling whether to display or hide the estimated structure position and range; A program that executes the following.

20. A method executed by an information processing device, acquiring a surgical image from the surgical video; estimating the location and extent of a structure from the surgical image; a step of controlling whether to display or hide the estimated structure position and range; A method comprising:

Images