Information processing device, method, and program
The information processing device improves surgical image analysis by estimating and controlling the display of organ positions and ranges, addressing the challenges of existing interfaces and enhancing surgical precision.
Patent Information
- Application Number
- JP2024080848
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-05-17
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2044-05-17
AI Technical Summary
Doctors face challenges in effectively displaying and managing the positions and ranges of organs in surgical images, necessitating an improved user interface for surgical image analysis.
An information processing device that acquires surgical images, estimates the position and range of structures within them, and controls their display or non-display, utilizing machine-learned models to enhance usability and adapt to various surgical procedures and conditions.
Enhances the usability of surgical image interfaces by accurately displaying and managing organ positions and ranges, aiding surgeons in safer and more efficient surgical procedures.
Smart Images

Figure 0007587898000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, a method, and a program. [Background technology]
[0002] Conventionally, there is known a technology that displays the positions and extents of organs in an image of a patient undergoing surgery (hereinafter, also referred to as a surgical image). By checking the positions and extents of the displayed organs, a doctor can avoid high-risk actions (e.g., actions that may cause postoperative complications) and determine whether the removal of a malignant tumor and lymph nodes surrounding the tumor has been completed. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 7031925 Summary of the Invention [Problem to be solved by the invention]
[0004] However, doctors are unable to display the positions and extents of organs within surgical images in a desired manner, and a more user-friendly user interface is desired.
[0005] Therefore, an object of the present invention is to improve the ease of use of a user interface that provides the location and extent of an organ within a surgical image. [Means for solving the problem]
[0006] An information processing device according to one embodiment of the present invention includes a surgical image acquisition unit that acquires surgical images from surgical video, a position and range estimation unit that estimates the position and range of a structure from the surgical images, and a position and range display control unit that controls the display or non-display of the estimated position and range of the structure. Effect of the Invention
[0007] The present invention provides an improved ease of use user interface that provides the location and extent of organs within a surgical image. [Brief description of the drawings]
[0008] [Figure 1] 1 is a diagram showing an overall configuration according to an embodiment of the present invention; [Diagram 2] 1 is a functional block diagram of an information processing device according to an embodiment of the present invention; [Diagram 3] 11 is a flowchart of a control process for displaying and not displaying the position and range of a structure according to an embodiment of the present invention. [Figure 4] 11 is an example of a screen according to an embodiment of the present invention. [Diagram 5] 11 is a diagram for explaining pre-processing of a surgical video according to one embodiment of the present invention. FIG. [Figure 6] 11 is an example of a screen according to an embodiment of the present invention. [Figure 7] FIG. 13 is a diagram for explaining color shades (presets) of organs according to one embodiment of the present invention. [Figure 8] 11 is a diagram for explaining an area for setting the display of the position and range of a structure according to one embodiment of the present invention. FIG. [Figure 9] 11 is an example of a screen according to an embodiment of the present invention. [Figure 10] 11 is an example of a screen according to an embodiment of the present invention. [Figure 11] 11 is an example of a screen according to an embodiment of the present invention. [Figure 12] 1 is a hardware configuration diagram of an information processing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, an embodiment of the invention will be described with reference to the drawings.
[0010] <Terminology explanation> In this specification, "surgery" may refer to any type of surgery, including surgery performed by humans alone, surgery performed by humans operating medical equipment such as endoscopes (e.g., laparoscopic surgery, thoracoscopic surgery, etc.), surgery performed by humans operating a robot, or surgery performed by a robot alone. For example, surgery is a surgical procedure to remove cancer tissue, including the primary lesion and lymph nodes, from tissue. For example, surgery may be esophagectomy, gastrectomy, colectomy, prostatectomy, pancreatectomy, liver resection, cholecystectomy, lung resection, or hysterectomy, but may also be endoscopic submucosal dissection for early-stage cancer. In this specification, a "structure" refers to any object that constitutes the human body. The structure is at least one of an organ and a body fluid (e.g., bile, intestinal fluid, or bleeding blood). Note that this specification mainly describes the case of an organ, but the present invention can also be applied to estimating the position and range of a body fluid and controlling whether it is displayed or not. In this specification, an "organ" refers to any organ in the body. For example, an organ may be an organ whose shape changes due to a high-risk action of the surgeon, an organ that appears in a surgical image due to a high-risk action of the surgeon, or an organ that appears in a surgical image when something that should be removed (also called dissection) during surgery (for example, a malignant tumor and lymph nodes around the tumor, but may also be a benign tumor) is removed.
[0011] <System configuration> Fig. 1 is a diagram showing the overall configuration according to one embodiment of the present invention. The surgery support system 1 can include an information processing device (e.g., surgery support device) 10, an imaging device 20, a display device 30, and a surgery support robot 12. A case will be described in which a doctor 11 operates the surgery support robot 12 to perform surgery. Each of these will be described below.
[0012] <<Information processing devices>> The information processing device 10 is a device that acquires surgical images from surgical videos, estimates the positions and ranges of structures such as organs from the surgical images, and controls display and non-display of the positions and ranges of the structures such as organs. The information processing device 10 can acquire images captured by the surgical support robot 12 from the surgical support robot 12, or acquire images captured by the imaging device 20 from the imaging device 20. The information processing device 10 can also display the positions and ranges of structures such as organs on the display device 30. The information processing device 10 is composed of one or more computers.
[0013] The information processing device 10 may be used not only during surgery as shown in FIG. 1, but also after surgery for education or confirmation (i.e., estimating the position and range of structures such as organs from surgical images of previous surgeries, and controlling the display or non-display of the position and range of the structures such as organs).
[0014] <<Imaging device>> The imaging device 20 is a device that captures an image of a site where surgery is being performed on the patient 13 (hereinafter also referred to as a surgical site). An image of the surgical site captured by the imaging device 20 is also referred to as a surgical image, and an image included in the surgical image (moving image) is also referred to as a surgical image (still image).
[0015] <<Display device>> The display device 30 is a monitor, a display, or the like that displays information acquired from the information processing device 10. The doctor 11 can confirm the positions and ranges of structures such as organs of the patient 13 by looking at the information displayed on the display device 30.
[0016] In addition, at least two of the information processing device 10, the imaging device 20, the display device 30, and the surgical support robot 12 may be implemented in a single device (for example, the surgical support robot 12 may have the functions of the imaging device 20).
[0017] <Function block> 2 is a functional block diagram of an information processing device 10 according to an embodiment of the present invention. The information processing device 10 may include a surgical image acquisition unit 101, a position / range estimation unit 102, a position / range display control unit 103, a surgical procedure selection unit 104, a surgical procedure determination unit 105, a robot selection unit 106, a preprocessing unit 107, a risk estimation unit 108, a risk display control unit 109, a surgical progress estimation unit 110, a surgical progress display control unit 111, a bleeding amount estimation unit 112, a bleeding amount display control unit 113, an organ blood flow estimation unit 114, an organ blood flow display control unit 115, a resection completion estimation unit 116, a resection completion display control unit 117, a model storage unit 118, and a surgical video storage unit 119. By executing the programs, the information processing device 10 can function as a surgical image acquisition unit 101, a position / range estimation unit 102, a position / range display control unit 103, a surgical procedure selection unit 104, a surgical procedure determination unit 105, a robot selection unit 106, a preprocessing unit 107, a risk estimation unit 108, a risk display control unit 109, a surgical progress estimation unit 110, a surgical progress display control unit 111, a bleeding amount estimation unit 112, a bleeding amount display control unit 113, an organ blood flow estimation unit 114, an organ blood flow display control unit 115, a resection completion estimation unit 116, and a resection completion display control unit 117. Each of these units will be described below.
[0018] The surgical image acquisition unit 101 acquires a surgical image from a surgical video.
[0019] [Surgery video] Here, the surgical video will be described. The surgical image acquisition unit 101 may acquire the video captured by the surgical support robot 12 or the imaging device 20 in real time and acquire the surgical image from the surgical video, or may acquire the surgical video captured in the past and stored in the surgical video storage unit 119 and acquire the surgical image from the surgical video.
[0020] The position / range estimation unit 102 estimates the position and range of structures such as organs from at least one surgical image acquired by the surgical image acquisition unit 101. The position / range estimation unit 102 estimates the position and range of structures such as organs in a surgical image by inputting the surgical image acquired by the surgical image acquisition unit 101 into a trained model (estimation model) that has been machine-learned so that when a surgical image is input, the position and range of structures such as organs in the surgical image is output.
[0021] [Location and extent of organs and other structures] Here, the position and range of a structure such as an organ will be described. The position and range of a structure such as an organ may be the region of the structure such as an organ obtained by image segmentation, or may be a rectangle (bounding box) surrounding the structure such as an organ obtained by object detection within the image. Furthermore, the position / range estimation unit 102 may determine the position and range of a structure such as an organ by processing the output of the estimation model by referring to past frames, for example.
[0022] For example, the position / range estimation unit 102 can estimate the position and range of structures such as organs from the surgical image acquired by the surgical image acquisition unit 101 using an estimation model according to the surgical procedure. The structures such as organs whose positions and ranges are estimated are assumed to be determined for each surgical procedure (for example, the positions and ranges of organs 1 and 2 are estimated in surgical procedure A, and the positions and ranges of organs 2 and 3 are estimated in surgical procedure B). Therefore, a user (for example, doctor 11) can estimate the positions and ranges of structures such as organs to be confirmed for that surgical procedure simply by selecting the surgical procedure.
[0023] Furthermore, the position / range estimation unit 102 may use an estimation model suited to the surgery support robot 12 .
[0024] In addition, in a specific case or a specific state (for example, a case with a lot of fat, a case with a lot of liquid (body fluid) such as blood, bile, intestinal fluid, lymphatic fluid, or a state with a lot of mist), it is difficult to recognize structures such as organs, so the position / range estimation unit 102 can lower the threshold for recognizing structures such as organs (that is, the threshold for recognizing structures such as organs in image segmentation and object detection in an image) according to the amount of fat, the amount of liquid such as blood, bile, intestinal fluid, lymphatic fluid, or the amount of mist (that is, when each amount is large). Therefore, the position and range of structures such as organs can be displayed at an early stage. In addition, whether a specific case or a specific state is present may be estimated by inputting a surgical image acquired by the surgical image acquisition unit 101 into a trained model that has been machine-learned to output whether a specific case or a specific state is present (for example, whether there is a lot of fat, whether there is a lot of liquid such as blood, bile, intestinal fluid, lymphatic fluid, or the like, or whether there is a lot of mist) when a surgical image is input.
[0025] The position / range display control unit 103 controls whether to display or not display the positions and ranges of structures such as organs estimated by the position / range estimation unit 102 .
[0026] [Display or hide the location and extent of structures such as organs] The display and non-display of the positions and ranges of structures such as organs will be described.
[0027] For example, the position / range display control unit 103 can control whether to display or hide the positions and ranges of structures such as organs according to the surgical procedure. Specifically, the position / range display control unit 103 can display or hide structures such as organs that are determined for each surgical procedure. Therefore, the user can select whether to display or hide the positions and ranges of structures such as organs that should be confirmed for the surgical procedure selected by the user.
[0028] For example, the position / range display control unit 103 can display structures such as organs in a predetermined color tone.
[0029] [Color] Here, the color tone will be described. The color tone may be determined based on at least one of the following: the way of overlaying colors, the contrast of the base image (surgical image), the contrast of the mask (region of the structure such as an organ), the transparency of the mask (region of the structure such as an organ), and the blurring of the boundary of the mask (region of the structure such as an organ).
[0030] The position / range display control unit 103 can display structures such as organs in a color tone selected by the user. For example, the position / range display control unit 103 can receive a preset color tone selected by the user and display structures such as organs in the color tone of the preset (that is, the user selects a desired preset from among a plurality of preset color tone tones (each of which has a predetermined value for the color overlay, the base image contrast, the mask contrast, the mask transparency, and the mask boundary blur)). Note that the user may specify each value for determining the color tone (that is, each of the values for the color overlay, the base image contrast, the mask contrast, the mask transparency, and the mask boundary blur).
[0031] The position / range display control unit 103 can determine a color tone according to a specific case or a specific state (e.g., the state of the patient 13) and display structures such as organs in that color tone. For example, specific cases or states are cases with a lot of fat, cases with a lot of liquid such as blood, bile, intestinal fluid, lymphatic fluid, and a state with a lot of mist. In addition, whether it is a specific case or a specific state may be estimated by inputting a surgical image acquired by the surgical image acquisition unit 101 into a trained model that has been machine-learned to output whether it is a specific case or a specific state (e.g., whether there is a lot of fat, whether there is a lot of liquid such as blood, bile, intestinal fluid, lymphatic fluid, or whether there is a lot of mist) when a surgical image is input.
[0032] For example, in cases with a lot of fat, or a lot of liquid such as blood, bile, intestinal fluid, or lymphatic fluid, or in cases with a lot of mist, the following colors are preferable to make it easier for the user to confirm the position and extent of structures such as organs. - Color overlay: A method that emphasizes the mask (Screen (the base image color is inverted and then multiplied with the mask color)) Base image contrast, mask contrast: Increase Mask transparency: Set it to low (the mask will be more clearly visible) Mask edge feathering: Reduced (mask is more clearly visible)
[0033] The larger the area for setting the display of the positions and ranges of structures such as organs (for example, an area on the screen such as 3001 in FIG. 8), the easier it is to operate, but displaying it all the time would be an impediment to surgery. For this reason, it may be possible for the user to switch between display and non-display. In other words, the position / range display control unit 103 can display or hide the area for setting the display of the positions and ranges of structures such as organs in response to a user's designation of display or non-display.
[0034] If an area for setting the display of the position and range of structures such as organs (for example, an area within the screen such as 3001 in FIG. 8) is not transparent, the surgical video will be partially hidden during display. On the other hand, if it is transparent, it will be difficult to check the settings. For this reason, it may be semi-transparent (like frosted glass). In other words, the position / range display control unit 103 can display the area for setting the display of the position and range of structures such as organs semi-transparently in response to a display specification from the user.
[0035] The surgical procedure selection unit 104 accepts a surgical procedure selected by the user.
[0036] The surgical procedure determination unit 105 determines a surgical procedure from a surgical video. For example, the surgical procedure determination unit 105 can determine a surgical procedure by inputting the surgical video to a trained model that has been machine-learned to output a surgical procedure when the surgical video is input.
[0037] In addition, when the surgical procedure selected by the user and the surgical procedure determined by the surgical procedure determination unit 105 are different, the surgical procedure determined by the surgical procedure determination unit 105 may be used instead of the surgical procedure selected by the user (i.e., the position / range estimation unit 102 and the position / range display control unit 103 may use the surgical procedure determined by the surgical procedure determination unit 105 instead of the surgical procedure selected by the user). Alternatively, the fact that the two are different may be displayed (i.e., the user is notified), and the user may specify which one to use. This can prevent the user from making a mistake in selecting a surgical procedure.
[0038] The robot selection unit 106 accepts the type of surgery support robot selected by the user.
[0039] The pre-processing unit 107 performs pre-processing of the surgical video in accordance with the type of surgical support robot selected by the user. The pre-processing of the surgical video will be described in detail later with reference to FIG.
[0040] The risk estimation unit 108 estimates the risk of the surgeon's actions (e.g., the degree of causing postoperative complications (recurrent laryngeal nerve paralysis, etc.)) from the surgical image acquired by the surgical image acquisition unit 101 and the position and range of a structure such as an organ (e.g., a structure such as an organ whose shape changes due to the surgeon's highly risky actions) estimated by the position / range estimation unit 102. For example, the risk estimation unit 108 can estimate the risk of the surgeon's actions by inputting the surgical image into a trained model that has been machine-learned to output the risk of the surgeon's actions when a "surgical image" and the "position and range of a structure such as an organ" are input.
[0041] The risk level display control unit 109 controls whether to display or not the risk level of the surgeon's behavior estimated by the risk level estimation unit 108. For example, the risk level of the surgeon's behavior is displayed by a bar.
[0042] The surgery progress estimation unit 110 estimates the degree of progress of the surgery (for example, how much of the whole surgery has been completed, how much of the process has been completed, etc.). For example, the surgery progress estimation unit 110 can estimate the degree of progress of the surgery by inputting a surgery image to a trained model that has been machine-learned to output the degree of progress of the surgery when a surgery image is input.
[0043] The surgery progress display control unit 111 controls whether to display or hide the progress of the surgery. For example, the progress of the surgery is displayed by a bar.
[0044] The bleeding amount estimation unit 112 estimates the amount of bleeding. For example, the bleeding amount estimation unit 112 can estimate the amount of bleeding by inputting a surgical image to a trained model that has been machine-learned to output the amount of bleeding when a surgical image is input.
[0045] The bleeding amount display control unit 113 controls whether to display the bleeding amount. For example, the bleeding amount is displayed by a bar.
[0046] The organ blood flow degree estimation unit 114 estimates the degree of blood flow in an organ. For example, the organ blood flow degree estimation unit 114 can estimate the degree of blood flow in an organ by inputting a surgical image to a trained model that has been machine-learned to output the degree of blood flow in an organ when a surgical image is input.
[0047] The organ blood flow level display control unit 115 controls whether to display or not the level of blood flow in an organ. For example, the level of blood flow in an organ is displayed by a bar.
[0048] The resection completion estimation unit 116 estimates the completion of resection of the resection object (e.g., a malignant tumor and lymph nodes surrounding the tumor) from the surgical image acquired by the surgical image acquisition unit 101 and the position and range of the structure such as an organ estimated by the position / range estimation unit 102. For example, the resection completion estimation unit 116 can estimate the completion of resection of the resection object by inputting the surgical image into a trained model that has been machine-learned to output whether or not the resection of the resection object has been completed when a "surgical image" and a "position and range of the structure such as an organ" are input.
[0049] The resection completion display control unit 117 controls whether to display or hide the resection completion. If the next step is not performed within a certain time after the resection completion is estimated (specifically, the user has not changed to the screen for the next step), the resection completion display control unit 117 issues a notification to prompt the user to proceed to the next step after the certain time. The certain time may be specified by the user.
[0050] The resection completion display control unit 117 may determine whether the next step has started (for example, by inputting a surgical image into a trained model that has been machine-learned to output which step it is when a surgical image is input, thereby determining whether the next step has started), and may notify if the next step has started but the screen has not changed to that for the next step, or may change to the screen for the next step without notifying the user.
[0051] The model storage unit 118 stores trained models used by the information processing device 10.
[0052] The surgery video storage unit 119 stores surgery videos captured in the past.
[0053] <Processing method> FIG. 3 is a flowchart of a control process for displaying and not displaying the position and range of a structure according to one embodiment of the present invention.
[0054] In step 1 (S1), the surgical image acquisition unit 101 of the information processing device 10 acquires a surgical image from a surgical video.
[0055] In step 2 (S2), the position / range estimating unit 102 of the information processing device 10 estimates the position and range of a structure such as an organ from the surgical image acquired in S1.
[0056] In step 3 (S3), the position / range display control unit 103 of the information processing device 10 controls the display and non-display of the positions and ranges of structures such as organs estimated in S2.
[0057] <User Interface> The screen displayed on the display device 30 will now be described.
[0058] 4 is an example of a screen for a user to set information about surgery. The screen for a user to set information about surgery includes an area 1001 for selecting a surgical procedure, an area 1002 for selecting a type of surgery support robot, an area 1003 for selecting a surgery video (specifically, an area for selecting whether the surgery is currently being performed (i.e., the video captured by the surgery support robot 12 or the imaging device 20 is obtained in real time and used) or a surgery performed in the past (i.e., the video captured in the past and stored in the surgery video storage unit 119 is obtained and used)), and an area 1004 showing the available recording time.
[0059] For example, the user can select a surgical procedure, a type of robot, and a surgical video from a plurality of options.
[0060] Pre-processing of the surgical video is performed according to the type of surgical support robot selected by the user in area 1002 for selecting the type of surgical support robot. As shown in FIG. 5, when the video captured by the surgical support robot is a full-screen video (i.e., video with no margins), the video is used as is. When the video captured by the surgical support robot is not a full-screen video (i.e., video with margins), pre-processing is performed to delete unnecessary margins (black parts on both ends in the example of FIG. 5). Note that regardless of the type of surgical support robot, the presence or absence of margins may be determined from the image, and the margins may be deleted.
[0061] Fig. 6 is an example of a screen on which a surgical video is displayed. The surgical video selected on the screen of Fig. 4 is displayed, and the positions and ranges of structures such as organs estimated from the surgical image using an estimation model corresponding to the surgical procedure selected on the screen of Fig. 4 are displayed. As shown in Fig. 6, together with the surgical video, an area 2001 for selecting whether to display or not display the positions and ranges of structures such as organs (structures such as organs determined for each surgical procedure), an area 2002 for selecting a color when displaying structures such as organs, an area 2003 for selecting a blinking speed when displaying structures such as organs, and an area 2004 for selecting whether to display and record the positions and ranges of structures such as organs.
[0062] Depending on the surgical procedure, areas 2001 to 2004 in FIG. 6 may be displayed for each step (for example, "peri-right recurrent nerve lymph node dissection" and "peri-left recurrent nerve lymph node dissection").
[0063] Fig. 7 is a diagram for explaining the color tone (preset) of an organ according to an embodiment of the present invention. In the area 2002 for selecting the color tone when displaying a structure such as an organ in Fig. 6, the user can select a desired preset (i.e., a color tone that does not look strange to the user) from among a plurality of color tone presets (each value of the color overlay, the contrast of the base image, the contrast of the mask, the transparency of the mask, and the blur of the mask boundary is determined in advance), and the adjustment of the color tone can be simplified. As shown in Fig. 7, in each preset, the position and range of each organ are masked with a predetermined color tone, but the user may also be able to adjust the color tone.
[0064] 8 is a diagram for explaining an area for setting the display of the position and range of a structure. Area 3001 for setting the display of the position and range of a structure such as an organ (i.e., area including 2001 to 2004 in FIG. 6) may be semi-transparent (frosted glass-like), and may be able to be switched between displayed and hidden by a switching button 3002. Therefore, it is possible to prevent area 3001 for setting the display of the position and range of a structure such as an organ from getting in the way when the user views the surgical video.
[0065] As shown in Fig. 9, when the resection of the resection object is completed, a message indicating that the resection of the resection object is completed is displayed (for example, "EXPOSURE" is displayed in white below the surgical image screen). Note that the message is not limited to the completion of the resection of the resection object, and any other completion may be displayed, or an alert may be displayed (for example, "WIPE" when the camera becomes foggy, or "take out a specimen / gauze" when a specimen or gauze is in the abdominal cavity).
[0066] As shown in FIG. 10, the risk of the surgeon's actions is displayed as a bar (also called a bar gauge). As the risk increases, the bar's scale can be increased and the color can be changed (it is not necessary to change the color). In cases with a lot of bar activity (i.e., cases with a lot of high-risk actions), the color display can be changed (for example, the entire bar can be displayed in red to inform the surgeon that any actions that are more risky than this should be avoided). Note that the bar in FIG. 10 is horizontal, but it is not limited to this and may be vertical.
[0067] Although FIG. 10 illustrates the display of the risk level of the surgeon's actions using a bar, various information may be displayed using a bar as described below.
[0068] The progress of the surgery (for example, how much of the whole procedure has been completed, how much of the process has been completed, etc.) may be displayed using a bar.
[0069] During a surgical procedure, particularly during vascular anastomosis, the amount of bleeding may be displayed by a bar.
[0070] During intestinal anastomosis or intestinal resection, the degree of blood flow in the organ may be displayed by a bar.
[0071] When mist generated by an electric scalpel or an ultrasonic scalpel appears, the level of the mist may be displayed by a bar.
[0072] As shown in FIG. 11, if the next step is not performed within a certain time after the completion of resection is estimated, a notification (on the right side of FIG. 11) is sent after the certain time to urge the user to proceed to the next step.
[0073] For example, in the case of esophageal cancer resection, when switching from right recurrent nerve lymph node dissection to left recurrent nerve lymph node dissection, the user must manually change from right recurrent nerve lymph node dissection mode (screen displayed during right recurrent nerve lymph node dissection) to left recurrent nerve lymph node dissection mode (screen displayed during left recurrent nerve lymph node dissection), but there is a possibility that the user may forget. Therefore, if the mode is not changed from right recurrent nerve lymph node dissection mode to left recurrent nerve lymph node dissection mode within a certain time after right recurrent nerve lymph node dissection is completed (i.e., after resection completion (EXPOSURE) is displayed), a notification is issued after the certain time to prompt the user to change to left recurrent nerve lymph node dissection mode.
[0074] <Effects> In one embodiment of the present invention, a doctor or the like can display the position and extent of a structure (at least one of an organ and a body fluid) in a surgical image in a desired manner.
[0075] <Hardware configuration> 12 is a hardware configuration diagram of an information processing device 10 according to an embodiment of the present invention. The information processing device 10 can include a control unit 1001, a main memory unit 1002, an auxiliary memory unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.
[0076] The control unit 1001 is a processor (for example, a central processing unit (CPU), a graphics processing unit (GPU), etc.) that executes various programs installed in the auxiliary storage unit 1003.
[0077] The main memory unit 1002 includes a non-volatile memory (Read Only Memory (ROM)) and a volatile memory (Random Access Memory (RAM)). The ROM stores various programs, data, etc. required for the control unit 1001 to execute various programs installed in the auxiliary memory unit 1003. The RAM provides a working area into which the various programs installed in the auxiliary memory unit 1003 are expanded when the control unit 1001 executes them.
[0078] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when the various programs are executed.
[0079] The input unit 1004 is an input device through which the operator of the information processing device 10 inputs various instructions to the information processing device 10 .
[0080] The output unit 1005 is an output device that outputs the internal state of the information processing device 10 and the like.
[0081] The interface unit 1006 is a communication device for connecting to a network and communicating with other devices.
[0082] Although the examples of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]
[0083] 1. Surgical support system 10 Surgical support equipment 20 Imaging device 30 Display device 11 Doctor 12 Surgical robots 13 patients 101 Surgical image acquisition unit 102 Position and range estimation unit 103 Position / range display control section 104 Surgical Procedure Selection Section 105 Surgical Procedure Judgment Department 106 Robot Selection Section 107 Pretreatment section 108 Risk Estimation Department 109 Risk level display control unit 110 Surgery Progress Estimation Department 111 Surgery progress display control unit 112 Bleeding amount estimation section 113 Bleeding volume display control unit 114 Organ Blood Flow Estimation Unit 115 Organ blood flow display control unit 116 Resection completion estimation section 117 Resection completion display control unit 118 Model Memory Section 119 Surgery video storage unit 1001 Control section 1002 Main memory 1003 Auxiliary storage unit 1004 Input section 1005 Output section 1006 Interface section
Claims
1. a surgical image acquisition unit for acquiring a surgical image from a surgical video; a position and range estimation unit that estimates the position and range of a structure from the surgical image using an estimation model corresponding to a surgical procedure; a position / range display control unit that controls display and non-display of the estimated position and range of the structure according to the surgical procedure, determines a color tone according to a specific case or a specific condition, and displays the structure in the color tone; A surgical procedure selection unit that accepts the surgical procedure selected by a user; A surgical procedure determination unit that determines the surgical procedure from the surgical image, When the procedure selected by the user is different from the procedure determined by the procedure determination unit, the position / range estimation unit and the position / range display control unit use the procedure determined by the procedure determination unit instead of the procedure selected by the user. Information processing device.
2. The information processing device according to claim 1 , wherein the structure is at least one of an organ and a body fluid.
3. A robot selection unit that accepts the type of surgical support robot selected by the user; A pre-processing unit that performs pre-processing of the surgical video according to the type of the surgical support robot selected by the user; The information processing device according to claim 1 , further comprising:
4. The information processing device according to claim 1 , wherein the color tone is determined based on at least one of a color overlay method, a contrast of a base image, a contrast of a mask, a transparency of the mask, and a blur of a boundary of the mask.
5. The information processing device according to claim 1 , wherein the area for setting the display of the position and range of the structure can be switched between displayed and hidden by the user.
6. The information processing device according to claim 1 , wherein the area for setting the display of the position and range of the structure is semi-transparent.
7. a risk estimation unit that estimates a risk of the surgeon's actions from the surgical image and the position and range of the structure; a risk level display control unit that controls whether to display or hide the risk level of the action of the surgeon; Further equipped with The information processing device according to claim 1 , wherein the degree of risk of the action of the surgeon is displayed by a bar.
8. a surgery progress estimation unit that estimates a degree of progress of the surgery; A surgery progress display control unit that controls whether to display or hide the progress of the surgery; Further equipped with The information processing device according to claim 1 , wherein the degree of progress of the surgery is displayed by a bar.
9. A bleeding amount estimation unit that estimates a bleeding amount; A bleeding amount display control unit that controls whether to display or hide the bleeding amount; Further equipped with The information processing device according to claim 1 , wherein the amount of bleeding is displayed by a bar.
10. an organ blood flow degree estimation unit that estimates the degree of blood flow in the organ; an organ blood flow degree display control unit that controls whether to display or hide the degree of blood flow of the organ; Further equipped with The information processing device according to claim 2 , wherein the degree of blood flow in the organ is displayed by a bar.
11. a resection completion estimation unit that estimates the completion of resection from the surgical image and the position and range of the structure; Controlling display and non-display of the completion of the resection; a resection completion display control unit that, when the process has not been shifted to the next process within a certain time period after the completion of the resection has been estimated, issues a notification to urge the process to be shifted to the next process after the certain time period has elapsed; The information processing device according to claim 1 , further comprising:
12. In the information processing device, Obtaining a surgical image from the surgical video; Estimating the position and range of a structure from the surgical image using an estimation model corresponding to the surgical procedure; Controlling display and non-display of the position and range of the estimated structure according to the surgical procedure, determining a color tone according to a specific case or a specific condition, and displaying the structure in the color tone; Accepting the technique selected by a user; determining the surgical procedure from the surgical video; When the surgical procedure selected by the user differs from the surgical procedure determined from the surgical video, the program uses the surgical procedure determined from the surgical video rather than the surgical procedure selected by the user when estimating the position and range of the structure and controlling the display or non-display of the position and range of the structure.
13. A method executed by an information processing device, comprising: acquiring a surgical image from the surgical video; estimating the position and range of a structure from the surgical image using an estimation model corresponding to the surgical procedure; A step of controlling whether to display or not display the position and range of the estimated structure according to the surgical procedure, determining a color tone according to a specific case or a specific condition, and displaying the structure in the color tone; A step of accepting the surgical technique selected by a user; determining the surgical procedure from the surgical image; When the surgical procedure selected by the user differs from the surgical procedure determined from the surgical video, the method uses the surgical procedure determined from the surgical video rather than the surgical procedure selected by the user when estimating the position and range of the structure and controlling the display or non-display of the position and range of the structure.
Citation Information
Patent Citations
Biological information monitoring apparatus, screen display method, and screen display program
JP2011182872A
Healthcare systems and related methods
JP2020518366A
Surgery support system. surgery support method and surgery support program
JP2023079038A
Computer program, surgery assistance device, and information processing method
JP7031925B1
Surgical microscope system, image processing method, program, and image processing device
WO2020179588A1
