Surgical assistance program, surgical assistance device, and surgical assistance method

Abstract

The purpose of the present invention is to notify dangerous behaviors during surgery with high accuracy. A program according to one embodiment of the present invention causes a computer to execute: a process for acquiring a surgical image in which a surgical site is captured; a dangerous behavior detection unit that detects a dangerous behavior from the acquired surgical image using a model generated by machine learning of learning data, the input of which is the surgical image and the output of which is information related to the dangerous behavior; and notifying a case where the dangerous behavior is detected, wherein the information associated with the dangerous behavior includes region information of a structure in the surgical image that changes due to the dangerous behavior.

Description

Technical Field

[0001] The present invention relates to a surgical assistance procedure, a surgical assistance device and a surgical assistance method. Background Art

[0002] In the past, postoperative complications not only reduced the patient's QOL (quality of life), but also led to a deterioration in long-term prognosis and an increase in medical expenses. Reducing postoperative complications is one of the most important issues for surgeons. It is known that the incidence of complications is closely related to the experience of the surgeon and the size of the facility. There is no doubt that the surgical technique during surgery is the cause of postoperative complications. Regarding the cause of postoperative complications, compared with the lack of technical skills in the technique, there are many cases where the lack of understanding or lack of attention to the intraoperative situation is the cause.

[0003] As a method for reducing postoperative complications, for example, Patent Document 1 discloses learning information related to surgical images and complication risks, inputting surgical images during inference, and outputting risk analysis information.

[0004] <Prior Art Literature>

[0005] <Patent Documents>

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-029258 Summary of the invention

[0007] <Problems to be Solved by the Invention>

[0008] However, Patent Document 1 merely learns and infers information related to surgical images and complication risks, and it is expected that when dangerous behavior that may cause postoperative complications occurs during surgery, notification can be made with high accuracy.

[0009] Therefore, an object of the present invention is to notify risky behaviors during surgery with high accuracy.

[0010] <Methods used to solve the problem>

[0011] A program involved in one embodiment of the present invention causes a computer to perform the following processing: obtaining a surgical image that captures a surgical site; detecting dangerous behavior from the acquired surgical image using a model generated by machine learning of learning data that has a surgical image as input and outputs information associated with dangerous behavior; and notifying a situation in which the dangerous behavior has been detected, wherein the information associated with the dangerous behavior includes area information of structures within the surgical image that have changed due to the dangerous behavior.

[0012] According to the present invention, dangerous behavior during surgery can be notified with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a diagram showing the overall structure according to one embodiment of the present invention.

[0014] Figure 2 This is a functional block diagram of a surgical support device according to an embodiment of the present invention.

[0015] Figure 3 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0016] Figure 4 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0017] Figure 5 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0018] Figure 6 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0019] Figure 7 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0020] Figure 8 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0021] Fig. 9 This is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention.

[0022] Fig.10 This is a diagram for explaining learning according to one embodiment of the present invention.

[0023] Fig.11 This is a diagram for explaining learning according to one embodiment of the present invention.

[0024] Fig.12 This is a flowchart of a learning process according to one embodiment of the present invention.

[0025] Fig.13 This is a flowchart of a dangerous behavior notification process according to one embodiment of the present invention.

[0026] Fig.14 This is a flowchart of a learning data generation process according to one embodiment of the present invention.

[0027] Fig.15 This is a flowchart of a dangerous behavior notification process that also uses other information according to one embodiment of the present invention.

[0028] Fig.16 This is a flowchart of an evaluation process according to one embodiment of the present invention.

[0029] Fig.17 This is an example of information and evaluation results that serve as a basis for evaluation according to one embodiment of the present invention.

[0030] Fig.18 This is a hardware configuration diagram of a surgical assistance device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the invention will be described based on the drawings.

[0032] <Explanation of terms>

[0033] In this specification, "surgery" may be any surgery, and may be a surgery performed only by humans, a surgery performed by humans operating medical equipment such as endoscopes (e.g., laparoscopic surgery, thoracoscopic surgery, etc.), a surgery performed by humans operating robots, or a surgery performed only by robots. For example, surgery is a surgical procedure to remove cancerous tissue including primary lesions and lymph nodes from tissues. For example, surgery is esophagectomy, gastrectomy, colectomy, hepatectomy, pancreatectomy, endoscopic surgery for early cancers such as early digestive tract cancer (endoscopic submucosal dissection).

[0034] ·In this specification, "dangerous behavior" refers to behavior that occurs during surgery and may become the cause of postoperative complications. In other words, "dangerous behavior" is behavior (e.g., behavior that pulls the recurrent laryngeal nerve, behavior that exposes blood vessels) that has the possibility of easily turning into behavior that causes postoperative complications (e.g., behavior that paralyzes the recurrent laryngeal nerve, behavior that causes blood vessels to bleed). "Dangerous behavior" can be said to be the behavior that precedes the behavior that causes postoperative complications. For example, dangerous behavior is behavior that deforms any element (structure) that constitutes the human body, or behavior that exposes any element (structure) that constitutes the human body (i.e., appears in a surgical image (an image of the surgical site)). For example, a dangerous behavior is behavior that causes paralysis of the recurrent laryngeal nerve (i.e., behavior that pulls the recurrent laryngeal nerve).

[0035] In this specification, a "structure" is any element constituting the human body. For example, a structure is an organ in the body (such as the recurrent laryngeal nerve, muscle layer, blood vessel), body fluid (such as bile, intestinal fluid, bleeding blood), etc.

[0036] <System Structure>

[0037] Figure 1 1 is a diagram showing the overall configuration of one embodiment of the present invention. A surgery support system 1 includes a surgery support device 10, an imaging device 20, and a display device 30. A case where a doctor 11 performs surgery on a patient 12 will be described. Each of the following will be described.

[0038] <<Surgery assist devices>>

[0039] The surgical assisting device 10 is a device that detects dangerous behavior based on the image captured by the camera device 20 during surgery, and notifies the doctor 11 (for example, displays it on the display device 30) that the dangerous behavior has been detected. The surgical assisting device 10 is composed of one or more computers. The surgical assisting device 10 can obtain the image captured by the camera device 20 from the camera device 20. In addition, the surgical assisting device 10 can display information such as the detection of dangerous behavior on the display device 30.

[0040] <<Camera device>>

[0041] The imaging device 20 is a device that captures an image of a region where surgery is performed on the patient 12 (hereinafter also referred to as a surgical region). Hereinafter, an image of the surgical region captured by the imaging device 20 is also referred to as a surgical image.

[0042] <<Display device>>

[0043] The display device 30 is a monitor, a display, or the like that displays information acquired from the surgery support device 10. The doctor 11 can recognize that he or she is performing a dangerous behavior by observing the information displayed on the display device 30.

[0044] In addition, at least two of the surgery support device 10 , the imaging device 20 , and the display device 30 may be realized by a single device.

[0045] <Function box>

[0046] Figure 21 is a functional block diagram of a surgical assisting device 10 according to an embodiment of the present invention. The surgical assisting device 10 includes a learning data storage unit 101, a learning data acquisition unit 102, a dangerous behavior detection unit 103, an error calculation unit 104, a parameter update unit 105, a parameter storage unit 106, a surgical image acquisition unit 107, a dangerous behavior detection unit 108, and a dangerous behavior notification unit 109. The surgical assisting device 10 functions as the learning data acquisition unit 102, the dangerous behavior detection unit 103, the error calculation unit 104, the parameter update unit 105, the surgical image acquisition unit 107, the dangerous behavior detection unit 108, and the dangerous behavior notification unit 109 by executing a program. Each of these functions will be described below.

[0047] Learning data is stored in the learning data storage unit 101. The learning data is a surgical image in which a surgical site is captured and information associated with a dangerous behavior in the surgical image.

[0048] The learning data acquisition unit 102 acquires the learning data stored in the learning data storage unit 101 .

[0049] Here, the "information associated with dangerous behavior" is explained. The "information associated with dangerous behavior" includes: "regional information of structures within the surgical image that change due to dangerous behavior (in addition, the region information is at least one of segmentation information representing the region of the structure within the image (information representing the structure to be segmented) and frame information representing the region of the structure within the image (information such as a rectangle surrounding the structure))"; and "a label indicating whether the surgical image is an image showing dangerous behavior or an image showing non-dangerous behavior (behavior that is not dangerous behavior)". Below, refer to Figure 3 , Figures 4 to 9 , the regional information of the structures within the surgical image is described in detail.

[0050] The regional information of the structure in the surgical image that changes due to the dangerous behavior is the regional information of the structure that changes due to the dangerous behavior of the doctor 11 or the like (for example, the regional information of the structure changes due to the shape change of the structure, the regional information of the structure changes due to the appearance of the structure). Specifically, the regional information of the structure in the surgical image that changes due to the dangerous behavior includes the regional information of the structure that changes in shape due to the dangerous behavior (that is, the regional information of the structure that is deformed due to the dangerous behavior of the doctor 11 or the like. Figure 3 ), and regional information of structures that appear in the surgical image due to dangerous behavior (i.e., regional information of structures that appear in the surgical image due to dangerous behavior by the doctor 11, etc. Figures 4 to 9 .

[0051] Figure 3 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 3 , the state of nerve deformation caused by dangerous behavior of doctor 11 and others is shown.

[0052] Figure 3 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0053] about Figure 3 The “regional information of structures in the surgical image that change due to dangerous behavior” is given to the surgical image. The regional information of structures that change due to dangerous behavior is used in the learning of the model.

[0054] Figure 4 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 4 , a state in which the chest duct is revealed due to dangerous behavior of the doctor 11 or the like is shown.

[0055] Figure 4 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0056] about Figure 4 The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images. In esophagectomy, the regional information of the thoracic duct (a structure that appears in surgical images due to risky behaviors) can be used for model learning.

[0057] In this way, during esophagectomy or lymph node removal, when the chest duct, which is usually difficult to identify and easily damaged (if damaged, it will cause chylothorax), is identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications.

[0058] Figure 5 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 5 , the pulmonary artery is shown to be visible due to the dangerous behavior of the doctor 11 or the like.

[0059] Figure 5 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0060] about Figure 5 The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images. In esophagectomy, the regional information of the pulmonary artery (a structure that appears in surgical images due to risky behaviors) can be used for model learning.

[0061] In this way, during esophagectomy and lymph node removal, when the pulmonary artery, which is usually difficult to identify and easily damaged (if damaged, it will cause massive bleeding and there is a risk of death), is identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications.

[0062] Figure 6 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 6 , the hypogastric nerve is shown to be exposed due to the dangerous behavior of the doctor 11 and others.

[0063] Figure 6 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0064] about Figure 6 The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images with regional information of the hypogastric nerve (a structure that appears in surgical images due to risky behaviors). In colectomy, regional information of the hypogastric nerve and pelvic plexus (structures that appear in surgical images due to risky behaviors) can be used for model learning.

[0065] In this way, during colectomy or lymph node removal, when nerves that should not be exposed (if damaged, ejaculation disorders, erectile dysfunction, and urination disorders may occur) are identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications.

[0066] Figure 7 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 7 , it is shown that small blood vessels are visible under the mucosa due to dangerous behavior of the doctor 11 and others.

[0067] Figure 7 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0068] about Figure 7The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images. In endoscopic surgery for early gastrointestinal cancer, the regional information of small blood vessels (structures that appear in surgical images due to risky behaviors) can be used for model learning.

[0069] In this way, during endoscopic surgery for early gastrointestinal cancer, when small blood vessels that are usually easy to miss and easily damaged (if damaged, bleeding will occur) are identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications before they occur.

[0070] In addition, the “regional information of structures in the surgical image that change due to dangerous behavior” may be regional information of structures covered by mucous membranes or the like (eg, Figure 7 The area information of the mucous membrane etc. may be used, or the area information of the mucous membrane etc. may not be used, or the area information of the structure not covered by the mucous membrane etc. may be used.

[0071] Figure 8 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Figure 8 , the hepatic veins and liver capsule (Glisson) are shown to be visible due to the dangerous behavior of the doctor 11 and others.

[0072] Figure 8 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0073] about Figure 8 The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images. In liver resection, the regional information of hepatic veins and liver capsule (structures that appear in surgical images due to risky behaviors) can be used for model learning.

[0074] In this way, during liver resection, when the hepatic veins and liver capsules, which are usually difficult to identify and easily damaged (if damaged, bile fistula and bleeding may occur), are identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications.

[0075] Fig. 9 FIG. 1 is a diagram for explaining region information of a structure in a surgical image that changes due to a dangerous behavior according to an embodiment of the present invention. Fig. 9 , the state where small blood vessels are revealed due to dangerous behavior of the doctor 11 or the like is shown.

[0076] Fig. 9 The "surgical images" are images obtained by photographing the surgical site during surgery.

[0077] about Fig. 9 The “regional information of structures in surgical images that change due to risky behaviors” is given to surgical images. In pancreatic resection, the regional information of small blood vessels (structures that appear in surgical images due to risky behaviors) can be used for model learning.

[0078] Thus, during pancreatic resection, when branches of large blood vessels such as the inferior vena cava, which are usually easy to miss and easily damaged (major bleeding will occur if damaged), are identified in the image, the operator is notified as a dangerous behavior, thereby preventing serious complications.

[0079] Alternatively, the region information of the structure may be generated by specifying the region of the structure in the surgical image by a person, or by inferring the region information of the structure by using a machine learning model using the surgical image in which the region of the structure is specified by a person. Alternatively, the label may be added by judging whether the surgical image is a dangerous behavior or a non-dangerous behavior by a person, or by using information detected by a sensor as described in the following paragraph.

[0080] Return to Figure 2 The dangerous behavior detection unit 103 detects dangerous behavior using the model being learned.

[0081] [Example 1]

[0082] In reference Fig.10 In the first embodiment described above, the dangerous behavior detection unit 103 inputs the surgical image acquired by the learning data acquisition unit 102 into the model being learned ( Fig.10 The feature extractor 111 and restorer 112 of the learning data acquisition unit 102 are used to output the regional information of the structure in the surgical image. In addition, the dangerous behavior detection unit 103 inputs the surgical image obtained by the learning data acquisition unit 102 into the model being learned ( Fig.10 The feature extractor 111 and the classifier 113 of the present invention are used to output the presence or absence of dangerous behavior (whether it is a dangerous behavior or not). In addition, the presence or absence of dangerous behavior can also be expressed by a ratio.

[0083] [Example 2]

[0084] In reference Fig.11 In the second embodiment described above, the dangerous behavior detection unit 103 inputs the surgical image acquired by the learning data acquisition unit 102 into the model being learned ( Fig.11The feature extractor A211A and restorer 212 of the learning data acquisition unit 102 are used to output the regional information of the structure in the surgical image. In addition, the dangerous behavior detection unit 103 inputs the surgical image obtained by the learning data acquisition unit 102 and the regional information of the structure in the surgical image restored by the restorer 212 into the model being learned ( Fig.11 The feature extractor B211B and feature extractor C211C and classifier 213) output the presence or absence of dangerous behavior (dangerous behavior or not dangerous behavior). In addition, the presence or absence of dangerous behavior can also be expressed by a ratio.

[0085] The error calculation unit 104 calculates an error between the dangerous behavior related information detected from the surgical image by the dangerous behavior detection unit 103 and the dangerous behavior related information of the surgical image acquired by the learning data acquisition unit 102 .

[0086] The parameter updating unit 105 updates the parameters of the model being learned so that the error calculated by the error calculating unit 104 becomes smaller, and stores the updated parameters in the parameter storage unit 106 .

[0087] The parameter storage unit 106 stores the parameters of the model updated by the parameter updating unit 105 .

[0088] Below, refer to Fig.10 (Example 1) and Fig.11 (Example 2) The learning process is described in detail.

[0089] Fig.10 1 is a diagram for explaining learning according to one embodiment of the present invention (Example 1). The feature extractor 111 extracts feature quantities from the surgical image as learning data, the restorer 112 restores the regional information of the structure or the surgical image containing the regional information of the structure based on the feature quantities, and the classifier 113 determines whether it is a dangerous behavior or not based on the feature quantities. That is, in Example 1, when inferring, the classifier 113 uses the feature quantities extracted by the feature extractor 111 during learning to classify the surgical image. The following is a description of each.

[0090] The feature extractor (also referred to as encoder) 111 extracts the feature quantity of the surgical image from the surgical image. Specifically, the feature extractor 111 extracts the feature quantity by performing dimensionality reduction on the surgical image.

[0091] The restorer (also called decoder) 112 restores the regional information of the structure or the surgical image containing the regional information of the structure based on the feature quantity of the surgical image extracted by the feature extractor 111 (in addition, the restored regional information of the structure may be only the regional information of the structure that changes due to the dangerous behavior, or may include not only the regional information of the structure that changes due to the dangerous behavior but also the regional information of other structures). The error between the result restored by the restorer 112 and the regional information of the structure of the learning data (correct answer data) (i.e., the regional information of the structure in the surgical image that changes due to the dangerous behavior) is calculated, and learning is performed (i.e., updating the parameters) in a way that the error becomes smaller.

[0092] The classifier 113 determines whether the surgical image is an image showing a dangerous behavior or an image showing a non-dangerous behavior based on the feature quantity of the surgical image extracted by the feature extractor 111. The error between the result determined by the classifier 113 and the label (label indicating a dangerous behavior or indicating a non-dangerous behavior) of the learning data (correct answer data) is calculated, and learning is performed (i.e., updating parameters) in a manner that the error becomes smaller.

[0093] Fig.11 2 is a diagram for explaining learning according to an embodiment of the present invention (Example 2). The feature extractor A211A extracts feature quantities from a surgical image serving as learning data, and the restorer 212 restores the regional information of the structure or the surgical image containing the regional information of the structure based on the feature quantities. The feature extractor B211B extracts feature quantities from the surgical image containing the regional information of the structure restored by the restorer 212. The feature extractor C211C extracts feature quantities from the surgical image serving as learning data. The classifier 213 determines whether it is a dangerous behavior or not based on the feature quantities obtained by combining the feature quantities extracted by the feature extractor B211B and the feature quantities extracted by the feature extractor C211C. That is, in Example 2, when inferring, the feature extractor A211A and the restorer 212 restore the regional information of the structure from the surgical image, and the classifier 213 uses the feature quantities of the surgical image and the feature quantities of the restored regional information of the structure to classify the surgical image. The following describes each of them.

[0094] The feature extractor (also referred to as an encoder) A211A extracts the feature quantity of the surgical image from the surgical image. Specifically, the feature extractor A211A performs dimensionality reduction on the surgical image to extract the feature quantity.

[0095] The restorer (also called decoder) 212 restores the regional information of the structure or the surgical image containing the regional information of the structure based on the feature quantity of the surgical image extracted by the feature extractor A211A (in addition, the restored regional information of the structure may be only the regional information of the structure that changes due to the dangerous behavior, or may include not only the regional information of the structure that changes due to the dangerous behavior but also the regional information of other structures). The error between the result restored by the restorer 212 and the regional information of the structure of the learning data (correct answer data) (i.e., the regional information of the structure that changes due to the dangerous behavior) is calculated, and learning is performed in a way to reduce the error (i.e., update the parameters).

[0096] The feature extractor (also referred to as an encoder) B211B extracts feature quantities of the region information of the structure from the region information of the structure restored by the restorer 212. Specifically, the feature extractor B211B performs dimensionality reduction on the region information of the structure restored by the restorer 212 to extract feature quantities.

[0097] The feature extractor (also called encoder) C211C extracts the feature quantity of the surgical image from the surgical image. Specifically, the feature extractor C211C performs dimensionality reduction on the surgical image to extract the feature quantity.

[0098] The classifier 213 determines whether the surgical image is an image showing a dangerous behavior or an image showing a non-dangerous behavior based on a feature quantity obtained by combining the feature quantity extracted by the feature extractor B211B and the feature quantity extracted by the feature extractor C211C. The error between the result determined by the classifier 213 and the label of the learning data (correct answer data) (i.e., the label indicating a dangerous behavior or the label indicating a non-dangerous behavior) is calculated, and learning is performed (i.e., updating parameters) in a manner that reduces the error.

[0099] By learning in this way, the surgical assisting device 10 can reflect the recognition degree and positional relationship of the structure that changes due to the dangerous behavior during the operation in the detection of the dangerous behavior. That is, in the present invention, the segmentation of the anatomical structure or the result of the frame information (i.e., the regional information of each structure) is used as attention to detect the dangerous behavior, so the information of each structure can help to judge whether it is a dangerous behavior.

[0100] For example, the regional information of the structure in the surgical image that changes due to dangerous behavior is information on the shape of the recurrent laryngeal nerve during esophagectomy (i.e., regional information on the recurrent laryngeal nerve). In this case, when the recurrent laryngeal nerve is deformed due to a certain behavior, by notifying the operator of the deformation as a dangerous behavior, serious complications can be prevented before they occur.

[0101] For example, the area information of the structure in the surgical image that changes due to the dangerous behavior is information of at least one of the muscular layer and the blood vessel in the endoscopic submucosal dissection (i.e., area information of at least one of the muscular layer and the blood vessel). In this case, when the muscular layer or the blood vessel that should not be exposed is identified, it is possible to prevent serious complications by notifying the operator of the dangerous behavior.

[0102] For example, the regional information of the structure in the surgical image that changes due to the dangerous behavior is information on at least one of bile, intestinal fluid, and bleeding (bleeding blood) during esophagectomy (i.e., regional information on at least one of bile, intestinal fluid, or bleeding). In this case, when a certain amount of bile, intestinal fluid, or bleeding is recognized during surgery, it is possible to prevent serious complications by notifying the operator of the dangerous behavior.

[0103] In addition, when machine learning was performed without using the regional information of the structure, the model's inference accuracy was 46.8%, and when machine learning was performed using the regional information of the structure, the model's inference accuracy was 88.0%. In this way, by using the regional information of the structure, the accuracy of the model's inference can be improved.

[0104] Back to Figure 2 The surgical image acquisition unit 107 acquires surgical images (eg, dynamic images) captured by the imaging device 20 during surgery.

[0105] The dangerous behavior detection unit 108 detects dangerous behaviors based on the surgical images acquired by the surgical image acquisition unit 107. Specifically, the dangerous behavior detection unit 108 inputs the surgical images acquired by the surgical image acquisition unit 107 into a model (specifically, a model using the parameters stored in the parameter storage unit 106), and outputs the presence or absence of dangerous behaviors (dangerous behaviors or non-dangerous behaviors). In addition, the presence or absence of dangerous behaviors can also be expressed as a ratio (a classification model can be used to output the probability of dangerous behaviors or the probability of non-dangerous behaviors, and a regression model can also be used to output the probability of dangerous behaviors or the probability of non-dangerous behaviors).

[0106] The dangerous behavior notification unit 109 notifies (e.g., displays on the display device 30) in real time the dangerous behavior detected by the dangerous behavior detection unit 108. For example, the dangerous behavior notification unit 109 displays the detection of the dangerous behavior on the surgical image acquired by the surgical image acquisition unit 107. For example, the dangerous behavior notification unit 109 displays "there is a dangerous behavior" or the like.

[0107] <Handling method>

[0108] Below, refer to Fig.12 For an explanation of the learning process, refer to Fig.13 The method of handling dangerous behavior notification is explained.

[0109] Fig.12 This is a flowchart of a learning process according to one embodiment of the present invention.

[0110] In step 101 ( S101 ), the surgery support device 10 acquires a surgery image as learning data.

[0111] In step 102 ( S102 ), the surgery support device 10 detects dangerous behavior using the model being learned.

[0112] In reference Fig.10 In the first embodiment described above, the surgery support device 10 inputs the surgery image acquired in S101 into the model being learned ( Fig.10 The surgical support device 10 inputs the surgical image obtained in S101 into the model ( Fig.10 The feature extractor 111 and the classifier 113) output the presence or absence of dangerous behavior (dangerous behavior or not dangerous behavior).

[0113] In reference Fig.11 In the second embodiment described above, the surgery support device 10 inputs the surgery image acquired in S101 into the model being learned ( Fig.11 The surgical assisting device 10 inputs the surgical image obtained in S101 and the regional information of the structure restored by the restorer 212 into the model being learned ( Fig.11 The feature extractor B211B, the feature extractor C211C and the classifier 213) output the presence or absence of dangerous behavior (either dangerous behavior or not).

[0114] In step 111 (S111), the surgical assistance device 10 obtains information associated with dangerous behavior (specifically, area information of structures within the surgical image that change due to dangerous behavior, and a label indicating whether the surgical image is an image showing dangerous behavior or an image showing non-dangerous behavior (behavior that is not dangerous behavior)) from the surgical image serving as learning data (i.e., the surgical image obtained in S101).

[0115] In step 112 (S112), the surgical support device 10 converts the data acquired in S111 into correct answer data. Specifically, the surgical support device 10 makes the data acquired in S111 consistent with the format of the output of the model.

[0116] In step 103 ( S103 ), the surgery support device 10 calculates the error between the information related to the dangerous behavior detected in S102 and the correct answer data in S112 .

[0117] In step 104 ( S104 ), the surgery support device 10 updates the parameters of the model being learned so that the error calculated in S103 becomes smaller, and stores the updated parameters in the parameter storage unit 106 .

[0118] In step 105 (S105), the surgery support device 10 determines whether the learning is completed. If the learning is not completed, the process returns to S101 and S111 to acquire the next learning data. If the learning is completed, the present process is terminated.

[0119] Fig.13 This is a flowchart of a dangerous behavior notification process according to one embodiment of the present invention.

[0120] In step 201 ( S201 ), the surgery support device 10 acquires a surgery image (for example, a moving image) captured by the imaging device 20 during surgery.

[0121] In step 202 (S202), the surgery support device 10 detects dangerous behavior based on the surgery image obtained in S201. Specifically, the surgery support device 10 inputs the surgery image obtained in S201 into Fig.12 The model generated in the output is the presence or absence of dangerous behavior (dangerous behavior or not dangerous behavior). When dangerous behavior is detected, enter step 203.

[0122] In reference Fig.10 In the described first embodiment, the classifier 113 classifies the surgical image using the feature quantity extracted by the feature extractor 111 during learning.

[0123] In reference Fig.11 In the second embodiment described above, the feature extractor A211A and the restorer 212 restore the region information of the structure from the surgical image, and the classifier 213 classifies the surgical image using the feature quantity of the surgical image and the feature quantity of the restored region information of the structure.

[0124] In step 203 (S203), the surgery support device 10 notifies (for example, displays on the display device 30) in real time that the dangerous behavior was detected in S202. For example, the surgery support device 10 displays the detection of the dangerous behavior on the surgery image acquired in S201.

[0125] <Effect>

[0126] In one embodiment of the present invention, when dangerous behavior that may cause postoperative complications occurs during surgical procedures, notification is given in real time on the surgical image, so that the operator can immediately avoid dangerous behavior, and as a result, postoperative complications can be avoided.

[0127] Other embodiments will be described below. In addition, a plurality of the embodiments described below may be implemented in combination.

[0128] [Generating learning data]

[0129] The surgical assistance device 10 can generate learning data using information detected by sensors. Specifically, the surgical assistance device 10 can add labels to surgical images that serve as learning data based on information detected by sensors (e.g., sensors of an intraoperative nerve monitoring system). For example, the information detected by the sensor is information related to dangerous behaviors such as excessive pulling of the recurrent laryngeal nerve or surrounding tissues, damage to the recurrent laryngeal nerve, etc. that may cause recurrent laryngeal nerve paralysis. Fig.14 , explaining the labeling of learning data.

[0130] Fig.14 This is a flowchart of a learning data generation process according to one embodiment of the present invention.

[0131] In step 301 ( S301 ), the surgery support device 10 acquires a surgery image (for example, a moving image) captured by the imaging device 20 during surgery.

[0132] In step 302 ( S302 ), the surgery support device 10 acquires information detected by sensors during the surgery in which the surgery image in S301 is acquired (for example, information detected by sensors of an intraoperative nerve monitoring system).

[0133] In step 303 (S303), the surgery support device 10 determines whether the information acquired in S302 is a notification of a dangerous behavior. If the information acquired in S302 is a notification of a dangerous behavior, the process proceeds to step 304, and if the information acquired in S302 is not a notification of a dangerous behavior, the process proceeds to step 305.

[0134] In step 304 ( S304 ), the surgery support device 10 adds a label indicating that the dangerous behavior is present to the surgical image corresponding to the time period in which the notification of the dangerous behavior in S303 exists.

[0135] In step 305 ( S305 ), the surgery support device 10 adds a label indicating that it is not a dangerous behavior to the surgical image corresponding to the time period of the notification that it is not a dangerous behavior in S303 .

[0136] In this way, by using the information detected by the sensor to generate learning data, it is possible to eliminate the burden of humans adding labels to surgical images to indicate the presence or absence of dangerous behavior in order to generate learning data.

[0137] [Information about structures not shown in the surgical image]

[0138] The information associated with the dangerous behavior may also include information about structures not shown in the surgical image. It is also important to grasp the information about structures not shown in the surgical image for detecting dangerous behavior. Therefore, the information about structures not shown in the surgical image is obtained for determining whether it is a dangerous behavior. For example, the information about structures not shown in the surgical image is information indicating an area in the surgical image predicted to be the course of the recurrent laryngeal nerve.

[0139] In this case, in addition to referring to Fig.10 and Fig.11 In addition to the "regional information of structures within the surgical image that change due to dangerous behavior" described above, learning is also performed using "regional information of structures not explicitly shown in the surgical image (for example, the region within the surgical image predicted to be the course of the recurrent laryngeal nerve) (in addition, the region information is at least one of segmentation information representing the region of the structure within the image (information representing the structure to be segmented) and frame information representing the region of the structure within the image (for example, information such as a rectangle enclosing the structure))."

[0140] In this way, by also using information on structures that are not clearly shown in the surgical image, it is possible to prevent unnecessary peeling operations, reduce complications, reduce the amount of bleeding, shorten the surgical time, and the like.

[0141] [Detection and notification of dangerous behavior using other information]

[0142] In one embodiment of the present invention, other information can also be used to detect and notify dangerous behavior. The following describes the case where [[information on the patient's condition before surgery]] is used.

[0143] [[Information on the patient's condition before surgery]]

[0144] The surgical assisting device 10 can also use the information of the patient's state before the operation to detect dangerous behavior. Whether it is a dangerous behavior during the operation depends on the patient's state. Therefore, the information of the patient's state is obtained before the operation to determine whether it is a dangerous behavior. For example, the information of the patient's state before the operation is a preoperative CT image or a patient background when detecting and notifying a dangerous behavior that may cause pneumonia complications during esophagectomy.

[0145] In this way, by also using the patient's pre-operative information, it is possible to select a surgical procedure that matches the patient's condition and risk.

[0146] Fig.15 This is a flowchart of a dangerous behavior notification process that also uses other information according to an embodiment of the present invention.

[0147] In step 401 ( S401 ), the surgery support device 10 acquires a surgery image captured by the imaging device 20 during surgery.

[0148] In step 411 ( S411 ), the surgery support device 10 acquires other information (for example, information on the patient's condition before surgery).

[0149] In step 402 (S402), the surgical assistance device 10 detects dangerous behavior based on the surgical image obtained in S401 and other information obtained in S411. Specifically, when learning the model, other information (for example, information on the patient's condition before surgery) is input in addition to the surgical image. The surgical assistance device 10 inputs the surgical image obtained in S401 and other information obtained in S411 to the model generated by machine learning of learning data that inputs surgical images and other information (for example, information on the patient's condition before surgery) and outputs information associated with dangerous behavior, and outputs the presence or absence of dangerous behavior (dangerous behavior or not dangerous behavior). When dangerous behavior is detected, proceed to step 403.

[0150] In step 403 ( S403 ), the surgery support device 10 notifies (for example, displays on the display device 30 ) in real time that the dangerous behavior is detected in S402 .

[0151] [Evaluation of Manipulative Procedures]

[0152] The surgery support device 10 can evaluate the manipulation of operators such as surgeons and endoscopists based on information related to risky behaviors outputted by inputting surgical images into a model. Fig.16 as well as Fig.17 , the evaluation of the manipulation operation is explained.

[0153] Fig.16 This is a flowchart of an evaluation process according to one embodiment of the present invention.

[0154] In step 501 ( S501 ), the surgery support device 10 acquires information related to dangerous behaviors output by inputting a surgery image into a model.

[0155] In step 502 (S502), the surgery support device 10 obtains information related to the surgery. The information related to the surgery may be the overall surgery time, the surgery time at each stage, data for identifying the movement of the instrument and the operator, and the like.

[0156] In step 503 (S503), the surgery support device 10 evaluates the operator's manipulation based on the information acquired in S501 and S502. For example, the surgery support device 10 may evaluate the operator's manipulation using machine learning.

[0157] Fig.17 It is an example of information as a basis for evaluation and an example of an evaluation result according to an embodiment of the present invention. For example, the information as a basis for evaluation is the time of each stage of the operation (e.g., lymph node removal around the recurrent laryngeal nerve), the category of dangerous behavior (e.g., recurrent laryngeal nerve traction), etc. For example, the evaluation result can be a score.

[0158] By evaluating the manipulation technique in this way, it is possible to provide feedback on the manipulation technique after surgery or use it for surgical education.

[0159] <Hardware Structure>

[0160] Fig.18 1 is a hardware configuration diagram of a surgical support device 10 according to an embodiment of the present invention. The surgical support device 10 may include a control unit 1001, a main storage unit 1002, an auxiliary storage unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0161] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003 .

[0162] The main storage unit 1002 includes a nonvolatile memory (ROM (Read Only Memory)) and a volatile memory (RAM (Random Access Memory)). The ROM stores various programs and data required for the control unit 1001 to execute various programs installed in the auxiliary storage unit 1003. The RAM provides a work area that is developed when the control unit 1001 executes various programs installed in the auxiliary storage unit 1003.

[0163] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0164] The input unit 1004 is an input device for the operator of the surgery support apparatus 10 to input various instructions to the surgery support apparatus 10 .

[0165] The output unit 1005 is an output device that outputs the internal state of the surgery support device 10 and the like.

[0166] The interface unit 1006 is a communication device connected to a network for communicating with other devices.

[0167] As mentioned above, although the embodiment of the present invention was described in detail, the present invention is not limited to the above-mentioned specific embodiment, and various deformation|transformation and changes are possible within the scope of the gist of the present invention described in the claims.

[0168] This application claims priority based on Japanese Patent Application No. 2022-143972 filed on September 9, 2022, and the entire contents of No. 2022-143972 are hereby incorporated by reference into this application.

[0169] Explanation of symbols

[0170] 1: Surgery Assistance System

[0171] 10: Surgical assist devices

[0172] 20: Camera device

[0173] 30: Display device

[0174] 11: Doctor

[0175] 12: Patient

[0176] 101: Learn Data Storage

[0177] 102: Learning data acquisition unit

[0178] 103: Dangerous Behavior Detection Department

[0179] 104: Error calculation unit

[0180] 105: Parameter update unit

[0181] 106: Parameter storage unit

[0182] 107: Surgical Image Acquisition Department

[0183] 108: Dangerous Behavior Detection Department

[0184] 109: Dangerous Behavior Notification Department

[0185] 111: Feature Extractor

[0186] 112: Restorer

[0187] 113: Classifier

[0188] 211A: Feature Extractor A

[0189] 211B: Feature Extractor B

[0190] 211C: Feature Extractor C

[0191] 212: Restorer

[0192] 213: Classifier

[0193] 1001: Control Department

[0194] 1002: Main storage unit

[0195] 1003: Auxiliary storage unit

[0196] 1004: Input unit

[0197] 1005: Output unit

[0198] 1006: Interface part.

Claims

1. A program that causes a computer to execute the following processing: Obtaining a surgical image showing the surgical site; detecting the dangerous behavior from the acquired surgical image using a model generated by machine learning on learning data whose input is the surgical image and whose output is information associated with the dangerous behavior; as well as Notification of detection of said dangerous behavior; The information associated with the dangerous behavior includes region information of a structure in the surgical image that changes due to the dangerous behavior.

2. The program according to claim 1, wherein: The region information of the structure in the surgical image that changes due to the dangerous behavior includes at least one of region information of the structure whose shape changes due to the dangerous behavior and region information of the structure that appears in the surgical image due to the dangerous behavior.

3. The program according to claim 1 or 2, wherein: The program causes the computer to execute a process of displaying, on the acquired surgical image, that the dangerous behavior has been detected.

4. The program according to claim 1 or 2, wherein: The learning data is generated using information detected by a sensor.

5. The program according to claim 1 or 2, wherein: The information associated with the risky behavior also includes information on structures that are not explicitly shown in the surgical image.

6. The program according to claim 1 or 2, wherein: The program causes the computer to execute processing for detecting the dangerous behavior also using information on the patient's state before surgery.

7. The program according to claim 1 or 2, wherein: The program causes the computer to execute a process of evaluating a surgical technique of an operator based on information associated with the risky behavior output by inputting the acquired surgical image into the model.

8. The program according to claim 1 or 2, wherein: The risky behavior is a behavior that causes recurrent laryngeal nerve paralysis.

9. A surgical assisting device comprising: a surgical image acquisition unit that acquires a surgical image in which a surgical site is photographed; a dangerous behavior detection unit that detects dangerous behavior from the acquired surgical image using a model generated by machine learning of learning data that has the surgical image as input and the information associated with the dangerous behavior as output; and a dangerous behavior notification unit for notifying that the dangerous behavior is detected, The information associated with the dangerous behavior includes region information of a structure in the surgical image that changes due to the dangerous behavior.

10. A method, which is a method performed by a surgical assisting device, comprising: The step of obtaining a surgical image having a surgical site; The step of detecting the dangerous behavior from the acquired surgical image using a model generated by performing machine learning on learning data whose input is the surgical image and whose output is information associated with the dangerous behavior; as well as the step of notifying a situation in which said dangerous behavior is detected, The information associated with the dangerous behavior includes region information of a structure in the surgical image that changes due to the dangerous behavior.

Images