Surgical assistance program, surgical assistance device, and surgical assistance method

Abstract

The purpose of the present invention is to notify the resection state of an object to be resected with high precision during surgery. A program according to one embodiment of the present invention causes a computer to execute: a process for acquiring a surgical image in which a surgical site is captured during a surgery in which an object to be resected is resected; a detection unit that detects a resection state of the object to be resected from the acquired surgical image using a model generated by machine learning of learning data that is input as the surgical image and output as information relating to resection completion; and notifying the state of resection of the object to be resected, wherein the information related to resection completion includes information on the region of the organ appearing in the surgical image when the object to be resected is resected.

Description

Technical Field

[0001] The present invention relates to a surgical assistance procedure, a surgical assistance device and a surgical assistance method. Background Art

[0002] Traditionally, surgical procedures, particularly for malignant tumors, have required not only tumor resection but also adequate removal (also known as clearance) of surrounding lymph nodes. Inadequate resection can lead to postoperative recurrence, ultimately worsening the patient's long-term prognosis. On the other hand, excessive resection can increase postoperative complications.

[0003] As a method for assisting in the proper resection of an object to be resected, for example, Patent Document 1 discloses a near-infrared fluorescent tracer and a fluorescence imaging method. Specifically, it is described that by chemically binding an anti-tumor antibody to an indocyanine green-high-density lipoprotein (ICG-HDL) complex, the location and size of an external tumor in vivo can be measured in real time in vitro. However, in the method of Patent Document 1, the operator is only able to distinguish between tumor tissue and normal tissue, and the determination of the completion of resection depends on the operator.

[0004] As a method for notifying completion of a manipulation using machine learning, for example, Non-Patent Document 1 discloses learning surgical images and information on completion of a manipulation, inputting surgical images during inference and outputting the status of completion of the manipulation.

[0005] <Prior Art Literature>

[0006] <Patent Document>

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 9-309845

[0008] <Non-patent literature>

[0009] Non-patent document 1: IRCAD, Artificial Intelligence for Surgical SafetyAutomatic Assessment of the Critical View of Safety in LaparoscopicCholecystectomy Using Deep Learning Summary of the Invention

[0010] <Problems to be Solved by the Invention>

[0011] However, in Non-Patent Document 1, surgical images and information on completion of a manipulation are merely learned and estimated, and it is expected that the status of the resection of the resection object can be notified with higher accuracy during surgery.

[0012] Therefore, an object of the present invention is to accurately notify the status of resection of an object to be resected during surgery.

[0013] <Methods used to solve the problem>

[0014] A program according to one embodiment of the present invention causes a computer to perform the following processing: obtaining a surgical image of a surgical site captured during an operation to remove an object to be removed; detecting the status of the removal of the object to be removed from the obtained surgical image using a model generated by machine learning of learning data having the surgical image as input and information associated with the completion of the removal as output; and notifying the computer of the status of the removal of the object to be removed, wherein the information associated with the completion of the removal includes regional information of an organ that appears in the surgical image when the object to be removed is removed.

[0015] According to the present invention, the status of the resection of the resection object can be notified with high accuracy during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a diagram showing the overall structure according to one embodiment of the present invention.

[0017] Figure 2 This is a functional block diagram of a surgical support device according to one embodiment of the present invention.

[0018] Figure 3 This is a diagram for explaining organ region information and labels according to one embodiment of the present invention.

[0019] Figure 4 This is an example of an organ that appears in a surgical image when the resection object according to one embodiment of the present invention is resected.

[0020] Figure 5 This is an example of an organ that appears in a surgical image when the resection object according to one embodiment of the present invention is resected.

[0021] Figure 6 This is an example of an organ that appears in a surgical image when the resection object according to one embodiment of the present invention is resected.

[0022] Figure 7 This is an example of an organ that appears in a surgical image when the resection object according to one embodiment of the present invention is resected.

[0023] Figure 8 This is a diagram for explaining learning according to one embodiment of the present invention.

[0024] Figure 9This is a diagram for explaining learning according to one embodiment of the present invention.

[0025] Figure 10 This is a flowchart of a learning process according to one embodiment of the present invention.

[0026] Figure 11 This is a flowchart of the resection status notification process according to one embodiment of the present invention.

[0027] Figure 12 This is a flowchart of the incomplete resection notification process according to one embodiment of the present invention.

[0028] Figure 13 This is a flowchart of the evaluation process according to one embodiment of the present invention.

[0029] Figure 14 This is an example of information and evaluation results that serve as a basis for evaluation according to one embodiment of the present invention.

[0030] Figure 15 This is a hardware configuration diagram of a surgical support device according to one embodiment of the present invention. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the invention will be described with reference to the drawings.

[0032] <Explanation of terms>

[0033] In this specification, "surgery" can refer to any procedure, including surgery performed solely by humans, surgery performed by humans operating medical equipment such as an endoscope (e.g., laparoscopic surgery, thoracoscopic surgery, etc.), surgery performed by humans operating a robot, or surgery performed solely by a robot. For example, a surgery is a surgical procedure to remove cancerous tissue, including the primary tumor and lymph nodes. Examples of such procedures include esophagectomy, gastrectomy, colectomy, prostatectomy, and pancreatectomy, as well as endoscopic submucosal dissection for early-stage cancer.

[0034] In this specification, "object to be removed" refers to an object to be removed during surgery (e.g., a lesion). For example, the object to be removed may be a malignant tumor and lymph nodes surrounding the tumor, but may also be a benign tumor.

[0035] In this specification, an "organ" is any organ located in the body.

[0036] System Structure

[0037] Figure 1This figure shows the overall structure of one embodiment of the present invention. A surgery support system 1 includes a surgery support device 10, an imaging device 20, and a display device 30. A description will be given of a situation in which a doctor 11 performs surgery on a patient 12 while performing surgery on the object to be removed. The following sections will describe each of these.

[0038] <<Surgical Assistance Device>>

[0039] The surgical support device 10 is a device that detects the status of the resection of a patient 12 based on images captured by an imaging device 20 during surgery. The device then notifies the doctor 11 of the status of the resection (e.g., by displaying it on a display device 30). The surgical support device 10 is comprised of one or more computers. The device can obtain images captured by the imaging device 20 from the imaging device 20. Furthermore, the device can display information such as the status of the resection of the patient 12 on the display device 30.

[0040] <<Camera equipment>>

[0041] The imaging device 20 is a device that captures an image of a region where surgery is performed on the patient 12 (hereinafter also referred to as a surgical region). Hereinafter, an image of the surgical region captured by the imaging device 20 is also referred to as a surgical image.

[0042] <<Display device>>

[0043] The display device 30 is a monitor or display that displays information acquired from the surgery support device 10. The doctor 11 can observe the information displayed on the display device 30 to learn the status of the resection of the object (eg, completion of the resection).

[0044] In addition, at least two of the surgery support device 10 , the imaging device 20 , and the display device 30 may be realized by a single device.

[0045] <Function box>

[0046] Figure 2 This is a functional block diagram of the surgical support device 10 according to one embodiment of the present invention.

[0047] The surgical support device 10 includes a learning data storage unit 101, a learning data acquisition unit 102, a resection status detection unit 103, an error calculation unit 104, a parameter update unit 105, a parameter storage unit 106, a surgical image acquisition unit 107, a resection status detection unit 108, and a resection status notification unit 109. The surgical support device 10 functions as the learning data acquisition unit 102, the resection status detection unit 103, the error calculation unit 104, the parameter update unit 105, the surgical image acquisition unit 107, the resection status detection unit 108, and the resection status notification unit 109 by executing a program. Each of these functions will be described below.

[0048] Learning data is stored in the learning data storage unit 101. The learning data is a surgical image showing a surgical site and information related to the completion of resection in the surgical image.

[0049] The learning data acquisition unit 102 acquires the learning data stored in the learning data storage unit 101 .

[0050] Here, the "information associated with the completion of resection" is explained. The "information associated with the completion of resection" includes: "regional information of the organ that appears in the surgical image when the resection object is resected (in addition, the region information is at least one of segmentation information representing the region of the organ in the image (information representing the segmented organ), and frame information representing the region of the organ in the image (information such as a rectangle surrounding the organ))"; and "a label indicating whether the surgical image is an image in which the resection of the resection object is completed or an image in which the resection of the resection object is not completed". hereinafter, with reference to Figure 3 Provide detailed information about organ regions and labels.

[0051] Figure 3 It is a diagram for explaining the regional information and labels of the organs involved in one embodiment of the present invention. For example, when the resection object is resected, "trachea", "subclavian artery" and "nerve" appear in the surgical image. Surgical images 1 to 3 are examples of learning data (correct answer data). In addition, as learning data, the regional information of the resection object (at least one of the segmentation information indicating the region of the resection object in the image (information indicating the segmented resection object) and the frame information indicating the region of the resection object in the image (information such as a rectangle surrounding the resection object)) can also be used.

[0052] The surgical image 1 is an image obtained by photographing the surgical site before the object to be resected is resected. Figure 3 In the example, "trachea", "subclavian artery", and "nerve" are not yet visible. The label of surgical image 1 indicates that the surgical image is an image in which the resection of the resection object is not completed.

[0053] Surgical image 2 is an image of the surgical site midway through resection of the resected object. Surgical image 2 includes information about the region of the organ that appears midway through resection. The label for surgical image 2 indicates that the surgical image represents an image of an incomplete resection of the resected object.

[0054] Surgical image 3 is an image of the surgical site after the object has been removed. Surgical image 3 is assigned information about the region of the organ that appears after the resection is complete. The label for surgical image 3 indicates that the surgical image is an image of the completed resection of the object. Furthermore, the point in time when the organ that should appear in the surgical image upon completion of the resection is recognizable is defined as the time of resection completion.

[0055] Alternatively, the organ region information may be generated by a human specifying an organ region within a surgical image, or by inference using a machine learning model using a surgical image in which the organ region has been specified by a human.

[0056] Figure 4 It is an example of an organ that appears in a surgical image when the resection object involved in one embodiment of the present invention is resected. In gastrectomy, when the resection object (tumor and lymph node) is resected, the transverse mesocolon, pancreas, right gastroepiploic vein, right gastroepiploic artery, subpyloric artery and duodenum appear in the surgical image. That is, "information associated with the completion of resection" can include regional information of at least one of the transverse mesocolon, pancreas, right gastroepiploic vein, right gastroepiploic artery, subpyloric artery and duodenum. In this way, in gastrectomy, the condition of resection of the resection object (for example, whether the resection of the resection object is completed or incomplete, or the ratio of completed or unresected parts of the resection object) can be determined by the exposure of at least one of the transverse mesocolon, pancreas, right gastroepiploic vein, right gastroepiploic artery, subpyloric artery and duodenum.

[0057] Figure 5 This is an example of an organ that appears in a surgical image when the resection object involved in one embodiment of the present invention is resected. In colectomy, when the resection object (tumor and lymph node) is resected, the abdominal aorta and the inferior mesenteric artery appear in the surgical image. That is, "information associated with the completion of resection" can include regional information of at least one of the abdominal aorta and the inferior mesenteric artery. In this way, in colectomy, the status of resection of the resection object (for example, whether the resection of the resection object is completed or incomplete, or the ratio of completed or unresected parts of the resection object) can be determined by the exposure of at least one of the abdominal aorta and the inferior mesenteric artery.

[0058] Figure 6This is an example of an organ that appears in a surgical image when the resection object involved in one embodiment of the present invention is resected. In bladder neck resection during prostatectomy, when the resection object is resected, the prostate, bladder, urethra, seminal vesicle, and vas deferens appear in the surgical image. That is, "information associated with the completion of resection" can include regional information of at least one of the prostate, bladder, urethra, seminal vesicle, and vas deferens. In this way, during prostatectomy, the status of resection of the resection object (for example, whether the resection of the resection object is completed or incomplete, or the ratio of completed or unresected resection objects) can be determined by the exposure of at least one of the prostate, bladder, urethra, seminal vesicle, and vas deferens.

[0059] Figure 7 This is an example of an organ that appears in a surgical image when the resection object involved in one embodiment of the present invention is resected. During pancreatic resection, when the resection object (tumor and lymph nodes) is resected, the superior mesenteric vein appears in the surgical image. In other words, "information associated with resection completion" can include regional information about the superior mesenteric vein. In this way, during pancreatic resection, the exposure of the superior mesenteric vein can be used to determine the status of resection of the resection object (for example, whether resection of the resection object is complete or incomplete, or the percentage of resection objects that are complete or incomplete).

[0060] Back to Figure 2 The resection condition detection unit 103 detects the resection condition of the resection object using the model being learned.

[0061] [Example 1]

[0062] In use Figure 8 In the first embodiment described above, the resection status detection unit 103 inputs the surgical image acquired by the learning data acquisition unit 102 into the model being learned ( Figure 8 The feature extractor 111 and restorer 112 of the learning data acquisition unit 102 are used to output the region information of the organ. In addition, the resection status detection unit 103 inputs the surgical image obtained by the learning data acquisition unit 102 into the model being learned ( Figure 8 The feature extractor 111 and the classifier 113 are used to output whether the resection of the resected object is completed or not.

[0063] [Example 2]

[0064] In use Figure 9 In the second embodiment described above, the resection status detection unit 103 inputs the surgical image acquired by the learning data acquisition unit 102 into the model being learned ( Figure 9The feature extractor A211A and restorer 212 of the learning data acquisition unit 102 are used to output the region information of the organ. In addition, the resection condition detection unit 103 inputs the surgical image obtained by the learning data acquisition unit 102 and the region information of the organ restored by the restorer 212 into the model being learned ( Figure 9 The feature extractor B211B, the feature extractor C211C and the classifier 213 are used to output whether the resection of the resected object is completed or not.

[0065] The error calculation unit 104 calculates the error between the state of resection of the resection object detected from the surgical image by the resection state detection unit 103 and the information related to resection completion in the surgical image acquired by the learning data acquisition unit 102 .

[0066] The parameter updating unit 105 updates the parameters of the model being learned so as to reduce the error calculated by the error calculating unit 104 , and stores the updated parameters in the parameter storage unit 106 .

[0067] The parameter storage unit 106 stores the parameters of the model updated by the parameter updating unit 105 .

[0068] Below, refer to Figure 8 (Example 1) and Figure 9 (Example 2) The learning process is described in detail.

[0069] Figure 8 This figure illustrates learning according to one embodiment of the present invention (Example 1). Feature extractor 111 extracts features from surgical images serving as learning data. Reconstructor 112 uses these features to reconstruct organ region information or surgical images containing organ region information. Classifier 113 uses these features to determine whether resection of the object is complete or incomplete. Specifically, in Example 1, during inference, classifier 113 uses the features extracted by feature extractor 111 during learning to classify surgical images. Each of these is described below.

[0070] The feature extractor (also referred to as an encoder) 111 extracts features of the surgical image from the surgical image. Specifically, the feature extractor 111 performs dimensionality reduction on the surgical image to extract features.

[0071] The restorer (also called a decoder) 112 restores the regional information of the organ or the surgical image containing the regional information of the organ based on the feature quantity of the surgical image extracted by the feature extractor 111. (Note that the restored regional information of the organ may be only the regional information of the organ that appears when the resection object is resected, or may include not only the regional information of the organ that appears when the resection object is resected but also the regional information of other organs.) The error between the restored result by the restorer 112 and the regional information of the organ in the learning data (correct answer data) (i.e., the regional information of the organ that appears in the surgical image when the resection object is resected) is calculated, and learning is performed (i.e., parameters are updated) to minimize the error.

[0072] Classifier 113 determines whether the surgical image shows complete or incomplete resection of the object being resected based on the feature values ​​of the surgical image extracted by feature extractor 111. The error between the result of the determination by classifier 113 and the label of the learning data (correct answer data) (i.e., the label indicating complete resection or the label indicating incomplete resection) is calculated, and learning (i.e., updating parameters) is performed to minimize the error.

[0073] Figure 9 This is a diagram used to illustrate learning according to one embodiment of the present invention (Example 2). Feature extractor A 211A extracts feature quantities from a surgical image serving as learning data, and restorer 212 restores organ region information or a surgical image containing organ region information based on the feature quantities. Feature extractor B 211B extracts feature quantities from the organ region information or the surgical image containing organ region information restored by restorer 212. Feature extractor C 211C extracts feature quantities from the surgical image serving as learning data. Classifier 213 determines whether resection of the object to be resected is complete or incomplete based on a feature quantity obtained by combining the feature quantities extracted by feature extractor B 211B with the feature quantities extracted by feature extractor C 211C. That is, in Example 2, during inference, feature extractor A 211A and restorer 212 restore organ region information from a surgical image, and classifier 213 uses the feature quantities of the surgical image and the restored organ region information to classify the surgical image. Each of these is described below.

[0074] The feature extractor (also called encoder) A211A extracts features of the surgical image from the surgical image. Specifically, the feature extractor A211A performs dimensionality reduction on the surgical image to extract features.

[0075] The restorer (also called a decoder) 212 restores the regional information of the organ or the surgical image containing the regional information of the organ based on the feature quantity of the surgical image extracted by the feature extractor A211A. (Note that the restored regional information of the organ may be only the regional information of the organ that appears when the resection object is resected, or may include not only the regional information of the organ that appears when the resection object is resected but also the regional information of other organs.) The error between the restored result by the restorer 212 and the regional information of the organ in the learning data (correct answer data) (i.e., the regional information of the organ that appears in the surgical image when the resection object is resected) is calculated, and learning is performed (i.e., parameters are updated) in a manner that reduces the error.

[0076] The feature extractor (also called encoder) B211B extracts feature quantities of the organ region information from the organ region information restored by the restorer 212. Specifically, the feature extractor B211B performs dimensionality reduction on the organ region information restored by the restorer 212 to extract feature quantities.

[0077] The feature extractor (also called encoder) C211C extracts features of the surgical image from the surgical image. Specifically, the feature extractor C211C performs dimensionality reduction on the surgical image to extract features.

[0078] Classifier 213 determines whether the surgical image shows complete or incomplete resection of the object based on a feature quantity obtained by combining the feature quantity extracted by feature extractor B 211B and the feature quantity extracted by feature extractor C 211C. The error between the result determined by classifier 213 and the label of the learning data (correct answer data) (i.e., the label indicating complete resection or the label indicating incomplete resection) is calculated, and learning is performed (i.e., parameters are updated) in a manner that reduces the error.

[0079] By learning in this way, the surgical assistance device 10 can reflect the degree of recognition and positional relationship of the resection object and surrounding organs corresponding to the progress of the resection in the detection of the resection status. For example, in the resection of lymph nodes around the right recurrent laryngeal nerve, the status of the lymph node resection can be determined based on the segmentation results of the right and posterior walls of the trachea, the subclavian artery, and the recurrent laryngeal nerve. That is, in the present invention, the results of the segmentation or frame information of the anatomical structure (i.e., the regional information of each organ) are used as attention to detect the resection status of the resection object, so that the information of each organ can help determine which part should be resected and which part should be retained without resection.

[0080] Furthermore, when machine learning was performed without using organ region information, the model's inference accuracy was 66.6%, while when machine learning was performed using organ region information, the model's inference accuracy was 81.3%. Thus, using organ region information can improve model inference accuracy.

[0081] Return to Figure 2 The surgical image acquisition unit 107 acquires surgical images (eg, moving images) captured by the imaging device 20 during surgery.

[0082] The resection status detection unit 108 detects the status of resection of the resection object based on the surgical image acquired by the surgical image acquisition unit 107. Specifically, the resection status detection unit 108 inputs the surgical image acquired by the surgical image acquisition unit 107 into a model (specifically, a model using the parameters stored in the parameter storage unit 106) and outputs the status of resection of the resection object (e.g., whether resection of the resection object is complete or incomplete, or the ratio of complete resection or incomplete resection of the resection object). Alternatively, a classification model may be used to output the probability of complete resection of the resection object (i.e., the complete resection ratio) or the probability of incomplete resection of the resection object (i.e., the incomplete resection ratio), or a regression model may be used to output the ratio of complete resection or incomplete resection.

[0083] The resection status notification unit 109 notifies the operator (e.g., displays on the display device 30) in real time the status of the resection of the resection object detected by the resection status detection unit 108. For example, the resection status notification unit 109 notifies the operator of the completion of resection of the resection object (i.e., using two values: resection completed or not completed), or notifies the operator of the percentage of resection completed or not resected within the resection object (i.e., expressing the resection status as a percentage). For example, the resection status notification unit 109 displays the status of resection of the resection object on the surgical image acquired by the surgical image acquisition unit 107. For example, the resection status notification unit 109 displays a message such as "Resection completed."

[0084] <Handling Method>

[0085] Below, refer to Figure 10 For an explanation of the learning process, refer to Figure 11 The method of processing the resection status notification will be described.

[0086] Figure 10 This is a flowchart of a learning process according to one embodiment of the present invention.

[0087] In step 101 ( S101 ), the surgery support device 10 acquires a surgical image as learning data.

[0088] In step 102 ( S102 ), the surgery support device 10 detects the status of the resection of the resection object using the model being learned.

[0089] In use Figure 8 In the first embodiment described above, the surgery support device 10 inputs the surgical image acquired in S101 into the model being learned ( Figure 8 The feature extractor 111 and restorer 112 of the device are used to output the region information of the organ. In addition, the surgical support device 10 inputs the surgical image obtained in S101 into the model being learned ( Figure 8 The feature extractor 111 and the classifier 113 are used to output whether the resection of the resected object is completed or not.

[0090] In use Figure 9 In the second embodiment described above, the surgery support device 10 inputs the surgical image acquired in S101 into the model being learned ( Figure 9 The surgical support device 10 inputs the surgical image obtained in S101 and the regional information of the organ restored by the restorer 212 into the model being learned ( Figure 9 The feature extractor B211B, the feature extractor C211C and the classifier 213 are used to output whether the resection of the resected object is completed or not.

[0091] In step 111 (S111), the surgical assistance device 10 obtains information associated with the completion of resection (specifically, regional information of the organ appearing in the surgical image when the resection object is resected, and a label indicating whether the surgical image is an image in which resection of the resection object is completed or an image in which resection of the resection object is not completed) of the surgical image serving as learning data (i.e., the surgical image obtained in S101).

[0092] In step 112 (S112), the surgery support device 10 converts the data acquired in S111 into correct answer data. Specifically, the surgery support device 10 makes the data acquired in S111 consistent with the output format of the model.

[0093] In step 103 ( S103 ), the surgery support device 10 calculates the error between the state of resection of the resection object detected in S102 and the correct answer data in S112 .

[0094] In step 104 ( S104 ), the surgery support device 10 updates the parameters of the model being learned so as to reduce the error calculated in S103 , and stores the updated parameters in the parameter storage unit 106 .

[0095] In step 105 (S105), the surgery support device 10 determines whether learning is completed. If learning is not completed, the process returns to S101 and S111 to acquire the next learning data. If learning is completed, the present process ends.

[0096] Figure 11 This is a flowchart of the resection status notification process according to one embodiment of the present invention.

[0097] In step 201 ( S201 ), the surgery support device 10 acquires a surgical image (for example, a moving image) captured by the imaging device 20 during surgery.

[0098] In step 202 (S202), the surgical support device 10 detects the state of the resection object based on the surgical image acquired in S201. Specifically, the surgical support device 10 inputs the surgical image acquired in S201 into the Figure 10 The model generated in the embodiment outputs the status of the resection of the resection object (for example, whether the resection of the resection object is completed or incomplete, or the ratio of the resection object to be resected or not).

[0099] In use Figure 8 In the first embodiment described above, the classifier 113 classifies the surgical image using the feature quantity extracted by the feature extractor 111 during learning.

[0100] In use Figure 9 In the second embodiment described above, the feature extractor A211A and the restorer 212 restore organ region information from the surgical image, and the classifier 213 classifies the surgical image using the feature values ​​of the surgical image and the restored feature values ​​of the organ region information.

[0101] In step 203 (S203), the surgical support device 10 notifies the patient of the status of the resection of the object detected in S202 (e.g., by displaying it on the display device 30) in real time. For example, the surgical support device 10 notifies the patient of the completion of resection of the object or the percentage of the object that has been resected or not resected. For example, the surgical support device 10 displays the status of the resection of the object on the surgical image acquired in S201.

[0102] Effects

[0103] In one embodiment of the present invention, during a surgical procedure, when the removal of cancerous tissue, such as a primary lesion or lymph node, is complete, the surgeon is notified of the completion of the surgical procedure in real time. This notification, clearly displayed in the surgical image, allows the surgeon to fully remove the lymph nodes while avoiding complications.

[0104] Other embodiments will be described below. In addition, a plurality of the embodiments described below may be implemented in combination.

[0105] [Notice of unresected areas]

[0106] The surgical support device 10 can notify the operator of the unresected portion of the resection object when resection of the resection object is incomplete. Specifically, the surgical support device 10 can identify the unresected portion of the resection object within the surgical image captured by the imaging device 20 during surgery and notify the operator of the identified portion. (For example, the surgical support device 10 can determine the unresected portion of the resection object based on the segmentation information output from the restorer 112 and the rule base. For example, if a "nerve" (an example of an organ that appears in the surgical image when the resection object is resected) is not visible in the surgical image (i.e., if segmentation information for the "nerve" is not available), the portion where the "nerve" should appear can be identified as the unresected portion.) For example, the surgical support device 10 provides notification when the operator advances from a surgical resection step to the next step or during a resection step. For example, the surgical support device 10 displays a message such as "Insufficient dissection around the trachea" on the display device 30.

[0107] By being informed of which part of the body has not been completely excised, the operator can be informed of the next surgical procedure to be performed.

[0108] [Notice of unfinished removal]

[0109] When the excision of the object is not completed, if the operator advances from the excision step of the operation to the next step, the surgery support device 10 can notify the operator to return to the excision step. Figure 12 The processing of the incomplete resection notification will be described. In addition, it is assumed that each step in the operation (for example, the resection step, etc.) is managed.

[0110] Figure 12 This is a flowchart of the incomplete resection notification process according to one embodiment of the present invention.

[0111] In step 301 ( S301 ), the surgery support device 10 acquires a surgical image captured by the imaging device 20 during surgery.

[0112] In step 302 ( S302 ), the surgery support device 10 detects the status of the resection object based on the surgical image acquired in S301 .

[0113] In step 303 ( S303 ), the surgery support device 10 obtains information on which step of each step in the surgery the current step is.

[0114] In step 304 (S304), the surgical support device 10 determines whether the resection of the resection object is not completed and the current step is not a resection step. If the resection of the resection object is not completed and the current step is not a resection step, the process proceeds to step 305. In other cases, the process ends.

[0115] In step 305 ( S305 ), the surgery support device 10 notifies the user to return to the resection procedure. For example, the surgery support device 10 displays a message such as “Please return to the previous resection procedure” on the display device 30 .

[0116] By notifying the user to return to the excision step in this manner, it is possible to prevent the procedure from being terminated before sufficient excision is completed.

[0117] [Management of resection time]

[0118] If a certain time has passed since the start of the surgical resection step, and resection of the object to be resected is not complete, the surgical support device 10 can notify the operator that the certain time has elapsed. For example, the certain time can be the average time of past data or a time exceeding the average time of past data + 2SD (standard deviation). For example, the surgical support device 10 may display a message such as "A certain time has elapsed since the start of resection" on the display device 30.

[0119] By managing the time of the excision step in this way, it is possible to prevent the surgical procedure from being completed before sufficient excision is completed, and it is also possible to predict complications and to know the timing for changing the operator to a senior doctor.

[0120] [Evaluation of Manipulation]

[0121] The surgical support device 10 can evaluate the manipulation of the operator such as a surgeon or endoscopist based on the information related to the completion of the resection outputted by inputting the surgical image into the model. Figure 13 and Figure 14 Describe the evaluation of manual manipulation.

[0122] Figure 13 This is a flowchart of the evaluation process according to one embodiment of the present invention.

[0123] In step 401 ( S401 ), the surgery support device 10 acquires information related to resection completion output by inputting a surgical image into a model.

[0124] In step 402 (S402), the surgery support device 10 obtains information related to the surgery. The information related to the surgery may include the total surgery time, the surgery time at each stage, data identifying the movement of the instrument or operator, and the like.

[0125] In step 403 (S403), the surgery support device 10 evaluates the operator's manual operation based on the information acquired in S401 and S402. For example, the surgery support device 10 may use machine learning to evaluate the operator's manual operation.

[0126] Figure 14 This is an example of information and evaluation results used as a basis for evaluation according to one embodiment of the present invention. For example, the information used as a basis for evaluation may include the time of each stage of the surgery (e.g., lymph node resection around the recurrent laryngeal nerve), the type of resection status (e.g., resection completion), etc. The evaluation result may be a score, for example.

[0127] By evaluating the surgical manipulation in this way, it is possible to provide feedback on the surgical manipulation after surgery or use it for surgical education.

[0128] <Hardware Structure>

[0129] Figure 15 This is a hardware configuration diagram of a surgical support device 10 according to one embodiment of the present invention. The surgical support device 10 may include a control unit 1001, a main storage unit 1002, an auxiliary storage unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0130] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003 .

[0131] The main storage unit 1002 includes nonvolatile memory (ROM (Read Only Memory)) and volatile memory (RAM (Random Access Memory)). The ROM stores various programs and data required for the control unit 1001 to execute various programs installed in the auxiliary storage unit 1003. The RAM provides a work area used by the control unit 1001 when executing various programs installed in the auxiliary storage unit 1003.

[0132] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when executing the various programs.

[0133] The input unit 1004 is an input device used by the operator of the surgery support apparatus 10 to input various instructions to the surgery support apparatus 10 .

[0134] The output unit 1005 is an output device that outputs the internal state of the surgery support apparatus 10 and the like.

[0135] The interface unit 1006 is a communication device connected to a network for communicating with other devices.

[0136] While the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and changes can be made within the scope of the gist of the present invention described in the claims.

[0137] This application claims priority based on Japanese Patent Application No. 2022-143971 filed on September 9, 2022, the entire contents of which are incorporated herein by reference.

[0138] Explanation of symbols

[0139] 1: Surgical assistance system

[0140] 10: Surgical assist devices

[0141] 20: Camera

[0142] 30: Display device

[0143] 11: Doctor

[0144] 12: Patient

[0145] 101: Learning Data Storage

[0146] 102: Learning data acquisition unit

[0147] 103: Resection status detection unit

[0148] 104: Error calculation unit

[0149] 105: Parameter update unit

[0150] 106: Parameter storage unit

[0151] 107: Surgical Image Acquisition Department

[0152] 108: Resection status detection unit

[0153] 109: Resection Status Notification Department

[0154] 111: Feature Extractor

[0155] 112: Restorer

[0156] 113: Classifier

[0157] 211A: Feature Extractor A

[0158] 211B: Feature Extractor B

[0159] 211C: Feature Extractor C

[0160] 212: Reducer

[0161] 213: Classifier

[0162] 1001: Control Department

[0163] 1002: Main storage unit

[0164] 1003: Auxiliary storage unit

[0165] 1004: Input unit

[0166] 1005: Output unit

[0167] 1006: Interface department.

Claims

1. A program that causes a computer to execute the following processing: Acquiring a surgical image of a surgical site captured during surgery to remove an object to be removed; detecting the state of resection of the resection object from the acquired surgical image using a model generated by machine learning of learning data whose input is a surgical image and whose output is information associated with resection completion; and notifying the state of the excision of the excision object, The information related to the completion of resection includes region information of an organ that appears in the surgical image when the resection object is resected.

2. The program according to claim 1, wherein: The notification of the resection status of the resection object includes notification that the resection of the resection object has been completed, or notification of the ratio of the resection object that has been resected or the ratio of the object that has not been resected.

3. The program according to claim 1 or 2, wherein: The program causes the computer to execute a process of displaying a state of resection of the resection object on the acquired surgical image.

4. The program according to claim 1 or 2, wherein: The program causes the computer to execute a process of notifying an uncut portion of the resection object when resection of the resection object is not completed.

5. The program according to claim 1 or 2, wherein: The program causes the computer to execute processing for notifying a return to the resection step when the operator proceeds from the resection step of the surgery to the next step when the resection of the resection object is not completed.

6. The program according to claim 1 or 2, wherein: The program causes the computer to execute processing for notifying that the resection step of the surgery has elapsed for a certain period of time if resection of the resection object is not completed when the resection step of the surgery has elapsed for a certain period of time or more.

7. The program according to claim 1 or 2, wherein: The program causes the computer to execute a process of evaluating a surgical technique performed by an operator based on the information associated with the resection completion output by inputting the acquired surgical image into the model.

8. A surgical assisting device, comprising: a surgical image acquisition unit that acquires a surgical image that captures a surgical site during a surgery to remove a resection object; a resection condition detection unit that detects a resection condition of the resection object from the acquired surgical image using a model generated by machine learning of learning data that has a surgical image as input and outputs information associated with completion of resection; as well as a resection status notification unit for notifying the resection status of the resection object, The information related to the completion of resection includes region information of an organ that appears in the surgical image when the resection object is resected.

9. A method, performed by a surgical assisting device, comprising: A step of obtaining a surgical image of a surgical site captured during a surgery to remove an object to be removed; A step of detecting the state of resection of the resection object from the acquired surgical image using a model generated by machine learning of learning data whose input is a surgical image and whose output is information related to resection completion; as well as a step of notifying the state of the excision of the excision object, The information related to the completion of resection includes region information of an organ that appears in the surgical image when the resection object is resected.

Images