Surgery assistance device, learning device, surgery assistance method, learning model generation method, and program

Abstract

This surgery assistance device comprises: an inference unit that inputs a surgical field image including an image of a vascular tissue to be anastomosed to a trained model as an inference target image, acquires coordinates of a stump of the vascular tissue in the surgical field image from the trained model as one inference result, and further acquires intimal damage information of the vascular tissue from the trained model as one inference result; an assistance image processing unit that superimposes an image indicating a region of the stump and an image indicating intimal damage information in the surgical field image on the surgical field image on the basis of the coordinates of the stump and the intimal damage information that have been acquired, and generates an advice image including a character string image according to a damage level of the intimal damage information, to generate an assistance image for an anastomosis surgery of the vascular tissue; and an output unit that causes a display device to display the assistance image.

Description

Surgical Support Device, Learning Device, Surgical Support Method, Learning Model Generation Method, and Program

[0001] The present invention relates to a surgical support device, a learning device, a surgical support method, a learning model generation method, and a program. This application claims priority based on Japanese Patent Application No. 2024-178695 filed in Japan on October 11, 2024, and incorporates its content herein by reference.

[0002] In recent years, with the improvement of vascular anastomosis surgery techniques, transplantation of various tissues has become possible, enabling better life prognosis, functional prognosis, and quality of life (QOL) to be provided to various patients. Nationally, the success rate of vascular anastomosis surgery is about 97%. In contrast, at the Department of Plastic Surgery of Nihon University School of Medicine, the success rate of vascular anastomosis surgery exceeds 99%.

[0003] Further, Patent Document 1 discloses a surgical support device that recognizes blood vessels in an intraoperative field image of endoscopic surgery that should prompt attention. The surgical support device disclosed in Patent Document 1 recognizes blood vessels that should prompt attention by inputting an intraoperative field image into a learning model of machine learning.

[0004] Japanese Patent No. 7146318 Gazette

[0005] Vascular anastomosis surgery requires a high level of skill. However, the skill level of anastomosis surgery can vary significantly among surgeons, and conventional surgical support devices may be able to display blood vessels that should prompt attention but may not be able to support anastomosis surgery in some cases.

[0006] In view of the above circumstances, an object of the present invention is to provide a surgical support device, a learning device, a surgical support method, a learning model generation method, and a program that can support anastomosis surgery.

[0007] One aspect of the present invention is a surgical support device comprising: an inference unit that inputs a surgical field image including an image of vascular tissue to be anastomosed as an inference target image to a trained model, obtains the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results, and further obtains intimal damage information of the vascular tissue from the trained model as one of the inference results; an image processing unit that generates an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intimal damage information onto the surgical field image based on the obtained coordinates of the stump and the intimal damage information, and generates an advice image including a string image according to the damage level of the intimal damage information; and an output unit that displays the support image on a display device.

[0008] One aspect of the present invention is a learning device comprising: a preprocessing unit that associates a surgical field image including an image of vascular tissue to be anastomosed with a correct label indicating coordinates of the stump of the vascular tissue and damage information of the intima of the vascular tissue; and a learning unit that inputs the surgical field image associated with the correct label indicating coordinates of the stump and damage information of the intima as a training image into a learning model, obtains the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results, further obtains the intima damage information from the learning model as one of the inference results, and generates a trained model from the learning model based on a comparison result between the coordinates of the stump obtained from the learning model and the correct label indicating the coordinates of the stump, and a comparison result between the intima damage information obtained from the learning model and the correct label indicating the intima damage information.

[0009] One aspect of the present invention is a surgical support method performed by a surgical support device, comprising the steps of: inputting a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model; obtaining the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results; and further obtaining information on the intima of the vascular tissue's damage from the trained model as one of the inference results; generating an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intima damage information onto the surgical field image based on the obtained coordinates of the stump and the intima damage information, and generating an advice image including a string image according to the damage level of the intima damage information; and displaying the support image on a display device.

[0010] One aspect of the present invention is a learning model generation method performed by a learning device, comprising the steps of: associating a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and information on intima damage to the vascular tissue; inputting the surgical field image, which has been associated with the ground truth label indicating the coordinates of the stump and information on intima damage to the vascular tissue, as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results; further obtaining the intima damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on a comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and a comparison result between the intima damage information obtained from the learning model and the ground truth label indicating the intima damage information.

[0011] One aspect of the present invention is a program for causing a computer to perform the following steps: input a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model; obtain the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results; and further obtain information on intima damage to the vascular tissue from the trained model as one of the inference results; generate an image to support anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intima damage information onto the surgical field image based on the obtained coordinates of the stump and the intima damage information, and generating an advice image including a string image according to the level of damage of the intima damage information; and display the support image on a display device.

[0012] One aspect of the present invention is a program for causing a computer to perform the following steps: a procedure for associating a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and information on intima damage to the vascular tissue; inputting the surgical field image, which has been associated with the ground truth label indicating the coordinates of the stump and information on intima damage to the vascular tissue, as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results, further obtaining the intima damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and the comparison result between the intima damage information obtained from the learning model and the ground truth label indicating the intima damage information.

[0013] This invention makes it possible to support anastomosis surgery.

[0014] This figure shows an example of the configuration of the surgical support system in the first embodiment. This figure shows a first example of a surgical field image (training image) in the first embodiment. This figure shows a second example of a surgical field image (training image) in the first embodiment. This figure shows a third example of a surgical field image (training image) in the first embodiment. This figure shows a fourth example of a surgical field image (training image) in the first embodiment. This figure shows an example of a surgical field image (teacher image including training image and correct label) in the first embodiment. This figure shows an example of a surgical field image (teacher image including verification image and correct label) in the first embodiment. This figure shows an example of a surgical field image (teacher image including evaluation image and correct label) in the first embodiment. This figure shows a first example of a surgical field image (inference target image) in the first embodiment. This figure shows a first example of a support image, which is a surgical field image with an image based on the inference result superimposed on it, in the first embodiment. This figure shows a second example of a surgical field image (inference target image) in the first embodiment. This figure shows a second example of a support image, which is a surgical field image with an image based on the inference result superimposed on it, in the first embodiment. This flowchart shows an example of the operation of the learning device in the first embodiment. This is a flowchart showing an example of the operation of the surgical support device in the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the first modified example of the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the second modified example of the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the third modified example of the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the fourth modified example of the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the fifth data table representing the correspondence between the injury level and the advice, in the sixth modified example of the first embodiment. This is a diagram showing an example of a support image, which is a surgical field image with an advice image based on the inference result superimposed, in the sixth modified example of the first embodiment.This figure shows an example of the configuration of the surgical support system in the second embodiment. This figure shows examples of support images in the second embodiment, including a surgical field image (for the surgeon's left eye) without an image based on the inference result superimposed, and a surgical field image (for the surgeon's right eye) with an image based on the inference result superimposed. This figure shows examples of support images in the second embodiment, including a surgical field image (for the surgeon's left eye) without an image based on the inference result superimposed, and a surgical field image (for the surgeon's right eye) with the image based on the inference result removed. This figure shows the profiles of each physician who used the vascular anastomosis support program. This figure shows examples of evaluation results regarding the operability of the vascular anastomosis support program. This figure shows examples of evaluation results regarding the accuracy of the vascular anastomosis support program. This figure shows examples of evaluation results regarding the usefulness of the vascular anastomosis support program.

[0015] Embodiments of the present invention will be described in detail with reference to the drawings. (First Embodiment) Figure 1 is a diagram showing an example of the configuration of the surgical support system 1a in the first embodiment. The surgical support system 1a is a system that supports anastomosis surgery of tubular tissue. The tubular tissue is not limited to any particular tissue as long as it has a lumen, but for example, it is a blood vessel or the esophagus. Hereinafter, the tubular tissue is a blood vessel (vascular tissue) as an example.

[0016] The surgical support system 1a pre-runs a learning model having a neural network before performing vascular anastomosis surgery. Hereinafter, the stage in which machine learning is performed is referred to as the "learning stage." The surgical support system 1a performs machine learning of the learning model using, for example, training images that have been pre-generated based on surgical field images from past anastomosis surgeries. The learning model is not limited to a specific machine learning model, but for example, it could be a YOLO (You Only Look Once) learning model. A loss function may be predetermined for the machine learning of the learning model. In the learning stage, as a result of machine learning of the learning model, a trained model is generated from the learning model.

[0017] During the learning phase, the surgical support system 1a may verify the performance of the trained model using validation images generated in advance based on surgical field images from past anastomosis surgeries. The surgical support system 1a may adjust the hyperparameters based on instructions from the user who has confirmed the validation results. Examples of hyperparameters include the number of training iterations (epochs), the learning rate, the threshold, the batch size, the number of neuron layers, and the number of neurons per layer.

[0018] In the next stage after the learning stage, the surgical support system 1a may evaluate the versatility of the learned model using evaluation images (test images) that have been generated in advance based on surgical field images from past anastomosis surgeries. Hereinafter, the stage in which versatility is evaluated will be referred to as the "evaluation stage." The versatility of the learned model is improved by adjusting the parameters (neuron weighting) of the learned model based on the evaluation results of versatility. The learning stage and the evaluation stage may each be repeated in a predetermined order.

[0019] In the stage following the learning or evaluation stage (during vascular anastomosis surgery), the surgical support system 1a inputs surgical field images (high-resolution moving images) from the anastomosis surgery into the trained model in real time as explanatory variables. The trained model performs inference processing based on the input surgical field images. Hereinafter, the stage in which inference processing is performed will be referred to as the "inference stage." Inference processing includes, for example, object detection processing (segmentation processing) and object recognition (classification) processing on the surgical field images. The surgical support system 1a obtains one or more inference results (target variables) from the trained model.

[0020] The inference result is, for example, the coordinates of the stump of the blood vessel (target for anastomosis) in the surgical field image. Based on the coordinates of the stump, the surgical support system 1a superimposes an image representing the stump region onto the surgical field image in real time. The image representing the stump region may be, for example, a bounding box, or a "+" or "×" mark drawn with a line of a predetermined thickness.

[0021] When the surgical support system 1a detects a vascular stump in the surgical field image, it displays a surgical field image (support image) with an image representing the stump region superimposed on it on a predetermined display device in real time. By visually viewing the surgical field image with the image representing the stump region superimposed, the surgeon can easily grasp the stump region in the surgical field image. For example, in this way, the surgical support system 1a assists in vascular anastomosis surgery.

[0022] The surgical support system 1a comprises a learning device 2, an exoscopy 3a, an exoscopy display device 4, a surgical support device 5a, and a support display device 6. In the learning and evaluation stages of the first embodiment, the surgical support system 1a only needs to include the learning device 2. Also, in the reasoning stage of the first embodiment, the surgical support system 1a only needs to include the exoscopy 3a, the surgical support device 5a, the exoscopy display device 4, and the support display device 6.

[0023] The learning device 2 may be provided in the surgical support device 5a. The surgical support device 5a may be provided in the exoscopy device 3a. Furthermore, the exoscopy display device 4 and the support display device 6 may be separate or integrated. That is, the image displayed in the exoscopy display device 4 and the image displayed in the support display device 6 may be displayed using screens provided in each display device, or they may be displayed using a split screen (two screens) provided in the exoscopy display device 4 or the support display device 6.

[0024] The learning device 2 comprises a storage device 21, an operation unit 22, a display unit 23, a preprocessing unit 24, a learning unit 25, an evaluation unit 26, and a communication unit 27. The preprocessing unit 24 and the evaluation unit 26 may be provided in the learning unit 25.

[0025] In the learning device 2, some or all of the preprocessing unit 24, learning unit 25, and evaluation unit 26 are implemented as software by a processor such as a CPU (Central Processing Unit) executing a program stored in a storage device 21 having a non-volatile recording medium (non-temporary recording medium). The program may include a weight file. The weight file is used, for example, to determine the parameters (neuron weighting) of a learning model having a neural network. The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), CD-ROMs (Compact Disc Read Only Memory), hard disks built into computer systems, and non-temporary recording media such as solid-state drives. The communication unit 27 transmits the processing results from the learning device 2 to the surgical support device 5a. The communication unit 27 may also receive the program via a communication line. In the learning device 2, some or all of the preprocessing unit 24, learning unit 25, and evaluation unit 26 may be implemented using hardware including electronic circuits (or circuits) such as LSI (Large Scale Integrated Circuit), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field Programmable Gate Array).

[0026] The learning device 2 is an information processing device, such as a personal computer or a tablet terminal. The operation unit 22 is, for example, a keyboard and mouse. The operation unit 22 may also be a touch panel. If the operation unit 22 is a touch panel, it may be integrated with the display unit 23. The display unit 23 is, for example, a liquid crystal display or an organic electro-luminescence (OLED) display.

[0027] The exoscopy 3a comprises a communication unit 31, an output unit 32, an exoscopy image processing unit 33a, a storage device 34, an arm 35, and a camera 36. The camera 36 is, for example, a stereo camera.

[0028] In the exoscopy 3a, the exoscopy image processing unit 33a is implemented as software by a processor such as a CPU executing a program stored in a storage device 34 having a non-volatile recording medium (non-temporary recording medium). The communication unit 31 acquires support images generated by the surgical support device 5a. The communication unit 31 may also receive programs via a communication line. In the exoscopy 3a, the exoscopy image processing unit 33a may be implemented using hardware including electronic circuits such as LSI, ASIC, PLD, or FPGA.

[0029] The exoscopy display device 4 is, for example, a liquid crystal display or an organic EL display. A polarizing film may be attached to the screen of the exoscopy display device 4. The left eye image and the right eye image, which are stereo pair images generated by the camera 36, ​​may be displayed alternately on the screen of the exoscopy display device 4 at a predetermined frame rate. The surgical support device 5a includes a communication unit 51, a storage device 52, an operation unit 53, an inference unit 54, a support image processing unit 55a, and an output unit 56.

[0030] In the surgical support device 5a, some or all of the inference unit 54 and the support image processing unit 55a are implemented as software by a processor such as a CPU executing a program stored in a storage device 34 having a non-volatile recording medium (non-temporary recording medium). The communication unit 51 acquires a learned model in advance from the learning device 2. The communication unit 51 may also receive the program via a communication line. In the exoscopy device 3a, some or all of the inference unit 54 and the support image processing unit 55a may be implemented using hardware including electronic circuits such as LSI, ASIC, PLD, or FPGA. The operation unit 53 is, for example, a foot switch. The operation unit 53 may also be, for example, a light-shielding switch. The support display device 6 is, for example, a liquid crystal display or an organic EL display.

[0031] Next, the details of the learning phase before the anastomosis surgery will be described. The learning phase may include not only the phase in which the learning process is executed, but also a pre-processing phase for the learning process. The memory device 21 pre-stores multiple surgical field images in which blood vessels were imaged in past anastomosis surgeries as multiple training images. The memory device 21 may also pre-store multiple surgical field images in which blood vessels were imaged in past anastomosis surgeries as multiple verification images.

[0032] Figure 2 shows a first example of a surgical field image (training image) in the first embodiment. The surgical field image 101 shows, as an example, a blood vessel 201, a blood vessel 202, a clip 301, a clip 302, gauze 303, a stump 401 of the end of blood vessel 201, damage 402 to the intima of blood vessel 201, a thrombus 403 as an example of an embolic factor in the lumen of blood vessel 201, and rubbish 404 around blood vessel 201. The surgical field image 101 may also show calcium as an example of an embolic factor in the lumen of blood vessel 201.

[0033] Vessels 201 and 202 are to be anastomosed. A clip 301 is attached to vessel 201. A clip 302 is attached to vessel 202. The field of view of the surgical field image 101 is set so that vessels 201 and 202 are aligned horizontally. In addition, gauze 303 is placed behind vessels 201 and 202 so as to cover the background of vessels 201 and 202 in the surgical field image 101.

[0034] Figure 3 shows a second example of a surgical field image (training image) in the first embodiment. In the surgical field image 102, blood vessels 203 and 204, clips 301 and 302, gauze 303, the stump 405 of blood vessel 203, and forceps 304 as an example of a surgical instrument are captured. Thus, the surgical field image may capture not only the anastomosis target but also various surgical instruments.

[0035] Vessels 203 and 204 are to be anastomosed. A clip 301 is attached to vessel 203. A clip 302 is attached to vessel 204. The field of view of the surgical field image 102 is set so that vessels 203 and 204 are aligned horizontally. In addition, gauze 303 is placed behind vessels 203 and 204 so as to cover the background of vessels 203 and 204 in the surgical field image 102.

[0036] Figure 4 shows a third example of a surgical field image (training image) in the first embodiment. In the surgical field image 103, a blood vessel 205, a blood vessel 206, a clip 301, a clip 302, gauze 303, the stump 406 of the blood vessel 205, damage to the intima of the blood vessel 205 407, and forceps 304 are captured as an example. Thus, damage to the intima of a blood vessel may be captured in the surgical field image.

[0037] Vessels 205 and 206 are to be anastomosed. A clip 301 is attached to vessel 205. A clip 302 is attached to vessel 206. The field of view of the surgical field image 103 is set so that vessels 205 and 206 are aligned horizontally. In addition, gauze 303 is placed behind vessels 205 and 206 so as to cover the background of vessels 205 and 206 in the surgical field image 103.

[0038] Figure 5 shows a fourth example of a surgical field image (training image) in the first embodiment. In the surgical field image 104, blood vessels 207 and 208, clips 301 and 302, gauze 303, and the stump 408 of blood vessel 207 are captured as an example.

[0039] Vessels 207 and 208 are to be anastomosed. A clip 301 is attached to vessel 207. A clip 302 is attached to vessel 208. The field of view of the surgical field image 104 is set so that vessels 207 and 208 are aligned vertically. In addition, gauze 303 is placed behind vessels 207 and 208 so as to cover the background of vessels 207 and 208 in the surgical field image 104.

[0040] These surgical field images are just one example. In the field of view of multiple surgical field images, the arrangement of the anastomosis targets (blood vessels) is standardized to be aligned horizontally or vertically. As a result, the orientation of the anastomosis targets is consistent across all surgical field images, reducing the variability in the accuracy of object detection and recognition by learning models that use surgical field images as explanatory variables.

[0041] Furthermore, since a sheet such as gauze is placed in the background of the anastomosis target in the field of view of multiple surgical field images, the background of the anastomosis target remains constant across all surgical field images. This further reduces the variability in the accuracy of object detection and object recognition by the learning model that uses the surgical field image as an explanatory variable.

[0042] Figure 6 shows an example of a surgical field image (a training image and a teacher image including correct labels) in the first embodiment. The display unit 23 displays, for example, the surgical field image 101 illustrated in Figure 2. The operation unit 22 receives input operations from the user. The user's instructions are, for example, instructions regarding annotation operations on the surgical field image 101. That is, the operation unit 22 receives an operation to assign correct labels to the surgical field image 101.

[0043] The correct label is a correct label indicating points to be noted in vascular anastomosis surgery. The correct label may be, for example, a correct label indicating the coordinates of the vascular stump, a correct label indicating the damage information of the vascular intima, a correct label indicating the presence or absence information of embolic factors such as thrombus or lime in the vascular lumen, or a correct label indicating the presence or absence information of debris around the blood vessel. The damage information of the vascular intima may be, for example, the coordinates (position information) of the damage in the surgical field image or the information representing the level of the damage. The presence or absence information of embolic factors may be, for example, the coordinates (position information) of the embolic factors in the surgical field image. The presence or absence information of debris may be, for example, the coordinates (position information) of the debris in the surgical field image.

[0044] The preprocessing unit 24 associates the correct label with the surgical field image 101 displayed on the display unit 23 based on an instruction from the user who operated the operation unit 22. The display unit 23 displays the surgical field image 501 (= surgical field image 101 + correct label). As an example, the surgical field image 501 is the surgical field image 101 with the correct labels of the stump region image 601, the damage region image 602, the thrombus region image 603, and the debris region image 604 superimposed thereon.

[0045] The stump region image 601 is an image (bounding box) representing the region of the stump of the blood vessel 201. The damage region image 602 is an image (bounding box) representing the damaged region of the intima of the blood vessel 201. The thrombus region image 603 is an image (bounding box) representing the region of thrombus in the lumen of the blood vessel 201. The debris region image 604 is an image (bounding box) representing the region of debris around the blood vessel 201.

[0046] The learning unit 25 inputs the surgical field image 101, which is a component of the surgical field image 501 (= surgical field image 101 + correct label), into the learning model as a training image (explanatory variable). The learning unit 25 acquires, from the learning model, the coordinates of the stump of the blood vessel 201, the coordinates of the injury, the coordinates of the thrombus, and the coordinates of the debris in the surgical field image 101 as each inference result. The learning unit 25 generates a learned model (a learning model with improved accuracy) from the learning model based on the comparison result between the inference result and the correct label, which is a component of the surgical field image 501. For example, the learning unit 25 may generate a learned model from the learning model using the error backpropagation method for the comparison result (the difference between the inference result and the correct label).

[0047] FIG. 7 is a diagram showing an example of a surgical field image (teacher image including a verification image and a correct label) in the first embodiment. The display unit 23 displays, for example, the surgical field image 102 illustrated in FIG. 3. The operation unit 22 receives an input operation of an instruction by the user. The instruction by the user is, for example, an instruction regarding an annotation operation on the surgical field image 102. That is, the operation unit 22 receives an operation of assigning a correct label to the surgical field image 102.

[0048] The preprocessing unit 24 associates a correct label with the surgical field image 102 based on an instruction from the user who operated the operation unit 22. The display unit 23 displays the surgical field image 502 (= surgical field image 102 + correct label). As an example, the surgical field image 502 is the surgical field image 102 with the stump region image 605 (correct label) superimposed thereon. The stump region image 605 is an image representing the region of the stump 405 of the blood vessel 203. A character string image "Stump" representing the stump may be superimposed on the surgical field image 502 as the correct label of the object name (class name) based on the result of object detection.

[0049] The learning unit 25 inputs the surgical field image 102, which is a component of the surgical field image 502 (= surgical field image 102 + correct label), as a validation image (explanatory variable) to the trained model. The learning unit 25 obtains the coordinates of the stump of the blood vessel 201, the coordinates of the injury, the coordinates of the thrombus, and the coordinates of debris from the trained model as inference results in the surgical field image 101. The display unit 23 may display each of the obtained inference results. The operation unit 22 accepts input operations from the user. User instructions are, for example, instructions to change hyperparameters. Based on the user's instructions after confirming these validation results, the learning unit 25 adjusts the hyperparameters of the machine learning during the learning phase.

[0050] Next, we will describe the details of the evaluation stage before the anastomosis surgery. The memory device 21 may pre-store multiple surgical field images as multiple evaluation images (test images).

[0051] Figure 8 shows an example of a surgical field image (a training image including an evaluation image and correct labels) in the first embodiment. The display unit 23 displays, for example, the surgical field image 103 illustrated in Figure 4. The operation unit 22 receives input operations from the user. The user's instructions are, for example, instructions regarding annotation operations on the surgical field image 103. That is, the operation unit 22 receives an operation to assign correct labels to the surgical field image 103.

[0052] The preprocessing unit 24 associates correct labels with the surgical field image 103 based on instructions from the user. The display unit 23 displays the surgical field image 503 (= surgical field image 103 + correct labels). The surgical field image 503 is, as an example, the surgical field image 103 with the correct labels of the stump region image 606 and the injury region image 607 superimposed. The stump region image 606 is an image representing the stump region of the blood vessel 205. The injury region image 607 is an image representing the injury region of the intima of the blood vessel 205. Based on the results of object detection, the surgical field image 503 may also have a string image "Stump" representing the stump and a string image "Damage" representing the injury superimposed as correct labels for the object names.

[0053] The evaluation unit 26 inputs the surgical field image 103 (= surgical field image 103 + ground truth label), which is a component of the surgical field image 503, into the trained model as an evaluation image (explanatory variable). The learning unit 25 obtains the coordinates of the stump end of the blood vessel 205 and the coordinates of the injury in the surgical field image 103 from the trained model as inference results. The learning unit 25 evaluates the versatility of the trained model based on the comparison result between the inference result and the ground truth label, which is a component of the surgical field image 503. Based on the evaluation result, the evaluation unit 26 updates the weight file representing the parameters of the trained model. This further improves the accuracy of the trained model generated by the learning unit 25. The communication unit 27 transmits the trained model to the communication unit 51.

[0054] Before the inference phase begins, the communication unit 51 obtains the trained model from the communication unit 27 in advance. The communication unit 51 also records the trained model generated by the learning device 2 in the storage device 52 in advance. The storage device 52 stores the trained model generated by the learning device 2.

[0055] Next, we will explain the details of the reasoning stage during anastomosis surgery. Returning to Figure 1, we will continue the explanation of the configuration example of the surgical support system 1a. The communication unit 31 transmits the surgical field image (high-resolution moving image) generated by the camera 36 to the communication unit 51 in real time. The surgical field image captures the surgical field in patient 7, including the target of anastomosis (blood vessel).

[0056] When a surgical field image, which has undergone predetermined image processing (for example, image processing for three-dimensional display) by the exoscopy image processing unit 33a, and a support image generated by the support image processing unit 55a are displayed using the split screen (two screens) of the exoscopy display device 4, the communication unit 51 transmits the support image generated by the support image processing unit 55a to the communication unit 31. The communication unit 31 outputs the support image transmitted from the communication unit 51 to the output unit 32.

[0057] The output unit 32 outputs the surgical field image, which has undergone predetermined image processing by the exoscopy image processing unit 33a, to the exoscopy display device 4. If the output unit 32 obtains a support image generated by the support image processing unit 55a from the communication unit 31, the output unit 32 may, for example, display the surgical field image in three dimensions on the first split screen provided on the exoscopy display device 4, and display the support image in two or three dimensions on the second split screen provided on the exoscopy display device 4.

[0058] The exoscopy image processing unit 33a performs predetermined image processing on the surgical field image generated by the camera 36. The exoscopy image processing unit 33a outputs the surgical field image, on which the predetermined image processing has been performed, to the output unit 32. The storage device 34 stores in advance the program to be executed by the exoscopy image processing unit 33a. The storage device 34 may also store the surgical field image generated by the camera 36.

[0059] Arm 35 is a member that supports the camera 36. Here, for example, the position and imaging direction of the camera 36 can be adjusted by changing the angle of arm 35 by the surgeon or surgical assistant. The camera 36 images the surgical field, including the anastomosis target (blood vessel) in the patient 7, from a predetermined imaging direction and at a predetermined magnification. The camera 36 (stereo camera) generates stereo pair images (left eye image and right eye image) that constitute the surgical field image at a predetermined frame rate.

[0060] The exoscopy display device 4 alternately switches and displays the left-eye image and the right-eye image, which constitute the surgical field image generated by the camera 36, ​​at a predetermined frame rate. As a result, the surgical field image is displayed on the screen of the exoscopy display device 4 as a three-dimensional image for the surgeon who views the screen of the exoscopy display device 4 through polarized glasses.

[0061] The communication unit 51 acquires the surgical field image generated by the camera 36 from the communication unit 31 in real time. The communication unit 51 outputs the surgical field image at a predetermined frame rate to the inference unit 54 in real time.

[0062] The control unit 53 accepts input operations for instructions from the surgeon or surgical assistant. Instructions from the surgeon or surgical assistant include, for example, whether or not to superimpose an image based on the inference result onto the surgical field image on the support display device 6 (whether or not to turn on the display of bounding boxes, etc.). If the control unit 53 is, for example, a foot switch or a light-shielding switch, the surgeon can instruct the surgical support device 5a to turn the display of bounding boxes on or off on the support display device 6 without letting go of surgical instruments such as forceps 304.

[0063] The inference unit 54 acquires the trained model generated by the learning device 2 from the storage device 52. The inference unit 54 inputs the surgical field image, including the image of the blood vessel to be anastomosed, into the trained model as the image to be inferred (explanatory variable). The inference unit 54 acquires the inference result (dependent variable) from the trained model. The inference result is, for example, an inference result regarding points to be aware of in vascular anastomosis surgery. The inference result regarding points to be aware of in anastomosis surgery may be, for example, an inference result showing the coordinates of the vascular stump, an inference result showing information on damage to the vascular intima, an inference result showing information on the presence or absence of embolic factors such as thrombi or calcium in the lumen of the blood vessel, or an inference result showing information on the presence or absence of debris around the blood vessel.

[0064] The support image processing unit 55a generates a support image for anastomosis surgery by superimposing an image (e.g., bounding box) based on the acquired inference results onto the surgical field image. The output unit 56 outputs the support image, which is the surgical field image with the image based on the inference results superimposed, to the support display device 6. The support display device 6 displays the support image.

[0065] Figure 9 shows a first example of a surgical field image (image to be inferred) in the first embodiment. The surgical field image 105 illustrated in Figure 9 is displayed in real time on the screen of the exoscopy display device 4 via the output unit 32. The surgical field image 105 is also transmitted to the inference unit 54 via the communication unit 31 and the communication unit 51.

[0066] In the surgical field image 105, a blood vessel 209, a blood vessel 210, a clip 301, a clip 302, gauze 303, the stump 409 of blood vessel 209, and damage to the intima of blood vessel 209 410 are captured as an example. Blood vessels 209 and 210 are targets for anastomosis. Clip 301 is attached to blood vessel 209. Clip 302 is attached to blood vessel 210. The field of view of the surgical field image 105 is set so that blood vessels 209 and 210 are aligned horizontally. In addition, gauze 303 is laid behind blood vessels 209 and 210 so as to cover the background of blood vessels 209 and 210 in the surgical field image 105.

[0067] Figure 10 shows a first example of a support image, which is a surgical field image superimposed with an image based on the inference result, in the first embodiment. The support image 106 illustrated in Figure 10 is, as an example, a surgical field image 105 in which a stump region image 701 and a damaged region image 702 are superimposed. The stump region image 701 is an image representing the stump region of the blood vessel 209. The damaged region image 702 is an image representing the damaged region of the intima of the blood vessel 209. The support image 106 may also have a string image "Stump" representing the stump and a string image "Damage" representing the damage superimposed on it, based on the object detection result. The support image 106 may also have a confidence score for the object (class) detection result superimposed on it. The support image 106 is displayed on the screen of the support display device 6 via the output unit 56.

[0068] Figure 11 shows a second example of the surgical field image (image to be inferred) in the first embodiment. The surgical field image 107 illustrated in Figure 11 is displayed on the screen of the exoscopy display device 4 via the output unit 32. The surgical field image 107 is also transmitted to the inference unit 54 via the communication unit 31 and the communication unit 51.

[0069] The surgical field image 107 shows, as an example, blood vessels 211 and 212, clips 301 and 302, gauze 303, the stump 411 of blood vessel 211, a thrombus 412 in the lumen of blood vessel 211, and debris 413 around blood vessel 211. Blood vessels 211 and 212 are to be anastomosed. Clip 301 is attached to blood vessel 211. Clip 302 is attached to blood vessel 212. The field of view of the surgical field image 107 is set so that blood vessels 211 and 212 are aligned horizontally. In addition, gauze 303 is laid behind blood vessels 211 and 212 so as to cover the background of blood vessels 211 and 212 in the surgical field image 107.

[0070] Figure 12 shows a second example of a support image, which is a surgical field image superimposed with an image based on the inference results, in the first embodiment. The support image 108 illustrated in Figure 12 is, as an example, a surgical field image 107 in which a stump region image 701, a thrombus region image 703, and a rubbish region image 704 are superimposed. The stump region image 701 is an image representing the stump region of the blood vessel 211. The thrombus region image 703 is an image representing the thrombus region in the lumen of the blood vessel 211. The rubbish region image 704 is an image representing the rubbish region around the blood vessel 211. The support image 108 may also have a string image "Stump" representing the stump, a string image "Clot" representing the thrombus, and a string image "Rubbish" representing rubbish superimposed on it, based on the object detection results. The support image 108 may also have a confidence score for the object detection results superimposed on it. The support image 108 is displayed on the screen of the support display device 6 via the output unit 56.

[0071] Next, an example of the operation of the surgical support system 1a will be described. Figure 13 is a flowchart showing an example of the operation of the learning device 2 in the first embodiment. In the preprocessing stage, the preprocessing unit 24 acquires the surgical field image from the storage device 21 (step S101). The preprocessing unit 24 acquires the correct label from the operation unit 22 (step S102). The preprocessing unit 24 associates the surgical field image with the correct label (step S103).

[0072] During the learning phase, the learning unit 25 inputs the surgical field image into the learning model (step S104). The learning unit 25 obtains the inference result from the learning model (step S105). The learning unit 25 compares the inference result with the correct label (step S106). Based on the comparison result, the learning unit 25 generates a trained model from the learning model (step S107). The learning unit 25 determines whether the number of learning iterations is above a threshold (step S108). If it is determined that the number of learning iterations is below the threshold (step S108: NO), the learning unit 25 returns to step S104. If it is determined that the number of learning iterations is above the threshold (step S108: YES), the learning unit 25 terminates machine learning.

[0073] Figure 14 is a flowchart illustrating an example of the operation of the surgical support device 5a in the first embodiment. The inference unit 54 acquires a pre-trained model from the storage device 52 (step S201). During the inference stage, the inference unit 54 acquires a surgical field image from the exoscopy 3a in real time via the communication unit 51 (step S202). The inference unit 54 inputs the surgical field image into the pre-trained model (step S203). The inference unit 54 acquires the inference result from the pre-trained model (step S204). Based on the inference result, the inference unit 54 generates an image to support the anastomosis surgery (step S205).

[0074] The inference unit 54 determines whether or not to terminate the inference process. For example, if the operation unit 53 receives a termination instruction, the inference unit 54 determines to terminate the inference process (step S206). If it determines to continue the inference process, the inference unit 54 returns to step S202. If it determines to terminate the inference process, the inference unit 54 terminates the inference process.

[0075] As described above, in the preprocessing stage of the learning phase, the preprocessing unit 24 associates the surgical field image, which includes an image of the vascular tissue to be anastomosed, with the correct label indicating the coordinates of the vascular tissue's stump. In the learning phase, the learning unit 25 inputs the surgical field image, which is associated with the correct label indicating the stump coordinate, into the learning model as a training image. The learning unit 25 obtains the coordinates of the vascular tissue's stump in the surgical field image from the learning model as one of the inference results. Based on the comparison result between the stump coordinate obtained from the learning model and the correct label indicating the stump coordinate, the learning unit 25 generates a trained model from the learning model. This enables the learning device 2 to support (assist) vascular tissue anastomosis surgery.

[0076] In the preprocessing stage of the learning phase, the preprocessing unit 24 may associate the surgical field image with a correct label indicating intimal damage information of vascular tissue. In the learning phase, the learning unit 25 may input the surgical field image associated with the correct label indicating intimal damage information as a training image into the learning model. The learning unit 25 may acquire intimal damage information from the learning model as one of the inference results. The learning unit 25 may generate a trained model from the learning model based on the comparison result between the intimal damage information acquired from the learning model and the correct label indicating intimal damage information.

[0077] In the preprocessing stage of the learning phase, the preprocessing unit 24 may associate surgical field images with ground truth labels indicating the presence or absence of embolic factors such as thrombi or calcium in the lumen of vascular tissue. In the learning phase, the learning unit 25 may input surgical field images associated with ground truth labels indicating the presence or absence of thrombi as training images into the learning model. The learning unit 25 may obtain information on the presence or absence of embolic factors from the learning model as one of the inference results. The learning unit 25 may generate a trained model from the learning model based on the comparison result between the information on the presence or absence of embolic factors obtained from the learning model and the ground truth labels indicating the presence or absence of embolic factors.

[0078] In the preprocessing stage of the learning phase, the preprocessing unit 24 may associate the surgical field image with a ground truth label indicating the presence or absence of debris around the vascular tissue. In the learning phase, the learning unit 25 may input the surgical field image associated with the ground truth label indicating the presence or absence of debris as a training image into the learning model. The learning unit 25 may obtain the debris presence or absence information from the learning model as one of the inference results. The learning unit 25 may generate a trained model from the learning model based on the comparison result between the debris presence or absence information obtained from the learning model and the ground truth label indicating the presence or absence of debris.

[0079] In the inference stage, the inference unit 54 inputs a surgical field image, including an image of the vascular tissue to be anastomosed, as the image to be inferred into the trained model. The inference unit 54 obtains the coordinates of the vascular tissue margins in the surgical field image from the trained model as one of the inference results. The support image processing unit 55a generates a support image for the anastomosis surgery by superimposing an image indicating the region of the margins in the surgical field image onto the surgical field image based on the obtained margin coordinates. The output unit 56 displays the support image on the support display device 6.

[0080] This allows the surgical support device 5a to assist (support) vascular tissue anastomosis surgery. It can shorten surgical time. It can reduce the burden on the patient. It can improve the surgeon's technique. It can be used in real time as a cautionary aid tool based on the clinical surgeon's experience. Even an inexperienced surgical assistant can accurately perform tasks such as cleaning the vascular tissue stumps. Even in cases where a skilled surgeon might hesitate in judgment, it can dramatically improve the safety of the anastomosis surgery. It can dramatically improve the safety of anastomosis surgery performed by a surgeon who has not yet gained experience.

[0081] During the inference stage, the inference unit 54 may further acquire information on intimal damage to vascular tissue from the trained model as one of the inference results. The support image processing unit 55a may further superimpose an image representing the acquired intimal damage information onto the support image. The output unit 56 may display the support image with the superimposed image representing intimal damage information on the support display device 6.

[0082] During the inference stage, the inference unit 54 may further acquire information on the presence or absence of embolic factors in the lumen of vascular tissue from the trained model as one of the inference results. The support image processing unit 55a may further superimpose an image representing the acquired information on the presence or absence of embolic factors onto the support image. The output unit 56 may display the support image with the superimposed image representing the information on the presence or absence of embolic factors on the support display device 6.

[0083] During the inference stage, the inference unit 54 may further acquire information on the presence or absence of debris around the vascular tissue from the trained model as one of the inference results. The support image processing unit 55a may further superimpose an image representing the acquired information on the presence or absence of debris onto the support image. The output unit 56 may display the support image with the superimposed image representing the information on the presence or absence of debris on the support display device 6.

[0084] Next, various modifications of the first embodiment will be described. To avoid interfering with the surgeon's surgical operations, it is desirable that the image of the tissue to be anastomosed is not obscured by other images in the support image. Therefore, the form of the image based on the inference result may be the form shown in the following first to fifth modifications.

[0085] (First Modification of the First Embodiment) Figure 15 shows an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the first modification of the first embodiment. The support image 109 illustrated in Figure 15 is, as an example, a surgical field image with a stump region image 705 (bounding box) superimposed. The stump region image 705 is an image representing the stump region of the blood vessel 213. The support image 109 may also have a string image "Stump" representing the stump superimposed based on the object detection result. The support image 109 may also have a confidence score for the object detection result superimposed. The support image 109 is displayed on the screen of the support display device 6 via the output unit 56. The thickness of the frame of the stump region image 705 is set to be thinner than a predetermined thickness. This makes it possible to prevent the blood vessels 213 and 214 from being obscured as much as possible by the stump region image 705.

[0086] (Second Modification of the First Embodiment) Figure 16 shows an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in a second modification of the first embodiment. The support image 110 illustrated in Figure 16 is, as an example, a surgical field image with a stump region image 706 (bounding box) superimposed. The stump region image 706 is an image representing the stump region of the blood vessel 213. The support image 110 may also have a string image "Stump" representing the stump superimposed based on the object detection result. The support image 110 may also have a confidence score for the object detection result superimposed. The support image 110 is displayed on the screen of the support display device 6 via the output unit 56.

[0087] The transparency (transmittance) of the frame of the stump region image 706 is set to be higher than a predetermined transparency. This makes it possible to prevent the anastomosis target, such as the blood vessel 213, from being obscured as much as possible by the stump region image 706. In addition, the transparency of the text string image "Stump" representing the stump may also be set to be higher than a predetermined transparency.

[0088] (Third Modification of the First Embodiment) Figure 17 shows an example of a support image, which is a surgical field image superimposed with an image based on the inference result, in the third modification of the first embodiment. The support image 111 illustrated in Figure 17 is, as an example, a surgical field image superimposed with an advice image 801, which is a string image representing advice for anastomosis surgery. In this surgical field image, as an example, blood vessels 215 and 216, clips 301 and 302, gauze 303, forceps 304, the stump 415 of blood vessel 215, and damage 416 to the intima of blood vessel 215 are captured. The support image 111 is displayed on the screen of the support display device 6 via the output unit 56.

[0089] The support image processing unit 55a superimposes the advice image 801 onto an area away from the center of the support image 111. In Figure 17, as an example, the support image processing unit 55a superimposes the advice image 801 "Damage" onto the lower right corner of the support image 111. This makes it possible to prevent the blood vessels 215 and 216 displayed in areas other than the lower right corner of the support image 111 from being obscured as much as possible by the stump region image 706. Furthermore, it makes it easy for the surgeon to identify if there is damage somewhere in the blood vessel 215.

[0090] (Fourth Modification of the First Embodiment) Figure 18 shows an example of a support image, which is a surgical field image with an image based on the inference result superimposed, in the fourth modification of the first embodiment. The support image 112 illustrated in Figure 18 is, as an example, a surgical field image with an advice image 801 and a thin arrow 802 superimposed. In this surgical field image, a blood vessel 215, a blood vessel 216, a clip 301, a clip 302, gauze 303, forceps 304, the stump 415 of the blood vessel 215, and damage 416 to the intima of the blood vessel 215 are captured as examples. The support image 112 is displayed on the screen of the support display device 6 via the output unit 56.

[0091] The thin arrow 802 is an arrow that indicates the location of the injury 416 by drawing a straight line connecting the location of the injury 416 to the location of the advice image 801. The thickness of the thin arrow 802 is determined to be thinner than a predetermined thickness. This makes it possible to prevent the blood vessels 215 and 216 from being obscured by the thin arrow 802 as much as possible in the support image 112.

[0092] (Fifth Modification of the First Embodiment) Figure 19 shows an example of a support image in the fifth modification of the first embodiment, which is a surgical field image overlaid with an image based on the inference result (before deformation). The support image 113 illustrated in Figure 19 is, as an example, a surgical field image overlaid with an advice image 801 and a variable arrow 803. In this surgical field image, as an example, blood vessels 215 and 216, clips 301 and 302, gauze 303, forceps 304, the stump 415 of blood vessel 215, and damage 416 to the intima of blood vessel 215 are captured.

[0093] The inference unit 54 inputs this surgical field image as an explanatory variable into the trained model. The inference unit 54 obtains the object detection result from the trained model as one of the inference results. One of the inference results is, for example, the coordinates representing the image area of ​​the tweezers 304. The support image processing unit 55a draws a variable arrow 803 indicating the location of the injury 416 by connecting the location of the injury 416 with the location of the advice image 801 in this surgical field image. In Figure 19, the variable arrow 803 does not overlap with the image area of ​​the tweezers 304, so the shape of the variable arrow 803 may be a straight line. The support image 113 is displayed on the screen of the support display device 6 via the output unit 56.

[0094] Figure 20 shows an example of a support image, which is a surgical field image superimposed with an image (after deformation) based on the inference result, in a fifth modification of the first embodiment. The imaging time of the surgical field image of the support image 113 exemplified in Figure 20 is, for example, 1 second after the imaging time of the surgical field image of the support image 113 exemplified in Figure 19. In this way, the position of the forceps 304 in the surgical field image may change in a time-series surgical field image.

[0095] The inference unit 54 inputs the surgical field image as an explanatory variable into the trained model. The inference unit 54 obtains the object detection result from the trained model as one of the inference results. One of the inference results is, for example, the coordinates representing the image area of ​​the tweezers 304. The support image processing unit 55a deforms the variable arrow 803 based on the coordinates representing the image area of ​​the tweezers 304 so that the variable arrow 803 does not overlap the image area of ​​the tweezers 304. Since the variable arrow 803 is deformed so that it does not overlap the image area of ​​the tweezers 304, it is possible to ensure that the tweezers 304 are not hidden by the variable arrow 803 as much as possible in the support image 113.

[0096] One of the inference results may be, for example, the coordinates representing the image region of the blood vessel 215. The support image processing unit 55a may deform the variable arrow 803 based on the coordinates representing the image region of the blood vessel 215 so that the variable arrow 803 does not overlap the image region of the blood vessel 215 as much as possible. Because the variable arrow 803 is deformed, it is possible to ensure that the blood vessel 215 is not hidden by the variable arrow 803 as much as possible in the support image 113.

[0097] A text-based image (advice image) containing specific advice regarding anastomosis surgery may be superimposed on the support image. For example, the form of the image based on the inference result may be the form shown in the sixth modified example below.

[0098] (Sixth Modification of the First Embodiment) Figure 21 shows an example of a data table representing the correspondence between injury levels and advice in the sixth modification of the first embodiment. The storage device 52 may store the data table and training images. Each of the n (n is an integer of 2 or more) surgical field images taken in past anastomosis surgeries is assigned an identification number from "1" to "n" without duplication.

[0099] During the learning phase, these n surgical field images are treated as n training images (explanatory variables). In the data table illustrated in Figure 21, the identification number of the surgical field image (training image) is associated with the correct label of the injury information and the advice. For example, at a predetermined coordinate in the training image with identification number "2", an injury of "Level 3" is captured as an example. For example, at a predetermined coordinate in the training image with identification number "3", an injury of "Level 3" is captured as an example. For example, at a predetermined coordinate in the training image with identification number "n", an injury of "Level 2" is captured as an example.

[0100] During the learning phase, the learning unit 25 inputs these n surgical field images into the learning model. For each training image input to the learning model, the learning unit 25 obtains the coordinates of the vascular stump, the coordinates of the injury, the injury level, the coordinates of embolic factors such as thrombi, and the coordinates of debris from the learning model as inference results. Based on the comparison result between the inference result and the correct label, the learning unit 25 generates a trained model from the learning model.

[0101] During the inference stage, the inference unit 54 inputs a surgical field image, including an image of the blood vessel to be anastomosed, as the image to be inferred into the trained model. The inference unit 54 obtains the inference results from the trained model. The inference results include, for example, the coordinates of the vascular stump, the coordinates of the damage to the vascular intima, and the level of damage to the vascular intima in the surgical field image.

[0102] The support image processing unit 55a generates support images for anastomosis surgery by superimposing images based on the inference results onto the surgical field image, based on the inference results and the data table. For example, if the trained model estimates that the intimal damage to the blood vessel is "level 2", the support image processing unit 55a generates a support image that includes an advice string image corresponding to the correct label "damage level 2" in the data table. For example, if the trained model estimates that the intimal damage to the blood vessel is "level 3", the support image processing unit 55a generates a support image that includes an advice string image corresponding to the correct label "damage level 3" in the data table. The output unit 56 outputs the generated support image to the support display device 6. The support display device 6 displays the support image.

[0103] In addition, in the data table illustrated in Figure 21, the advice may differ for each surgical field image. During the inference stage, the inference unit 54 may obtain an index value (e.g., confidence level) from the trained model that indicates which of the training images from identification number "1" to "n" the input surgical field image (inference target image) is similar to. The support image processing unit 55a may generate a support image, including an image of advice associated with a training image similar to the input surgical field image (inference target image), based on the index value. This makes it possible for the support image processing unit 55a to change the advice depending on which training image the input surgical field image (inference target image) is similar to.

[0104] Figure 22 shows an example of a support image, which is a surgical field image with an advice image based on the inference result superimposed, in a sixth modification of the first embodiment. The support image 113 illustrated in Figure 22 is, as an example, a surgical field image with an advice image 801 superimposed. In this surgical field image, a blood vessel 215, a blood vessel 216, a clip 301, a clip 302, gauze 303, forceps 304, the stump 415 of the blood vessel 215, and damage 416 to the intima of the blood vessel 215 are captured as examples. The support image 111 is displayed on the screen of the support display device 6 via the output unit 56.

[0105] In Figure 22, the support image processing unit 55a superimposes an advice image 801 on the lower right corner of the support image 113, as an example. The injury level of injury 416 is, for example, "2". In this case, the advice image 801 displays a string image representing specific advice associated with injury level "2", based on the data table illustrated in Figure 21. The advice image includes, for example, a string image indicating that a new stump should be exposed in vascular tissue. Such specific advice makes it easy for the surgeon to understand what to do. If the injury level of injury 416 is, for example, "3", the advice image 801 displays a string image representing specific advice associated with injury level "3", based on the data table illustrated in Figure 21. The advice image includes, for example, a string image indicating that anastomosis should be performed in another vascular tissue. Such specific advice makes it easy for the surgeon to understand what to do.

[0106] (Second Embodiment) In the second embodiment, the main difference from the first embodiment is that an image based on the inference result (e.g., a bounding box) is superimposed on one of the stereo pair images (left eye image and right eye image) that constitute the support image. The second embodiment will be explained focusing on the differences from the first embodiment.

[0107] Figure 23 shows an example of the configuration of the surgical support system 1b in the second embodiment. The surgical support system 1b is a system that supports tubular tissue anastomosis surgery. The surgical support system 1b comprises a learning device 2, an exoscopy 3b, an exoscopy display device 4, and a surgical support device 5b. In the learning and evaluation stages of the second embodiment, the surgical support system 1b only needs to be equipped with the learning device 2. Also, in the reasoning stage of the second embodiment, the surgical support system 1b only needs to be equipped with the exoscopy 3b, the surgical support device 5b, and the exoscopy display device 4.

[0108] The communication unit 51 acquires the surgical field image generated by the camera 36 from the communication unit 31 in real time. The communication unit 51 outputs the surgical field image at a predetermined frame rate to the inference unit 54 in real time. The inference unit 54 inputs the surgical field image, including the image of the blood vessel to be anastomosed, as the image to be inferred (explanatory variable) into the trained model. The inference unit 54 acquires the inference result (dependent variable) from the trained model.

[0109] The support image processing unit 55b generates stereo pair images (left eye image and right eye image) that constitute the support image. The support image processing unit 55b superimposes an image based on the inference result onto one of the support images, either the left eye image or the right eye image. The support image processing unit 55b outputs the support image to the communication unit 51. The communication unit 51 transmits the support image to the output unit 32.

[0110] The output unit 32 acquires the support image generated by the support image processing unit 55b from the communication unit 31. The output unit 32 displays the support image in three dimensions on a single screen of the exoscopy display device 4. That is, the exoscopy display device 4 alternately switches and displays the support image for the left eye (surgical field image without superimposed bounding boxes, etc.) and the support image for the right eye (surgical field image with superimposed bounding boxes, etc.) on a single screen at a predetermined frame rate. As a result, the support image is displayed on the screen of the exoscopy display device 4 as a three-dimensional image for the surgeon who views the screen of the exoscopy display device 4 through polarized glasses.

[0111] Figure 24 shows an example of a support image in the second embodiment, which is a surgical field image (for the surgeon's left eye) without an image based on the inference result superimposed, and a support image in which a surgical field image (for the surgeon's right eye) has an image based on the inference result superimposed. The exoscopy display device 4 displays support image 108-1 and support image 108-2 alternately at a predetermined frame rate.

[0112] Support image 108-1 (left eye image) does not have the stump region image 701, the thrombus region image 703, and the debris region image 704 superimposed. In contrast, support image 108-2 (right eye image) has the stump region image 701, the thrombus region image 703, and the debris region image 704 superimposed. Therefore, to a surgeon viewing support images 108-1 and 108-2 with both eyes, the stump region image 701, the thrombus region image 703, and the debris region image 704 appear semi-transparent.

[0113] Furthermore, even if support image 108-2 is displayed, if the surgeon closes their right eye, the surgeon will only see support image 108-1. Since support image 108-1 does not have the stump region image 701, the thrombus region image 703, and the debris region image 704 superimposed on it, the surgeon can easily see a surgical field image that does not interfere with surgical procedures.

[0114] Figure 25 shows an example of a support image in the second embodiment, which is a surgical field image (image for the surgeon's left eye) without an image based on the inference result superimposed, and a support image in which the image based on the inference result has been erased (image for the surgeon's right eye). The operation unit 53 may receive an instruction to erase the image based on the inference result (an instruction to turn off the display of the bounding box). If the operation unit 53 is, for example, a foot switch or a light-shielding switch, the surgeon can instruct the surgical support device 5b to turn on or off the display of the bounding box on the support display device 6 without letting go of surgical instruments such as forceps 304. When the operation unit 53 receives an instruction to erase the image based on the inference result, the support image processing unit 55b erases the image based on the inference result from the support image 108-2.

[0115] As described above, the support image processing unit 55b superimposes an image based on the inference result (e.g., a bounding box) onto either the support image for the left eye or the support image for the right eye. The exoscopy display device 4 alternately displays the support image for the left eye (surgical field image without the bounding box superimposed) and the support image for the right eye (surgical field image with the bounding box superimposed) on a single screen at a predetermined frame rate. This makes it possible to support anastomosis surgery.

[0116] (Evaluation of the use of the vascular anastomosis support program) An evaluation of the use of the vascular anastomosis support program was conducted after obtaining approval from the ethics committee.

[0117] In this evaluation, a support display device 6 was installed as a backup display device next to the display device that showed the surgical field image of the actual anastomosis surgery. During the anastomosis surgery, the support display device 6 displayed support images such as the support image 108-2 exemplified in Figure 24, according to the vascular anastomosis support program executed by the surgical support device 5a.

[0118] These evaluation results are based on feedback from four physicians who each used the vascular anastomosis support program implemented by the surgical support system in actual anastomosis surgeries.

[0119] Figure 26 shows the profiles of each physician who used the vascular anastomosis support program. Each physician is a board-certified plastic surgeon and a certified instructor in reconstructive microsurgery. In addition, each physician has experience performing a certain number of vascular anastomosis surgeries.

[0120] Figure 27 shows an example of the evaluation results regarding the operability of the vascular anastomosis support program. The visibility of movement was evaluated on a three-point scale for the display of surgical field images (support images) superimposed with text-based images such as "Stump" as exemplified in Figures 7 and 8, and bounding boxes as exemplified in Figures 6 through 8. The visibility of movement of the entire surgical field image (support image) was also evaluated on a three-point scale. These three levels range from "1" (difficult to see) to "3" (easy to see). As shown in Figure 27, the performance (operability) of the blood vessel recognition action is generally at a practical level. - Visibility of bounding box movement: Average "2" (neither good nor bad.) - Visibility of movement of the entire surgical field image: Average "1.25" (somewhat awkward and difficult to see.)

[0121] Furthermore, regardless of the doctor's experience or the number of surgeries performed, some users provided feedback such as "smaller display of the points of interest is preferable" and "the bounding box obstructs the view." Since there was also feedback indicating a desire for smoother image movement, improvements to the functionality would be beneficial.

[0122] Figure 28 shows an example of the evaluation results regarding the accuracy of the vascular anastomosis support program. The accuracy evaluation was conducted on a three-level scale, from a rating of "1" indicating that the recognition and identification by the surgical support system 1a were inappropriate, to a rating of "3" indicating that the recognition and identification were appropriate. Here, recognition and identification means, for example, as illustrated in Figure 8, that a bounding box representing the area of ​​the vascular stump is displayed as the recognition result, and the text image "Stump" representing the stump is displayed as the identification. Alternatively, recognition and identification may also mean, for example, as illustrated in Figure 12, that a bounding box representing the area of ​​a thrombus is displayed as the recognition result, and the text image "Clot" representing the thrombus is displayed as the identification. As shown in Figure 28, the recognition of blood vessels and the identification of thrombi are generally good. Further improvement may be made regarding more detailed identification (identification of vascular intimal damage and identification of debris around blood vessels). - Appropriateness of blood vessel recognition: Average "2.5" (relatively good) - Appropriateness of thrombus detection: Average "2.25" - Appropriateness of intimal damage detection: Average "1.5" - Appropriateness of rubbish detection around blood vessels: Average "1.75"

[0123] Figure 29 shows an example of the evaluation results regarding the usefulness of the vascular anastomosis support program. Because the surgical support system 1a has not yet been approved as a medical device, and from the perspective of ensuring the safety of anastomosis surgery, this evaluation had to be conducted so that the surgical field image of the actual anastomosis surgery was displayed on the display device, and then the support image was displayed on the screen of the support display device 6 installed next to the display device (hereinafter referred to as the "separate screen"). For this reason, the usefulness of the vascular anastomosis support program was evaluated in three stages. Here, the three stages range from an evaluation of "1" indicating that there was no time to check the support image displayed on the separate screen, to an evaluation of "3" indicating that there was time to check it. For less experienced physicians, there may not even be time to check the separate screen displaying the support image. Nevertheless, the usefulness of the vascular anastomosis support program (visibility of the screen) has generally reached a practical level. - Degree of margin for checking (visualizing) the screen: Average "2" (There is variation in the degree of margin.) - Frequency of being able to check (visualize) the screen: Average "1.75" (The frequency is not sufficient.) Note that screen visibility may be improved by improving the screen's layout and shape.

[0124] While embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention.

[0125] (Note) <Note 1> A surgical support device comprising: an inference unit that inputs a surgical field image including an image of the luminal tissue to be anastomosed as an inference target image to a trained model, and acquires the coordinates of the stump of the luminal tissue in the surgical field image from the trained model as one of the inference results; a support image processing unit that generates an image to support the anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image onto the surgical field image based on the acquired coordinates of the stump; and an output unit that displays the support image on a display device. <Note 2> The surgical support device according to Note 1, wherein the inference unit further acquires information on the intima of the luminal tissue as one of the inference results from the trained model; the support image processing unit further superimposes an image representing the acquired intima damage information onto the support image; and the output unit displays the support image with the image representing the intima damage information superimposed on it on the display device. <Note 3> The inference unit further acquires information on the presence or absence of embolic factors in the lumen of the tubular tissue as one of the inference results from the trained model; the support image processing unit further superimposes an image representing the acquired information on the presence or absence of embolic factors onto the support image; and the output unit causes the support image with the superimposed image representing the information on the presence or absence of embolic factors to be displayed on the display device, as described in Note 1 or Note 2. <Note 4> The inference unit further acquires information on the presence or absence of debris around the tubular tissue as one of the inference results from the trained model; the support image processing unit further superimposes an image representing the acquired information on the presence or absence of debris onto the support image; and the output unit causes the support image with the superimposed image representing the information on the presence or absence of debris to be displayed on the display device, as described in any one of Notes 1 to 3.<Note 5> A learning device comprising: a preprocessing unit that associates a surgical field image including an image of the luminal tissue to be anastomosed with a correct label indicating the coordinates of the stump of the luminal tissue; and a learning unit that inputs the surgical field image associated with the correct label indicating the coordinates of the stump as a training image into a learning model, obtains the coordinates of the stump of the luminal tissue in the surgical field image from the learning model as one of the inference results, and generates a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the correct label indicating the coordinates of the stump. <Note 6> The learning device according to Note 5, wherein the preprocessing unit associates the surgical field image with a correct label indicating damage information of the intima of the luminal tissue, the learning unit inputs the surgical field image associated with the correct label indicating the intima damage information as a training image into the learning model, obtains the intima damage information from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the intima damage information obtained from the learning model and the correct label indicating the intima damage information. <Note 7> The learning device according to Note 5 or Note 6, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of embolic factors in the lumen of the tubular tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of embolic factors as a training image into the learning model, obtains the presence or absence of embolic factors from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the presence or absence of embolic factors obtained from the learning model and the correct label indicating the presence or absence of embolic factors. <Note 8> The learning device according to any one of Notes 5 to 7, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of debris around the luminal tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of debris as a training image into the learning model, obtains the presence or absence of debris from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the presence or absence of debris obtained from the learning model and the correct label indicating the presence or absence of debris.<Note 9> A surgical support method performed by a surgical support device, comprising the steps of: inputting a surgical field image including an image of luminal tissue to be anastomosed as an inference target image into a trained model; obtaining the coordinates of the stump of the luminal tissue in the surgical field image from the trained model as one of the inference results; generating an image for supporting anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image onto the surgical field image based on the obtained coordinates of the stump; and displaying the support image on a display device. <Note 10> A method for generating a learning model executed by a learning device, comprising the steps of: associating a surgical field image including an image of luminal tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the luminal tissue; inputting the surgical field image associated with the ground truth label indicating the coordinates of the stump as a training image into a learning model; obtaining the coordinates of the stump of the luminal tissue in the surgical field image from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump. <Note 11> A program for causing a computer to perform the following steps: input a surgical field image including an image of the luminal tissue to be anastomosed as an inference target image into a trained model, and obtain the coordinates of the stump of the luminal tissue in the surgical field image from the trained model as one of the inference results; generate an image to support the anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image onto the surgical field image based on the obtained coordinates of the stump; and display the support image on a display device. <Note 12> A program for causing a computer to perform the following steps: a procedure to associate a surgical field image including an image of the luminal tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the luminal tissue; inputting the surgical field image associated with the ground truth label indicating the coordinates of the stump as a training image into a learning model; obtaining the coordinates of the stump of the luminal tissue in the surgical field image from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump.

[0126] <Note 1A> A surgical support device comprising: an inference unit that inputs a surgical field image including an image of vascular tissue to be anastomosed as an inference target image to a trained model, obtains the coordinates of the stump of the vascular tissue in the surgical field image from the trained model as one of the inference results, and further obtains intimal damage information of the vascular tissue from the trained model as one of the inference results; an image processing unit that generates an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intimal damage information onto the surgical field image based on the obtained coordinates of the stump and the intimal damage information, and generating an advice image including a string image according to the damage level of the intimal damage information; and an output unit that displays the support image on a display device. <Note 2A> The surgical support device according to Note 1A, wherein the advice image includes a string image indicating that a new stump should be exposed in the vascular tissue or that anastomosis should be performed with another vascular tissue. <Note 3A> The inference unit further acquires information on the presence or absence of embolic factors in the lumen of the vascular tissue as one of the inference results from the trained model; the support image processing unit further superimposes an image representing the acquired information on the presence or absence of embolic factors onto the support image; and the output unit causes the support image with the superimposed image representing the information on the presence or absence of embolic factors to be displayed on the display device, as described in Note 1A or Note 2A. <Note 4A> The inference unit further acquires information on the presence or absence of debris around the vascular tissue as one of the inference results from the trained model; the support image processing unit further superimposes an image representing the acquired information on the presence or absence of debris onto the support image; and the output unit causes the support image with the superimposed image representing the information on the presence or absence of debris to be displayed on the display device, as described in any one of Notes 1A to 3A.<Note 5A> A learning device comprising: a preprocessing unit that associates a surgical field image including an image of vascular tissue to be anastomosed with a correct label indicating the coordinates of the stump of the vascular tissue and the intimal damage information of the vascular tissue; and a learning unit that inputs the surgical field image associated with the correct label indicating the coordinates of the stump and the intimal damage information as a training image into a learning model, obtains the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results, further obtains the intimal damage information from the learning model as one of the inference results, and generates a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the correct label indicating the coordinates of the stump, and the comparison result between the intimal damage information obtained from the learning model and the correct label indicating the intimal damage information. <Appendix 6A> The learning device according to Appendix 5A, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of embolic factors in the lumen of the vascular tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of embolic factors as a training image into the learning model, obtains the presence or absence of embolic factors from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the presence or absence of embolic factors obtained from the learning model and the correct label indicating the presence or absence of embolic factors. <Note 7A> The learning device according to Note 5A or Note 6A, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of debris around the vascular tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of debris as a training image to the learning model, obtains the presence or absence of debris from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the presence or absence of debris obtained from the learning model and the correct label indicating the presence or absence of debris.<Note 8A> A surgical support method performed by a surgical support device, comprising the steps of: inputting a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model; obtaining the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results; and further obtaining information on the intima of the vascular tissue's damage from the trained model as one of the inference results; superimposing an image showing the region of the stump in the surgical field image and an image showing the intima damage information onto the surgical field image based on the obtained coordinates of the stump and the intima damage information; and generating an image to support the vascular tissue anastomosis surgery by generating an advice image including a string image according to the level of damage of the intima damage information; and displaying the support image on a display device. <Note 9A> A method for generating a learning model executed by a learning device, comprising the steps of: associating a surgical field image including an image of vascular tissue to be anastomosed with a correct label indicating the coordinates of the stump of the vascular tissue and the intima damage information of the vascular tissue; inputting the surgical field image associated with the coordinates of the stump and the correct label indicating the intima damage information of the vascular tissue as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results; further obtaining the intima damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the correct label indicating the coordinates of the stump, and the comparison result between the intima damage information obtained from the learning model and the correct label indicating the intima damage information.<Note 10A> A program for causing a computer to perform the following steps: input a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model, obtain the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results, and further obtain information on the intima of the vascular tissue's damage from the trained model as one of the inference results; generate an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intima damage information onto the surgical field image based on the obtained coordinates of the stump and the intima damage information, and generating an advice image including a string image according to the level of damage of the intima damage information; and display the support image on a display device. <Note 11A> A program for causing a computer to perform the following steps: a procedure for associating a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and information on intima damage to the vascular tissue; inputting the surgical field image, which has been associated with the ground truth label indicating the coordinates of the stump and information on intima damage to the vascular tissue, as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results; further obtaining the intima damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and the comparison result between the intima damage information obtained from the learning model and the ground truth label indicating the intima damage information.

[0127] The present invention is applicable to a system that assists in surgical procedures involving the anastomosis of tubular tissues (not only in plastic surgery or reconstructive treatment, but also, for example, in cardiac surgery or neurosurgery). This system is applicable to education regarding anastomotic surgery. Furthermore, this system is applicable to navigation during anastomotic surgery.

[0128] 1a, 1b... Surgical support system, 2... Learning device, 3a, 3b... Exoscopy, 4... Exoscopy display device, 5a, 5b... Surgical support device, 6... Support display device, 7... Patient, 21... Storage device, 22... Operation unit, 23... Display unit, 24... Pre-processing unit, 25... Learning unit, 26... Evaluation unit, 27... Communication unit, 31... Communication unit, 32... Output unit, 33a, 33b... Exoscopy image processing unit, 34... Storage device, 35... Arm, 36... Camera, 51... Communication unit, 52... Storage device 53...Operation unit, 54...Inference unit, 55a, 55b...Support image processing unit, 56...Output unit, 101...Surgical field image, 102...Surgical field image, 103...Surgical field image, 104...Surgical field image, 105...Surgical field image, 106...Support image, 107...Surgical field image, 108...Support image, 109...Support image, 110...Support image, 111...Support image, 112...Support image, 113...Support image, 201...Blood vessel, 202...Blood vessel, 203...Blood vessel, 204 ...blood vessel, 205...blood vessel, 206...blood vessel, 207...blood vessel, 208...blood vessel, 209...blood vessel, 210...blood vessel, 301...clip, 302...clip, 303...gauze, 304...tweezers, 401...stump, 402...injury, 403...thrombus, 404...debris, 405...stump, 406...stump, 407...injury, 408...stump, 409...stump, 410...injury, 411...stump, 412...thrombus, 413...debris, 414...stump, 415...stump, 416...Injury, 501...Surgical field image, 601...Stump region image, 602...Injury area image, 603...Thrombosis area image, 604...Debris area image, 605...Stump region image, 606...Stump region image, 607...Injury area image, 701...Stump region image, 702...Injury area image, 703...Thrombosis area image, 704...Debris area image, 705...Stump region image, 706...Stump region image, 801...Advice image, 802...Thin arrow, 803...Variable arrow

Claims

1. A surgical support device comprising: an inference unit that inputs a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model, obtains the coordinates of the vascular tissue's stump in the surgical field image as one of the inference results from the trained model, and further obtains intimal damage information of the vascular tissue as one of the inference results from the trained model; a support image processing unit that generates an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intimal damage information onto the surgical field image based on the obtained coordinates of the stump and the intimal damage information, and generating an advice image including a string image according to the damage level of the intimal damage information; and an output unit that displays the support image on a display device.

2. The surgical support device according to claim 1, wherein the advice image includes the text image of exposing a new stump to the vascular tissue or performing an anastomosis with another vascular tissue.

3. The surgical support device according to claim 1 or 2, wherein the inference unit further obtains information on the presence or absence of embolic factors in the lumen of the vascular tissue as one of the inference results from the trained model, the support image processing unit further superimposes an image representing the obtained information on the presence or absence of embolic factors onto the support image, and the output unit causes the support image with the image representing the presence or absence of embolic factors superimposed on it to be displayed on the display device.

4. The surgical support device according to claim 1 or 2, wherein the inference unit further acquires information on the presence or absence of debris around the vascular tissue as one of the inference results from the trained model, the support image processing unit further superimposes an image representing the acquired information on the presence or absence of debris onto the support image, and the output unit causes the support image with the image representing the presence or absence of debris superimposed on it to be displayed on the display device.

5. A learning device comprising: a preprocessing unit that associates a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and the intimal damage information of the vascular tissue; and a learning unit that inputs the surgical field image associated with the ground truth label indicating the coordinates of the stump and the intimal damage information as a training image into a learning model, obtains the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results, further obtains the intimal damage information from the learning model as one of the inference results, and generates a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and the comparison result between the intimal damage information obtained from the learning model and the ground truth label indicating the intimal damage information.

6. The learning device according to claim 5, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of embolic factors in the lumen of the vascular tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of embolic factors as a training image into the learning model, obtains the presence or absence of embolic factors as one of the inference results from the learning model, and generates the trained model from the learning model based on the comparison result between the presence or absence of embolic factors obtained from the learning model and the correct label indicating the presence or absence of embolic factors.

7. The learning device according to claim 5, wherein the preprocessing unit associates the surgical field image with a correct label indicating the presence or absence of debris around the vascular tissue, the learning unit inputs the surgical field image associated with the correct label indicating the presence or absence of debris as a training image to the learning model, obtains the presence or absence of debris from the learning model as one of the inference results, and generates the trained model from the learning model based on the comparison result between the presence or absence of debris obtained from the learning model and the correct label indicating the presence or absence of debris.

8. A surgical support method performed by a surgical support device, comprising the steps of: inputting a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model; obtaining the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results; and further obtaining information on the intima of the vascular tissue's damage from the trained model as one of the inference results; superimposing an image showing the region of the stump in the surgical field image and an image showing the intima damage information onto the surgical field image based on the obtained coordinates of the stump and the intima damage information; and generating an image to support the vascular tissue anastomosis surgery by generating an advice image including a string image according to the damage level of the intima damage information; and displaying the support image on a display device.

9. A method for generating a learning model performed by a learning device, comprising the steps of: associating a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and information on intima damage to the vascular tissue; inputting the surgical field image associated with the ground truth label indicating the coordinates of the stump and information on intima damage to the vascular tissue as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results; further obtaining the intima damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on a comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and a comparison result between the intima damage information obtained from the learning model and the ground truth label indicating the intima damage information.

10. A program for causing a computer to perform the following steps: input a surgical field image including an image of vascular tissue to be anastomosed as an inference target image into a trained model, obtain the coordinates of the vascular tissue's stump in the surgical field image from the trained model as one of the inference results, and further obtain information on the intimal damage of the vascular tissue from the trained model as one of the inference results; generate an image to support the vascular tissue anastomosis surgery by superimposing an image showing the region of the stump in the surgical field image and an image showing the intimal damage information onto the surgical field image based on the obtained coordinates of the stump and the intimal damage information, and generating an advice image including a string image according to the damage level of the intimal damage information; and display the support image on a display device.

11. A program for causing a computer to perform the following steps: a procedure for associating a surgical field image including an image of vascular tissue to be anastomosed with a ground truth label indicating the coordinates of the stump of the vascular tissue and information on intimal damage to the vascular tissue; inputting the surgical field image, which has been associated with the ground truth label indicating the coordinates of the stump and information on intimal damage to the vascular tissue, as a training image into a learning model; obtaining the coordinates of the stump of the vascular tissue in the surgical field image from the learning model as one of the inference results; further obtaining the intimal damage information from the learning model as one of the inference results; and generating a trained model from the learning model based on the comparison result between the coordinates of the stump obtained from the learning model and the ground truth label indicating the coordinates of the stump, and the comparison result between the intimal damage information obtained from the learning model and the ground truth label indicating the intimal damage information.

Images