Computer program, learned model generation method, image processing method, and image processing device

Abstract

The present invention provides a computer program, a learned model generation method, an image processing method, and an image processing device for easily acquiring the risk of nerve damage.　A computer executes processing for supporting endoscopic surgery according to a computer program according to the present invention. The computer program causes the computer to execute processing of acquiring an endoscopic image of the inside of the body, photographed by an endoscope, and outputting, on the basis of the acquired endoscopic image, information indicating a risk of damaging a nerve inside the body.

Description

Computer program, trained model generation method, image processing method, and image processing device

[0001] The present invention relates to a computer program, a trained model generation method, an image processing method, and an image processing device used in endoscopic surgery.

[0002] One surgical technique is endoscopic surgery, in which an endoscope or surgical instruments are inserted into the body to perform the surgery. To assist in endoscopic surgery, technology has been developed that displays endoscopic images of the inside of the body captured by an endoscope, allowing a doctor to operate the surgical instruments while viewing the endoscopic images. Patent Document 1 discloses an example of technology for assisting endoscopic surgery.

[0003] Nerve paralysis is one of the complications that can occur after surgical procedures. For example, recurrent laryngeal nerve paralysis can occur after cervical or esophageal surgery. Nerve paralysis occurs due to nerve damage such as overextension. During endoscopic surgery, nerve damage can occur due to manipulation with surgical tools. To assess the risk of nerve damage, intraoperative nerve monitoring (IONM) is used, which involves electrically stimulating the nerve and observing the nerve's action potential using a tube equipped with electrodes attached to the nerve.

[0004] International Publication No. 2021 / 152784

[0005] When using IONM, muscle relaxants cannot be used because they significantly reduce nerve response, and the practitioner must attach a tube with electrodes to the nerve to observe the nerve's action potential, which places a burden on the practitioner.

[0006] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a computer program, a trained model generation method, an image processing method, and an image processing device for easily obtaining the risk of nerve damage.

[0007] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute a process of acquiring endoscopic images of the inside of a body taken by an endoscope and outputting information indicating the risk of damage to nerves inside the body based on the acquired endoscopic images.

[0008] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute a process of inputting an acquired endoscopic image into a first trained model that outputs risk information regarding the risk of nerve damage when an endoscopic image is input, acquiring the risk information output by the first trained model, and outputting information indicating the risk of nerve damage related to the acquired endoscopic image in accordance with the acquired risk information.

[0009] In a computer program according to one aspect of the present invention, the risk includes a risk of overstretching, overbending, or thermal damage to the nerve caused by a surgical instrument.

[0010] A computer program according to one aspect of the present invention causes a computer to execute a process of outputting information indicating the risk in a format according to the level of the risk.

[0011] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute a process of inputting an acquired endoscopic image into a second trained model that detects the recurrent laryngeal nerve in the endoscopic image when the endoscopic image is input, obtaining a detection result of the recurrent laryngeal nerve output by the second trained model, and coloring the recurrent laryngeal nerve in the acquired endoscopic image according to the detection result before outputting the endoscopic image.

[0012] A computer program according to one aspect of the present invention is characterized in that it causes a computer to execute a process of adjusting the display form of the recurrent laryngeal nerve in the endoscopic image in accordance with the information indicating the risk.

[0013] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute processes that individually switch between outputting or ceasing output of information indicating the risk and coloring or ceasing coloring of the recurrent laryngeal nerve in an endoscopic image.

[0014] A computer program according to one aspect of the present invention causes a computer to execute a process for adjusting the operation of a surgical instrument in accordance with the information indicating the risk.

[0015] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute the following processes: acquire the positions of nerves and surgical instruments in an acquired endoscopic image; acquire the depth positions of the nerves and surgical instruments in the acquired endoscopic image; determine whether the surgical instruments are in a state in which they generate heat; determine the risk of thermal damage to the nerves based on the positions and depth positions of the nerves and surgical instruments in the acquired endoscopic image and the state of the surgical instruments; and output information indicating the determined risk.

[0016] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute a process of acquiring the position of a nerve in an acquired endoscopic image, acquiring the depth position of the nerve in the acquired endoscopic image, setting a plurality of points contained in the nerve in the acquired endoscopic image, measuring the change in distance between points over time or the change in angle between a plurality of straight lines connecting the points based on the positions and depth positions of the plurality of points, determining the risk of overextension or overflexion of the nerve based on the change in distance over time or the change in angle over time, and outputting information indicating the determined risk.

[0017] A computer program according to one embodiment of the present invention is characterized in that it causes a computer to execute a process of outputting information indicating points among the plurality of points that are at high risk of hyperextension or hyperflexion occurring.

[0018] A trained model generation method according to one embodiment of the present invention is characterized in that it acquires training data including endoscopic images of the inside of a body captured by an endoscope and risk information regarding the risk of damage to nerves inside the imaged body, and generates a trained model that outputs risk information when an endoscopic image is input by learning using the training data.

[0019] A trained model generation method according to one embodiment of the present invention is characterized in that it generates training data by acquiring endoscopic images, acquiring risk information using intraoperative neuromonitoring, and associating the risk information with the endoscopic images acquired at the time the risk information was acquired.

[0020] An image processing method according to one aspect of the present invention is characterized in that it acquires an endoscopic image of the inside of a body taken by an endoscope, and outputs information indicating the risk of damage to nerves inside the body based on the acquired endoscopic image.

[0021] An image processing device according to one embodiment of the present invention includes a calculation unit that acquires endoscopic images of the inside of the body taken by an endoscope, and outputs information indicating the risk of damage to nerves inside the body based on the acquired endoscopic images.

[0022] In one embodiment of the present invention, when using an endoscope, information indicating the risk of nerve damage is output based on endoscopic images taken of the inside of a patient's body. The user can perform endoscopic surgery while checking the output risk. Furthermore, compared to using IONM, the burden on the surgeon is reduced, and muscle relaxants can be used.

[0023] In one aspect of the present invention, risk information is acquired using a first trained model that outputs risk information regarding the risk of nerve damage when an endoscopic image is input, and information indicating the risk of nerve damage is output according to the risk information. The first trained model is generated by learning using training data including endoscopic images and risk information. The risk of nerve damage during endoscopic surgery is related to the shape of the nerve and the positional relationship between the nerve and the surgical instrument. Therefore, risk information regarding the risk of nerve damage is obtained based on an endoscopic image that shows the shape of the nerve and the positional relationship between the nerve and the surgical instrument.

[0024] In one embodiment of the present invention, the risk of nerve damage includes the risk of nerve overstretching, overbending, or thermal injury caused by a surgical instrument. Possible nerve damage includes overstretching, overbending, or thermal injury of the nerve, and the risk of these injuries occurring is obtained.

[0025] In one aspect of the present invention, information indicating the risk of nerve damage is output in a form corresponding to the level of risk, allowing the user to recognize the level of risk of nerve damage.

[0026] In one aspect of the present invention, a recurrent laryngeal nerve detection result is obtained using a second trained model that outputs a recurrent laryngeal nerve detection result when an endoscopic image is input, and an endoscopic image in which the recurrent laryngeal nerve is colored is output. The user can confirm the position of the recurrent laryngeal nerve when visually viewing the endoscopic image.

[0027] In one aspect of the present invention, the display form of the recurrent laryngeal nerve is adjusted according to the risk of nerve damage, and the user can visually recognize the recurrent laryngeal nerve and recognize the risk of nerve damage according to the change in the display form of the recurrent laryngeal nerve, such as a change in color.

[0028] In one embodiment of the present invention, switching is performed between outputting or not outputting information indicating the risk of nerve damage and between coloring or not coloring the recurrent laryngeal nerve, and an endoscopic image can be output in a state where information indicating the risk of nerve damage is not output or where the recurrent laryngeal nerve is not emphasized.

[0029] In one aspect of the present invention, the operation of a surgical instrument is adjusted in accordance with information indicating the risk of nerve damage, and the operation of the surgical instrument can be controlled to reduce the risk of nerve damage, thereby preventing nerve damage.

[0030] In one aspect of the present invention, the positions and depth positions of nerves and surgical instruments in endoscopic images are estimated, and it is determined whether the surgical instrument is in a state where it generates heat, and the risk of thermal damage to the nerve is judged. If the surgical instrument is close to the nerve and in a state where it generates heat, it can be determined that there is a high risk of thermal damage to the nerve.

[0031] In one aspect of the present invention, a plurality of points included in a nerve in an endoscopic image are set, and the change over time in the distance between the points or the angle between the lines connecting the points is measured. Based on the measured change over time, the risk of the nerve being overextended or overflexed is determined. If the increase in the distance between two points or the change in the angle between two lines connecting one point and each of the adjacent points is large, it can be determined that there is a high risk of the nerve being overextended or overflexed. Furthermore, by outputting information indicating the points at high risk of being overextended or overflexed, the user can identify areas of the recurrent laryngeal nerve at high risk of being overextended or overflexed.

[0032] The present invention has excellent effects such as easily reducing the risk of nerve damage.

[0033] 1 is a block diagram showing an example of the configuration of a surgery system for performing endoscopic surgery. FIG. 1 is a block diagram showing an example of the internal configuration of an image processing device according to embodiment 1. FIG. 2 is a conceptual diagram showing an example of the function of a nerve detection model. FIG. 3 is a conceptual diagram showing an example of the function of a risk detection model. FIG. 4 is a flowchart showing an example of processing procedures according to embodiment 1 executed by the image processing device when an endoscope is used. FIG. 5 is a schematic diagram showing an example of an endoscopic image. FIG. 6 is a schematic diagram showing an example of an endoscopic image in which the recurrent laryngeal nerve is emphasized. FIG. 7 is a schematic diagram showing an example of output of information indicating the risk of nerve damage. FIG. 8 is a schematic diagram showing an example of output of information indicating the risk of nerve damage. FIG. 9 is a block diagram showing an example of the internal functional configuration of a learning device. FIG. 10 is a flowchart showing an example of processing procedures by which the learning device generates a risk detection model. FIG. 11 is a block diagram showing an example of the internal configuration of an image processing device according to embodiment 2. FIG. 12 is a conceptual diagram showing an example of the function of an instrument detection model. FIG. 13 is a conceptual diagram showing an example of the function of a depth estimation model. FIG. 14 is a flowchart showing an example of processing procedures according to embodiment 2 executed by the image processing device when an endoscope is used. FIG. 15 is a flowchart showing an example of processing procedures according to embodiment 2 executed by the image processing device when an endoscope is used. FIG. 16 is a schematic diagram showing an example of setting a plurality of points included in the recurrent laryngeal nerve in an endoscopic image. 10A and 10B are schematic diagrams showing examples of notification images superimposed on an endoscopic image.

[0034] The present invention will be described in detail below with reference to the drawings illustrating embodiments. <Embodiment 1> In endoscopic surgery, an endoscope and surgical instruments are inserted into the body, the inside of the body is photographed with the endoscope, and the surgeon performs surgery by manipulating the surgical instruments while viewing the endoscopic image of the inside of the body. Endoscopes include thoracoscopes and laparoscopes, which are inserted by drilling a hole in the patient's body. Surgical instruments include forceps and electrocautery. During endoscopic surgery, nerve damage may occur due to the use of surgical instruments. Nerve damage can lead to nerve paralysis. For example, recurrent laryngeal nerve paralysis may occur as a complication after endoscopic surgery of the neck or esophagus. In this embodiment, safety of endoscopic surgery is ensured by outputting information indicating the risk of nerve damage depending on the situation when using the endoscope.

[0035] FIG. 1 is a block diagram showing an example configuration of a surgical system 100 for performing endoscopic surgery. Endoscopic surgery is performed on a patient 4. An endoscope 21 and a surgical instrument 22 are inserted into the body of the patient 4. The endoscope 21 generates an endoscopic image capturing the inside of the body of the patient 4. The endoscopic image is a color image. A plurality of surgical instruments 22, such as forceps or an electric scalpel, may be inserted into the body of the patient 4. For example, the surgical instrument 22 is connected to a drive mechanism (not shown), such as a robot arm, for driving the surgical instrument 22, and operates inside the body of the patient 4.

[0036] The surgical instrument 22 is connected to a control device 31 that controls the operation of the surgical instrument 22. The control device 31 is configured using a computer including a memory and an arithmetic unit that executes calculations to control the operation of the surgical instrument 22. For example, the control device 31 controls the operation of the drive mechanism that drives the surgical instrument 22, thereby controlling the operation of the surgical instrument 22. An operation unit 33 that is operated by a user 5, such as a surgeon, is connected to the control device 31. The operation unit 33 is configured using a touch panel, a pointing device, a button, a lever, or the like. The user 5 operates the operation unit 33, and the control device 31 controls the operation of the surgical instrument 22 in accordance with the operation received by the operation unit 33.

[0037] The endoscope 21 is connected to an image processing device 1. The image processing device 1 executes an image processing method for outputting information indicating the risk of nerve damage based on an endoscopic image. The image processing device 1 is connected to a control device 31. A display unit 32 for displaying an image is connected to the image processing device 1. The display unit 32 is, for example, a liquid crystal display or an electroluminescent display (EL display). The endoscope 21 inputs the generated endoscopic image to the image processing device 1, which acquires the endoscopic image input from the endoscope 21 and displays the endoscopic image on the display unit 32. The display unit 32 displays an endoscopic image of the inside of the body of the patient 4. The user 5 visually checks the displayed endoscopic image to confirm the inside of the body of the patient 4. While viewing the endoscopic image, the user 5 operates the operation unit 33 and operates the surgical instrument 22 to perform endoscopic surgery.

[0038] In this embodiment, an example will be described in which endoscopic surgery is performed on the neck or esophagus of a patient 4 using the surgical system 100, and information indicating the risk of damage to the recurrent laryngeal nerve is output. An endoscope 21 and a surgical instrument 22 are inserted into the body of the patient 4. The endoscope 21 captures images of the inside of the body. The surgical instrument 22 performs work inside the body, such as incising tissue, suturing, or removing lesions.

[0039] FIG. 2 is a block diagram showing an example of the internal configuration of the image processing device 1 according to the first embodiment. The image processing device 1 is configured using a computer such as a personal computer or a server device. The image processing device 1 executes a computer operation method. The image processing device 1 includes a calculation unit 11, a memory 12, a reading unit 13, and a storage unit 14. The calculation unit 11 is configured using, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a multi-core CPU. The calculation unit 11 may also be configured using a quantum computer. The memory 12 stores temporary data generated in conjunction with calculations. The memory 12 is, for example, a RAM (Random Access Memory). The reading unit 13 reads information from a recording medium 10 such as an optical disc or a portable memory. The storage unit 14 is non-volatile, for example, a hard disk or non-volatile semiconductor memory.

[0040] An endoscope 21, a control device 31, and a display unit 32 are connected to the image processing device 1. For example, the endoscope 21, the control device 31, and the display unit 32 are connected to the calculation unit 11 via a communication interface (not shown). An endoscopic image is input from the endoscope 21 to the image processing device 1. The calculation unit 11 transmits a control signal to the control device 31. The calculation unit 11 also transmits image data to the display unit 32, and causes the display unit 32 to display an image based on the image data.

[0041] The calculation unit 11 causes the reading unit 13 to read the computer program 141 recorded on the recording medium 10, and stores the read computer program 141 in the storage unit 14. The calculation unit 11 executes processing to realize the functions of the image processing device 1 in accordance with the computer program 141. The computer program 141 may be a computer program product. The computer program 141 may be stored in the storage unit 14 in advance, or may be downloaded from outside the image processing device 1. In this case, the image processing device 1 does not need to include the reading unit 13.

[0042] The computer program 141 can be deployed to run on a single computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network. That is, the image processing device 1 may be configured with multiple computers, and the computer program 141 may be executed on multiple computers connected via a communications network. The image processing device 1 may be configured using a cloud server.

[0043] The computer program 141 can be deployed to run on a single computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network. That is, the image processing device 1 may be configured with multiple computers, and the computer program 141 may be executed on multiple computers connected via a communications network. The image processing device 1 may be configured using a cloud server. The image processing device 1 and the control device 31 may be configured integrally.

[0044] The image processing device 1 includes a nerve detection model 142 that outputs a detection result of the recurrent laryngeal nerve contained in an endoscopic image when an endoscopic image capturing the inside of the body of a patient 4 is input. The nerve detection model 142 is a trained model that has been trained in advance to output a detection result of the recurrent laryngeal nerve when an endoscopic image is input. The nerve detection model 142 corresponds to a second trained model.

[0045] FIG. 3 is a conceptual diagram illustrating an example of the function of the nerve detection model 142. An endoscopic image of the inside of the body of a patient 4 is input to the nerve detection model 142. The nerve detection model 142 is trained in advance so that when an endoscopic image is input, the nerve detection model 142 outputs the position of the recurrent laryngeal nerve in the endoscopic image. The nerve detection model 142 is a trained model that performs segmentation. For example, the nerve detection model 142 is realized using U-net or YOLO (You Only Look Once). Alternatively, the nerve detection model 142 may be realized using a CNN (Convolutional Neural Network). The nerve detection model 142 may use other trained models.

[0046] For example, the nerve detection model 142 outputs the position of the recurrent laryngeal nerve by outputting information indicating whether each pixel in the endoscopic image corresponds to the recurrent laryngeal nerve. The nerve detection model 142 may also output information indicating a range of pixels included in the endoscopic image that correspond to the recurrent laryngeal nerve. The nerve detection model 142 may also output an image in which the recurrent laryngeal nerve is extracted from the endoscopic image. In this manner, the nerve detection model 142 outputs a detection result of the recurrent laryngeal nerve. By the nerve detection model 142 outputting the detection result of the recurrent laryngeal nerve according to the endoscopic image, it becomes possible to detect the recurrent laryngeal nerve using the nerve detection model 142.

[0047] The nerve detection model 142 is realized by the calculation unit 11 executing information processing in accordance with the computer program 141. The storage unit 14 stores data necessary to realize the nerve detection model 142. A method for training the nerve detection model 142 will be described later.

[0048] Furthermore, the image processing device 1 is equipped with a risk detection model 143 that, when an endoscopic image of the inside of the body of the patient 4 is input, outputs risk information regarding the risk that nerves inside the body and present in the area imaged by the endoscope 21 will be damaged by the surgical instrument 22. The risk detection model 143 is a trained model that has been trained in advance to output risk information when an endoscopic image is input. The risk detection model 143 corresponds to the first trained model.

[0049] Nerve damage includes, for example, hyperextension, hyperflexion, or thermal damage. Hyperextension or hyperflexion of the nerve can occur when the nerve is pulled or deformed during operation with the surgical instrument 22. Therefore, the risk of hyperextension or hyperflexion of the nerve is related to the shape of the nerve. Thermal damage of the nerve can occur when a heated surgical instrument 22, such as an electric scalpel, approaches the nerve and the heat from the surgical instrument 22 is transmitted to the nerve. Therefore, the risk of thermal damage of the nerve is related to the positional relationship between the nerve and the surgical instrument 22. The shape of the nerve and the positional relationship between the nerve and the surgical instrument 22 can be obtained from endoscopic images of the inside of the body captured by the endoscope 21. Therefore, risk information regarding the risk of nerve damage can be obtained based on the endoscopic images.

[0050] The risk information is information that indicates the risk of nerve damage at multiple levels. More specifically, the risk information indicates the probability that the nerve included in the endoscopic image is normal or the probability of nerve damage. Alternatively, the risk information may indicate the degree of nerve damage that will occur. For example, the risk information indicates the probability of nerve damage or the degree of nerve damage in three levels: safe, caution, and warning. For example, the risk information indicates the probability of nerve damage or the degree of nerve damage using a continuous number or multiple discrete numbers. The risk information may include information regarding the risk of nerve overextension, overbending, and thermal damage caused by the surgical instrument 22.

[0051] 4 is a conceptual diagram showing an example of the function of the risk detection model 143. An endoscopic image of the inside of the body of the patient 4 is input to the risk detection model 143. The risk detection model 143 is trained in advance so that, when an endoscopic image is input, it outputs, as risk information, the probability that the nerve is normal, the probability that nerve hyperextension will occur, the probability that nerve hyperflexion will occur, and the probability that nerve thermal damage will occur. The risk detection model 143 may be configured to output risk information including the degree of nerve hyperextension, hyperflexion, and thermal damage that will occur, instead of the probability of nerve hyperextension, hyperflexion, and thermal damage occurring.

[0052] For example, the risk detection model 143 is realized using a CNN. The neural detection model 142 may use other trained models such as a transformer. The risk detection model 143 is realized by the calculation unit 11 executing image processing in accordance with the computer program 141. The storage unit 14 stores data necessary to realize the risk detection model 143. A method for training the risk detection model 143 will be described later.

[0053] The nerve detection model 142 or the risk detection model 143 may be configured using hardware. For example, the nerve detection model 142 or the risk detection model 143 may be configured using hardware including a processor and a memory that stores necessary programs and data. The nerve detection model 142 or the risk detection model 143 may be realized using a quantum computer. Alternatively, the nerve detection model 142 or the risk detection model 143 may be provided outside the image processing device 1, and the image processing device 1 may perform processing using the external nerve detection model 142 or the risk detection model 143. For example, the nerve detection model 142 or the risk detection model 143 may be realized using the cloud.

[0054] The following describes the processing performed by the surgical system 100 when the endoscope 21 is used. Fig. 5 is a flowchart showing an example of the processing procedure according to the first embodiment, which is executed by the image processing device 1 when the endoscope 21 is used. Hereinafter, steps are abbreviated as S. The calculation unit 11 executes information processing in accordance with the computer program 141, causing the image processing device 1 to execute the following processing.

[0055] A hole is opened in the body of the patient 4, and the endoscope 21 and the surgical instrument 22 are inserted into the body of the patient 4. Multiple surgical instruments 22 may be inserted into the body of the patient 4. In this embodiment, the endoscope 21 and the surgical instrument 22 are inserted into the neck of the patient 4 or near the esophagus. The endoscope 21 generates an endoscopic image capturing an image of the inside of the body of the patient 4 and inputs the generated endoscopic image to the image processing device 1. The image processing device 1 receives the endoscopic image from the endoscope 21 and acquires the endoscopic image (S101). In S101, the calculation unit 11 stores the input endoscopic image in the storage unit 14 and acquires the endoscopic image by reading the endoscopic image from the storage unit 14 to the memory 12. The image processing device 1 outputs the acquired endoscopic image (S102). In S102, the calculation unit 11 displays the endoscopic image on the display unit 32.

[0056] FIG. 6 is a schematic diagram showing an example of an endoscopic image. The displayed endoscopic image is a color image. The endoscopic image shows tissue present inside the body of the patient 4, as well as the surgical instrument 22. The user 5 observes the inside of the body of the patient 4 by visually viewing the endoscopic image displayed on the display unit 32. The user 5 operates the operation unit 33 while visually viewing the endoscopic image. The control device 31 controls the operation of the surgical instrument 22 in accordance with operations from the user 5 received by the operation unit 33. Endoscopic surgery is performed by using the surgical instrument 22 to perform tasks such as incising tissue. The image processing device 1 supports the performance of the endoscopic surgery by performing information processing in parallel with the performance of the endoscopic surgery.

[0057] The image processing device 1 inputs the endoscopic image to the nerve detection model 142 (S103). In S103, the calculation unit 11 inputs the endoscopic image acquired in S101 to the nerve detection model 142. The nerve detection model 142 performs calculations in response to the input of the endoscopic image and outputs the position of the recurrent laryngeal nerve in the endoscopic image. The image processing device 1 acquires the position of the recurrent laryngeal nerve (S104). In S104, the calculation unit 11 acquires the position of the recurrent laryngeal nerve output by the nerve detection model 142.

[0058] The image processing device 1 then emphasizes the recurrent laryngeal nerve in the endoscopic image to be output (S105). In S105, the calculation unit 11 generates an image in which the recurrent laryngeal nerve is emphasized in the endoscopic image and displays the generated image on the display unit 32. FIG. 7 is a schematic diagram showing an example of an endoscopic image in which the recurrent laryngeal nerve is emphasized. The calculation unit 11 generates an image in which the recurrent laryngeal nerve 41 is emphasized in the endoscopic image by coloring the recurrent laryngeal nerve 41 in the endoscopic image based on the acquired position of the recurrent laryngeal nerve. At this time, the calculation unit 11 colors the recurrent laryngeal nerve 41 in a color that is easy to see. For example, because the tissues inside the body shown in the endoscopic image are mainly red, the calculation unit 11 colors the recurrent laryngeal nerve 41 blue. The calculation unit 11 displays the endoscopic image on the display unit 32 after coloring the recurrent laryngeal nerve 41.

[0059] By displaying an endoscopic image in which the recurrent laryngeal nerve 41 is emphasized, the user 5 viewing the endoscopic image can recognize the recurrent laryngeal nerve 41 included in the endoscopic image. The user 5 operates the operation unit 33 to control the operation of the surgical instrument 22 according to the position of the recurrent laryngeal nerve 41. For example, the surgical instrument 22 is used to perform a procedure while avoiding the recurrent laryngeal nerve 41. The image processing device 1 may emphasize the recurrent laryngeal nerve 41 in the endoscopic image by a method other than coloring. For example, the calculation unit 11 may emphasize the position of the recurrent laryngeal nerve 41 by adding an arrow or a bounding box indicating the position of the recurrent laryngeal nerve 41 to the endoscopic image, or by blanking out the recurrent laryngeal nerve 41 in the endoscopic image.

[0060] The image processing device 1 then acquires risk information regarding the risk of nerve damage based on the endoscopic image (S106). In S106, the calculation unit 11 inputs the endoscopic image to the risk detection model 143. In this embodiment, the nerve is the recurrent laryngeal nerve. The risk detection model 143 performs calculations in response to the input of the endoscopic image and outputs risk information. For example, as shown in FIG. 4 , the risk detection model 143 outputs risk information including the probability that the nerve is normal, the probability that nerve hyperextension will occur, the probability that nerve hyperflexion will occur, and the probability that thermal damage will occur to the nerve. The calculation unit 11 acquires the risk information output by the risk detection model 143.

[0061] The image processing device 1 may acquire risk information by generating risk information based on rules without using the risk detection model 143. In this configuration, the calculation unit 11 generates risk information according to predetermined rules. The predetermined rules are included in the computer program 141. For example, the calculation unit 11 repeatedly acquires endoscopic images and the position of the recurrent laryngeal nerve, determines the shape of the nerve each time it acquires an endoscopic image and the position of the recurrent laryngeal nerve, and generates risk information at multiple levels depending on the degree of deformation of the nerve shape from the initial state. For example, the calculation unit 11 generates risk information indicating a high probability of nerve hyperextension or hyperflexion when the rate of change in nerve size or the magnitude of change in nerve angle over the course of surgery exceeds a predetermined threshold.

[0062] For example, the calculation unit 11 determines the position of the surgical instrument 22 in the endoscopic image, determines the distance between the surgical instrument 22 and a nerve, and generates risk information at multiple levels according to the distance between the surgical instrument 22 and the nerve. For example, the calculation unit 11 generates risk information indicating a high probability of thermal damage to the nerve when the distance between the surgical instrument 22, which is an electric scalpel, and the nerve is less than a predetermined threshold. The generated risk information may represent the degree of hyperextension, hyperflexion, and thermal damage of the nerve at multiple levels.

[0063] The image processing device 1 may be configured to use both the method of using the risk detection model 143 and the method of generating risk information on a rule-based basis. In this configuration, the calculation unit 11 obtains final risk information based on the risk information obtained using the risk detection model 143 and the risk information generated on a rule-based basis. For example, the calculation unit 11 obtains final risk information by averaging the risk information obtained using the risk detection model 143 and the risk information generated on a rule-based basis. For example, the calculation unit 11 obtains final risk information by adopting risk information that indicates a higher risk of nerve damage from the risk information obtained using the risk detection model 143 and the risk information generated on a rule-based basis.

[0064] The calculation unit 11 may select and acquire either risk information acquired using the risk detection model 143 or risk information generated on a rule basis. For example, which risk information to select is set in advance, and the calculation unit 11 selects one of the risk information in accordance with the setting. For example, the user 5 operates the operation unit 33 to input a selection instruction to the image processing device 1, and the calculation unit 11 selects one of the risk information in accordance with the input instruction. Alternatively, the calculation unit 11 may individually select either the information acquired using the risk detection model 143 or the information generated on a rule basis for information indicating the risk of occurrence of each of nerve hyperextension, hyperflexion, and thermal damage.

[0065] The image processing device 1 then outputs information indicating the risk of nerve damage according to the risk information (S107). In S107, the calculation unit 11 generates an image by adding information indicating the position of the recurrent laryngeal nerve and information indicating the risk of nerve damage according to the risk information to the endoscopic image, and displays the generated image on the display unit 32. The calculation unit 11 displays an image on the display unit 32 that includes information indicating that the higher the probability of nerve damage or the degree of nerve damage indicated by the risk information, the higher the risk of nerve damage. For example, an image including an icon or a warning message indicating the high risk of nerve damage is displayed.

[0066] The calculation unit 11 adds information indicating the risk of nerve damage to the endoscopic image in multiple forms according to multiple levels of risk of damage based on the risk information. For example, multiple different icons are predefined according to the multiple levels represented by the risk information, and the calculation unit 11 outputs the risk of nerve damage by adding an icon corresponding to the level of risk of damage represented by the risk information to the endoscopic image. For example, multiple warning messages are predefined according to the multiple levels, and the calculation unit 11 adds the warning message corresponding to the level of risk of damage represented by the risk information to the endoscopic image. The warning message is superimposed on the endoscopic image and displayed on the display unit 32 to warn of the risk of nerve damage. For example, the calculation unit 11 adds a risk bar or a safety bar to the endoscopic image, the length of which indicates the level of risk or safety depending on the level of risk of damage represented by the risk information. The user 5 viewing the endoscopic image can recognize the level of risk of nerve damage.

[0067] The calculation unit 11 may output the risk of nerve damage by changing the color of the endoscopic image displayed on the display unit 32. For example, the calculation unit 11 changes the color of the entire endoscopic image or a portion of the endoscopic image. For example, the calculation unit 11 changes the color of the frame of the endoscopic image. The calculation unit 11 may change the color of the entire endoscopic image, the color of a portion of the endoscopic image, or the color of the frame of the endoscopic image depending on the level of risk of damage based on the risk information. The calculation unit 11 may output the risk of nerve damage by changing the display form of the recurrent laryngeal nerve 41 in the endoscopic image. For example, the calculation unit 11 may change the color of the recurrent laryngeal nerve 41 depending on the level of risk of damage based on the risk information. For example, the calculation unit 11 may cause the recurrent laryngeal nerve 41 to blink when the risk is high. The user 5 can recognize the risk of nerve damage.

[0068] The calculation unit 11 may output information indicating the risk of occurrence of nerve hyperextension, hyperflexion, or thermal damage. For example, the calculation unit 11 outputs information indicating the risk of occurrence of the most dangerous damage among nerve hyperextension, hyperflexion, and thermal damage, which are indicated by the risk information. The calculation unit 11 may output information indicating the risk of occurrence of each of nerve hyperextension, hyperflexion, and thermal damage.

[0069] 8 to 10 are schematic diagrams showing examples of output of information indicating the risk of nerve damage. Fig. 8 shows an example of output when the surgical instrument 22 is far from the recurrent laryngeal nerve 41 and the risk of damaging the recurrent laryngeal nerve 41 is relatively low. Fig. 9 shows an example of output when the distance between the recurrent laryngeal nerve 41 and the surgical instrument 22 is short and the risk is higher. Fig. 10 shows an example of output when the surgical instrument 22 is in contact with the recurrent laryngeal nerve 41 and the risk is even higher.

[0070] A frame 321 is added to the endoscopic image, and a safety level bar 322, whose length indicates the safety level, is displayed. The calculation unit 11 determines the length of the safety level bar 322 according to the level of risk of damage represented by the risk information. Specifically, the calculation unit 11 determines that the lower the risk level, the higher the safety level, and makes the safety level bar 322 longer, and the higher the risk level, the lower the safety level, and makes the safety level bar 322 shorter. In the example shown in FIG. 8 , the risk is relatively low, so the safety level bar 322 is long, indicating that the safety level is relatively high. Furthermore, the color of the frame 321 and the safety level bar 322 indicates a low risk. For example, the color of the frame 321 and the safety level bar 322 is green.

[0071] In the example shown in FIG. 9 , an icon 323 indicating a higher risk of nerve damage and a warning message 324 saying "Device is Close," which means that the distance between the recurrent laryngeal nerve 41 and the surgical instrument 22 is too short, are superimposed on the endoscopic image. If the surgical instrument 22 is closer to the recurrent laryngeal nerve 41, the surgical instrument 22 is more likely to act on the recurrent laryngeal nerve 41, increasing the risk of damage. The safety level bar 322 is shorter, indicating lower safety, i.e., higher risk. The frame 321 and the safety level bar 322 are colored to indicate a higher risk. For example, the frame 321 and the safety level bar 322 are colored yellow.

[0072] In the example shown in FIG. 10 , an icon 323 indicating a high risk of nerve damage and a warning message 324 saying "Strong traction!", indicating that the surgical instrument 22 is exerting a strong traction force on the recurrent laryngeal nerve 41, are superimposed on the endoscopic image. When the recurrent laryngeal nerve 41 is retracted with a strong traction force, there is a very high risk of hyperextension. The safety level bar 322 is further shortened, indicating an even lower level of safety and an even higher risk. The colors of the recurrent laryngeal nerve 41, the frame 321, and the safety level bar 322 indicate an even higher risk. For example, the frame 321 and the safety level bar 322 are red.

[0073] 9 and 10 , the user 5 recognizes the risk of nerve damage and adjusts the operation of the operation unit 33 to reduce the risk, allowing the user to perform endoscopic surgery while preventing nerve damage. The image processing device 1 may output information indicating the risk of nerve damage by blinking the frame 321. The image processing device 1 may further change the icon 323 according to the level of risk. When multiple surgical instruments 22 are inserted into the body, the image processing device 1 may acquire risk information for each surgical instrument 22 and output a safety level bar 322 according to the multiple pieces of risk information.

[0074] The image processing device 1 may output a warning message 324 according to the level of risk of nerve overextension, overflexion, or thermal damage. For example, warning messages 324 such as "Nerve stretched," "Nerve stretched **%," or "Strong traction!" are displayed depending on the level of risk of nerve overextension. For example, warning messages 324 such as "Nerve flexion," "Nerve flexion ** degrees," or "The angle of flexion is excessive!" are displayed depending on the level of risk of nerve overflexion. For example, warning messages 324 such as "Device is close," "Device is in contact," or "Risk of thermal nerve damage!" are displayed depending on the level of risk of nerve thermal damage. The image processing device 1 may be configured to include an audio generating unit such as a speaker and output an audio warning according to the level of risk of nerve damage.

[0075] In S107, when the risk of nerve damage indicated by the risk information is low, the image processing device 1 may output information indicating the low risk, or may not output the information indicating the risk. For example, when the level of risk of nerve damage indicated by the risk information has not reached a predetermined level, the calculation unit 11 may display on the display unit 32 an image in which information indicating the low risk has been added to the endoscopic image, or may not add the information indicating the risk to the endoscopic image.

[0076] The image processing device 1 determines whether there is a high risk of nerve damage (S108). In S108, the calculation unit 11 determines the risk of nerve damage based on the risk information, and determines whether the risk is high. For example, the calculation unit 11 determines that the risk is high when the level of risk of damage represented by the risk information exceeds a predetermined level.

[0077] If the risk of nerve damage is high (S108: YES), the image processing device 1 adjusts the operation of the surgical instrument 22 to reduce the risk of nerve damage (S109). In S109, the calculation unit 11 transmits a control signal to the control device 31 to adjust the operation of the surgical instrument 22. The control device 31 receives the control signal and adjusts the operation of the surgical instrument 22 in accordance with the control signal. At this time, the calculation unit 11 transmits a control signal that reduces the risk of nerve damage. For example, the calculation unit 11 transmits a control signal instructing the surgical instrument 22 to be further spaced away from the recurrent laryngeal nerve 41, and the control device 31 controls the operation of the surgical instrument 22 to be spaced away from the recurrent laryngeal nerve 41 in accordance with the control signal.

[0078] In this way, by adjusting the operation of the surgical instrument 22 in accordance with the information indicating the risk of nerve damage, the operation of the surgical instrument 22 can be controlled to reduce the risk of nerve damage, thereby preventing nerve damage in the patient 4. The image processing device 1 may execute the processes of S108 and S109 before S107.

[0079] After S109 is completed, or if the risk of nerve damage is low (S108: NO), the image processing device 1 determines whether to discontinue the emphasis on the recurrent laryngeal nerve (S110). The user 5 operates the operation unit 33 to input an instruction to discontinue the emphasis on the recurrent laryngeal nerve to the control device 31, and the control device 31 inputs an instruction to discontinue the emphasis on the recurrent laryngeal nerve to the image processing device 1. In S110, if the instruction to discontinue the emphasis on the recurrent laryngeal nerve is input from the control device 31, the calculation unit 11 determines to discontinue the emphasis on the recurrent laryngeal nerve. If the instruction to discontinue the emphasis on the recurrent laryngeal nerve is not input, the calculation unit 11 determines not to discontinue the emphasis on the recurrent laryngeal nerve.

[0080] When it is determined that the emphasis on the recurrent laryngeal nerve should be discontinued (S110: YES), the image processing device 1 discontinues the emphasis on the recurrent laryngeal nerve (S111). In S111, the calculation unit 11 displays an endoscopic image in which the recurrent laryngeal nerve is not colored on the display unit 32. When the recurrent laryngeal nerve is enhanced by a method other than coloring, the calculation unit 11 displays an endoscopic image in which the recurrent laryngeal nerve is not enhanced on the display unit 32. The image processing device 1 may execute the processes of S110 and S111 at other times.

[0081] After S111 is completed, or if it is determined that emphasis on the recurrent laryngeal nerve should not be discontinued (S110: NO), the image processing device 1 determines whether to discontinue output of information indicating risk of nerve damage (S112). The user 5 operates the operation unit 33 to input an instruction to discontinue output of information indicating risk to the control device 31, and the control device 31 inputs an instruction to discontinue output of information indicating risk to the image processing device 1. In S112, if an instruction to discontinue output of information indicating risk is input from the control device 31, the calculation unit 11 determines to discontinue output of information indicating risk. If an instruction to discontinue output of information indicating risk is not input, the calculation unit 11 determines not to discontinue output of information indicating risk.

[0082] When it is determined that the output of the information indicating the risk of nerve damage should be stopped (S112: YES), the image processing device 1 stops the output of the information indicating the risk (S113). In S113, the calculation unit 11 displays the endoscopic image to which the information indicating the risk of nerve damage has not been added on the display unit 32. The image processing device 1 may execute the processes of S112 and S113 at other times.

[0083] S110 to S113 switch between highlighting and not highlighting the recurrent laryngeal nerve, and switch between outputting and not outputting information indicating the risk of nerve damage. By canceling the highlighting of the recurrent laryngeal nerve or the output of information indicating the risk of nerve damage, an endoscopic image in which the recurrent laryngeal nerve is not highlighted or an endoscopic image to which no information indicating the risk of nerve damage has been added is displayed on the display unit 32. The user 5 can check the endoscopic image in detail without being distracted by the highlighted recurrent laryngeal nerve or the information indicating the risk of nerve damage.

[0084] After S113 is completed, or if it is determined that the output of information indicating the risk of nerve damage will not be stopped (S112: NO), the image processing device 1 determines whether to terminate the process of outputting endoscopic images (S114). When the endoscopic surgery is to be terminated, the control device 31 inputs information indicating the end of the endoscopic surgery to the image processing device 1. The user 5 may operate the operation unit 33 to input an instruction to terminate the endoscopic surgery to the control device 31, and the control device 31 may input the information indicating the end of the endoscopic surgery to the image processing device 1. In S114, if information indicating the end of the endoscopic surgery is input from the control device 31, the calculation unit 11 determines to terminate the process of outputting endoscopic images. If information indicating the end of the endoscopic surgery is not input, the calculation unit 11 determines not to terminate the process of outputting endoscopic images.

[0085] If it is determined that the process of outputting the endoscopic image should not be terminated (S114: NO), the image processing device 1 returns the process to S101. The image processing device 1 repeats the processes of S101 to S114 in parallel with the execution of the endoscopic surgery. If it is determined that the process of outputting the endoscopic image should be terminated (S114: YES), the image processing device 1 terminates the information processing.

[0086] A method for training the nerve detection model 142 and the risk detection model 143 will be described. The nerve detection model 142 and the risk detection model 143 are trained by a learning device 6. FIG. 11 is a block diagram showing an example of the internal functional configuration of the learning device 6. The learning device 6 is a computer such as a server device or a personal computer. The learning device 6 executes a computer operation method. The learning device 6 includes a calculation unit 61, a memory 62, a storage unit 63, a reading unit 64, an operation unit 65, and a display unit 66. The calculation unit 61 is configured using, for example, a CPU, a GPU, or a multi-core CPU. The calculation unit 61 may also be configured using a quantum computer. The memory 62 stores temporary data generated in conjunction with calculations. The memory 62 is, for example, a RAM. The reading unit 64 reads information from a recording medium 60 such as an optical disk or a portable memory. The storage unit 63 is non-volatile, for example, a hard disk or a non-volatile semiconductor memory.

[0087] The operation unit 65 receives input of information by receiving operations from the user 5. The operation unit 65 is, for example, a keyboard, a pointing device, or a touch panel. The display unit 66 displays images. The display unit 66 is, for example, a liquid crystal display or an EL display.

[0088] The calculation unit 61 causes the reading unit 64 to read the computer program 631 recorded on the recording medium 60, and stores the read computer program 631 in the storage unit 63. The calculation unit 61 executes processing to realize the functions of the learning device 6 in accordance with the computer program 631. The computer program 631 may be a computer program product. The computer program 631 may be stored in the storage unit 63 in advance, or may be downloaded from outside the learning device 6. In this case, the learning device 6 does not need to be equipped with the reading unit 64.

[0089] The computer program 631 can be deployed to run on a single computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network. That is, the learning device 6 may be configured with multiple computers, and the computer program 631 may run on multiple computers connected via a communications network. The learning device 6 may be configured using a cloud server.

[0090] The learning device 6 is provided with a learning model 632 that is the basis for the nerve detection model 142 or the risk detection model 143. The learning device 6 trains the learning model 632 to generate the nerve detection model 142 or the risk detection model 143. The learning model 632 is realized by the calculation unit 61 executing processing in accordance with a computer program 631. The learning model 632 has the same configuration as the nerve detection model 142 or the risk detection model 143. The learning model 632 may be configured using hardware.

[0091] The storage unit 63 stores training data 633 for training the learning model 632 to generate the nerve detection model 142 or the risk detection model 143. The training data 633 for generating the nerve detection model 142 records a data set that associates an endoscopic image with the position of the recurrent laryngeal nerve in the endoscopic image. For example, a person checks the generated endoscopic image, finds the recurrent laryngeal nerve in the endoscopic image, and records the position of the found recurrent laryngeal nerve in association with the endoscopic image. The training data 633 records a plurality of data sets.

[0092] The learning device 6 generates the nerve detection model 142 by training the learning model 632 using the training data 633. The calculation unit 61 inputs the endoscopic image recorded in the training data 633 to the learning model 632 that is the basis of the nerve detection model 142. The learning model 632 performs calculations in response to the input of the endoscopic image and outputs the position of the recurrent laryngeal nerve.

[0093] The calculation unit 61 adjusts the calculation parameters of the learning model 632 so as to reduce the error between the position of the recurrent laryngeal nerve output by the learning model 632 and the position of the recurrent laryngeal nerve associated with the input endoscopic image. That is, the calculation unit 61 adjusts the parameters so that the output position of the recurrent laryngeal nerve approaches the position of the recurrent laryngeal nerve associated with the input endoscopic image. For example, the calculation unit 61 adjusts the parameters using an error backpropagation algorithm.

[0094] The calculation unit 61 performs machine learning of the learning model 632 by repeating processing using multiple data sets recorded in the training data 633 and adjusting the parameters of the learning model 632. By adjusting the parameters of the calculation of the learning model 632 in this manner, the calculation unit 61 generates the nerve detection model 142. The calculation unit 61 stores the adjusted final parameters in the storage unit 63.

[0095] The generated nerve detection model 142 is provided in the image processing device 1. For example, the final parameters of the adjusted learning model 632 are input to the image processing device 1 and stored in the storage unit 14. The nerve detection model 142 is realized by the calculation unit 11 executing information processing using the stored parameters.

[0096] The risk detection model 143 is generated by the learning device 6 executing a trained model generation method. The training data 633 used to generate the risk detection model 143 includes a data set that associates endoscopic images with risk information related to the risk of nerve damage in the endoscopic images. The risk information included in the training data 633 is obtained, for example, using intraoperative neuromonitoring (IONM).

[0097] In IONM, a tube with electrodes is attached, and electrical stimulation of the nerve and observation of the nerve action potential are performed. IONM is executed when an endoscope is used, and an endoscopic image and an action potential of the nerve are output. If an abnormality, such as disappearance, occurs in the action potential of the nerve observed by IONM, there is a possibility that damage to the nerve has occurred. The action potential of the nerve is acquired by an information processing device, and if an abnormality occurs in the action potential of the nerve, the information processing device generates risk information corresponding to the abnormality. For example, risk information expressing the probability of nerve damage or the degree of damage that has occurred at multiple levels is generated according to the abnormality of the action potential of the nerve. For example, risk information including the probability of the nerve being normal, the probability of nerve hyperextension, the probability of nerve hyperflexion, and the probability of thermal damage to the nerve is generated according to the action potential of the nerve is generated.

[0098] The information processing device generates a dataset in which the endoscopic image and the risk information are associated by associating and storing an endoscopic image acquired when an abnormality occurs in the nerve action potential with the generated risk information. The learning device 6 receives the generated dataset as input, and stores training data 633 in which the input dataset is recorded in the storage unit 63, thereby generating the training data 633. The dataset included in the training data 633 may be generated by a method other than a method using IONM. For example, a person checks the endoscopic image and estimates the risk information, and a dataset in which the endoscopic image and the estimated risk information are associated is recorded in the training data 633. The training data 633 includes multiple datasets recorded therein.

[0099] 12 is a flowchart showing an example of the processing steps by which the learning device 6 generates a risk detection model 143. The learning device 6 executes the following processing by the calculation unit 61 executing information processing in accordance with the computer program 631. The calculation unit 61 reads out training data 633 stored in the storage unit 63, and the learning device 6 acquires the training data 633 (S21). In S21, the learning device 6 may acquire the training data 633 by inputting the training data 633 from an external source.

[0100] The learning device 6 then generates the risk detection model 143 by training the learning model 632 using the training data 633 (S22). In S22, the calculation unit 61 inputs the endoscopic image recorded in the training data 633 to the learning model 632, which is the basis of the risk detection model 143. The learning model 632 performs calculations in response to the input of the endoscopic image and outputs risk information.

[0101] The calculation unit 61 adjusts the calculation parameters of the learning model 632 so as to reduce the error between the risk information output by the learning model 632 and the risk information associated with the input endoscopic image. That is, the calculation unit 61 adjusts the parameters so that the risk information associated with the input endoscopic image and the output risk information approximately match. For example, the calculation unit 61 adjusts the parameters using the backpropagation algorithm.

[0102] The calculation unit 61 performs machine learning of the learning model 632 by repeating processing using multiple data sets recorded in the training data 633 and adjusting the parameters of the learning model 632. By adjusting the calculation parameters of the learning model 632 in this manner, the calculation unit 61 generates the risk detection model 143. The calculation unit 61 stores the adjusted final parameters in the storage unit 63. After S22 is completed, the learning device 6 ends the processing.

[0103] The risk detection model 143 generated by S21 to S22 is provided to the image processing device 1. For example, the final parameters of the adjusted learning model 632 are input to the image processing device 1 and stored in the memory unit 14. The risk detection model 143 is realized by the calculation unit 11 performing information processing using the stored parameters. Note that the image processing device 1 may also function as the learning device 6 that generates the nerve detection model 142 or the risk detection model 143. The image processing device 1 executes the processes of S101 to S114 using the generated nerve detection model 142 or the risk detection model 143.

[0104] As described above in detail, in this embodiment, the image processing device 1 outputs the risk of nerve damage caused by the surgical instrument 22 based on endoscopic images of the inside of the body of the patient 4. Information indicating the risk of nerve damage is displayed on the display unit 32, allowing the user 5 to perform endoscopic surgery while checking the risk of nerve damage. If the risk of nerve damage is high, the procedure can be stopped, for example, and endoscopic surgery can be performed while taking measures to prevent nerve damage.

[0105] In this embodiment, the risk of nerve damage can be determined based on endoscopic images, making it possible to detect a high risk of nerve damage before nerve damage actually occurs. This allows endoscopic surgery to be performed while effectively preventing nerve damage. Compared to using IONM, there is no need to attach an electrode-equipped tube to the nerve, reducing the burden on the surgeon. Furthermore, since there is no need to monitor nerve action potentials in this embodiment, muscle relaxants, which cannot be used with IONM, can be used. This allows the risk of nerve damage to be easily determined during endoscopic surgery. This allows endoscopic surgery to be performed while preventing nerve damage based on the easily determined risk, thereby improving the safety of endoscopic surgery. The ability to use muscle relaxants prevents sudden patient movements (bucking) during surgery, enabling safe surgery in terms of anesthesia management. Furthermore, the ability to use muscle relaxants improves the flexibility of endoscopic surgery.

[0106] When there is a high risk of nerve damage, stress may be caused to the nerve even if no actual nerve damage occurs. Adverse events such as numbness may occur due to nerve stress. When stress occurs in the recurrent laryngeal nerve, hoarseness or raspy voice may occur as an adverse event. Stress on the recurrent laryngeal nerve may also result in decreased vocal cord movement, which may lead to decreased swallowing function or aspiration pneumonia as an adverse event. In this embodiment, the risk of nerve damage is output, indicating that nerve stress may occur. By halting work when there is a high risk of nerve damage, it is possible to prevent adverse events caused by nerve stress from occurring.

[0107] In this embodiment, an example has been shown in which the risk detection model 143 outputs risk information including information on the risk of nerve hyperextension, hyperflexion, and thermal damage. Information on the risk of nerve hyperextension, hyperflexion, and thermal damage may be obtained using separate trained models. For example, a trained model that outputs the probability of nerve hyperextension occurring when an endoscopic image is input, a trained model that outputs the probability of nerve hyperflexion occurring when an endoscopic image is input, and a trained model that outputs the probability of nerve thermal damage occurring when an endoscopic image is input may be used separately. In addition to these trained models, a trained model that outputs the probability that a nerve is normal when an endoscopic image is input may also be used.

[0108] In this embodiment, the risk detection model 143 outputs risk information when an endoscopic image is input, but the risk detection model 143 may also be configured to output risk information when an endoscopic image and a nerve position are input. In this configuration, the risk detection model 143 is trained in advance to output risk information when an endoscopic image and a nerve position are input.

[0109] In this embodiment, the risk detection model 143 receives input of an endoscopic image of the inside of the body of the patient 4 and information indicating the position of the nerve in the endoscopic image. Specifically, the information indicating the position of the recurrent laryngeal nerve obtained in S104 is input. For example, the information indicating the position of the nerve is information indicating pixels that correspond to the nerve among the pixels included in the endoscopic image. An endoscopic image to which information indicating the position of the nerve has been added may be input to the risk detection model 143. The risk detection model 143 is generated by learning using training data in which the endoscopic image, the position of the nerve, and risk information are associated. Even in this embodiment, information indicating the risk of nerve damage can be output using the risk information output by the risk detection model 143.

[0110] <Embodiment 2> Embodiment 2 illustrates an embodiment in which the risk of nerve damage is determined using a method different from the method using the risk detection model 143. The configuration of the surgical system 100 according to embodiment 2 is the same as that of embodiment 1. FIG. 13 is a block diagram showing an example of the internal configuration of an image processing device 1 according to embodiment 2. As with embodiment 1, the image processing device 1 includes a nerve detection model 142. The image processing device 1 also includes an instrument detection model 144 that outputs a detection result of a surgical instrument included in an endoscopic image when the endoscopic image is input, and a depth estimation model 145 that outputs the depth positions of the recurrent laryngeal nerve and the surgical instrument included in the endoscopic image. The configuration of other parts of the image processing device 1 is the same as that of embodiment 1.

[0111] FIG. 14 is a conceptual diagram showing an example of the function of the instrument detection model 144. An endoscopic image of the inside of the body of a patient 4 is input to the instrument detection model 144. The instrument detection model 144 is a trained model that is trained in advance so as to output the position of a surgical instrument included in an endoscopic image when the endoscopic image is input. The output position of the surgical instrument is a position within the plane of the endoscopic image. The instrument detection model 144 is a trained model that performs segmentation. For example, the instrument detection model 144 is realized using U-net or YOLO. Alternatively, the instrument detection model 144 may be realized using a CNN or another trained model.

[0112] For example, the instrument detection model 144 outputs the position of a surgical instrument in the endoscopic image by outputting information indicating whether each pixel in the endoscopic image corresponds to a surgical instrument. The instrument detection model 144 may also output information indicating the range of pixels included in the endoscopic image that correspond to a surgical instrument. The instrument detection model 144 may also output an image in which the surgical instrument is extracted from the endoscopic image. In this way, the instrument detection model 144 outputs the detection result of the surgical instrument.

[0113] FIG. 15 is a conceptual diagram illustrating an example of the function of the depth estimation model 145. An endoscopic image capturing the inside of the body of a patient 4 is input to the depth estimation model 145. The depth estimation model 145 is a trained model that has been trained in advance so that, when an endoscopic image is input, it outputs the depth position of each part included in the endoscopic image. The depth position is a position in the depth direction perpendicular to the endoscopic image, and is the depth position of each object captured in the endoscopic image. For example, the depth position is the distance from the endoscope 21 to each object captured in the endoscopic image, or the relative depth position between objects captured in the endoscopic image.

[0114] For example, the depth estimation model 145 outputs the depth position of each part included in the endoscopic image by outputting information indicating the depth position of each pixel in the endoscopic image. The depth estimation model 145 outputs the depth position of each part of the recurrent laryngeal nerve and the surgical instrument included in the endoscopic image by outputting the depth position of the recurrent laryngeal nerve and the surgical instrument.

[0115] The instrument detection model 144 and the depth estimation model 145 are realized by the calculation unit 11 executing information processing in accordance with the computer program 141. The memory unit 14 stores data necessary to realize the instrument detection model 144 and the depth estimation model 145. The instrument detection model 144 is trained in advance using training data including data correlating an endoscopic image with the position of a surgical instrument in the endoscopic image. The depth estimation model 145 is trained in advance using training data including data correlating an endoscopic image with the position of the recurrent laryngeal nerve and a surgical instrument in the endoscopic image. For example, the instrument detection model 144 and the depth estimation model 145 are trained using a learning device 6.

[0116] 16 and 17 are flowcharts showing an example of a processing procedure according to the second embodiment, which is executed by the image processing device 1 when the endoscope 21 is used. The endoscope 21 and the surgical instrument 22 are inserted into the body of the patient 4, and the endoscope 21 generates an endoscopic image capturing an image of the inside of the body of the patient 4. As in the first embodiment, in the second embodiment, the endoscope 21 and the surgical instrument 22 are inserted into the neck or near the esophagus of the patient 4. The image processing device 1 receives an endoscopic image from the endoscope 21 and acquires the endoscopic image (S301). The image processing device 1 then acquires the position of the recurrent laryngeal nerve (S302). In S302, the calculation unit 11 acquires the position of the recurrent laryngeal nerve using the nerve detection model 142, as in the first embodiment. The acquired position of the recurrent laryngeal nerve is a position within the plane of the endoscopic image.

[0117] The image processing device 1 inputs the endoscopic image to the depth estimation model 145 (S303). In S303, the calculation unit 11 inputs the endoscopic image acquired in S301 to the depth estimation model 145. The depth estimation model 145 performs calculations in response to the input of the endoscopic image and outputs the depth position of each part in the endoscopic image. The image processing device 1 acquires the depth position of the recurrent laryngeal nerve (S304). In S304, the calculation unit 11 acquires the depth position of the recurrent laryngeal nerve in the endoscopic image output by the depth estimation model 145.

[0118] The image processing device 1 sets a plurality of points included in the recurrent laryngeal nerve 41 in the endoscopic image (S305). In S305, the calculation unit 11 associates the plurality of points with a plurality of portions of the recurrent laryngeal nerve by arranging the plurality of points within the recurrent laryngeal nerve along the longitudinal direction of the recurrent laryngeal nerve. The calculation unit 11 sets the position and depth position of each portion of the recurrent laryngeal nerve associated with each point in the endoscopic image as the position and depth position of each point, and stores the initial values ​​of the position and depth position of each point in the storage unit 14.

[0119] In S305, the calculation unit 11 may place multiple points on the center line of the recurrent laryngeal nerve, or may place multiple points along the contour of the recurrent laryngeal nerve. The calculation unit 11 may use both methods as appropriate. For example, the calculation unit 11 arranges multiple points at equal intervals within the recurrent laryngeal nerve. The intervals between the points may be predetermined. The number of points to be placed within the recurrent laryngeal nerve may be predetermined, and the calculation unit 11 may adjust the intervals between the points depending on the length of the recurrent laryngeal nerve and the number of points. The intervals between the multiple points do not have to be equal. Points may be placed at characteristic parts of the recurrent laryngeal nerve. The calculation unit 11 may set multiple points by using a trained model that detects key points based on an image. The positions of the points may be specified by instructions from the user 5.

[0120] 18 is a schematic diagram showing an example in which multiple points included in the recurrent laryngeal nerve 41 in an endoscopic image are set. The multiple points are indicated by multiple black circles included in the recurrent laryngeal nerve 41 in the endoscopic image. The image processing device 1 may or may not display an endoscopic image including multiple points as shown in FIG. 18 on the display unit 32.

[0121] The image processing device 1 receives an endoscopic image from the endoscope 21, acquires the endoscopic image (S306), and outputs the acquired endoscopic image (S307). In S307, the calculation unit 11 displays an endoscopic image such as that shown in FIG. 6 on the display unit 32. The image processing device 1 then acquires the position of the recurrent laryngeal nerve (S308). In S308, the calculation unit 11 acquires the position of the recurrent laryngeal nerve in the endoscopic image using the nerve detection model 142, as in the first embodiment. The acquired position of the recurrent laryngeal nerve is a position within the plane of the endoscopic image. Furthermore, as in the first embodiment, the image processing device 1 emphasizes the recurrent laryngeal nerve in the output endoscopic image.

[0122] The image processing device 1 inputs the endoscopic image to the instrument detection model 144 (S309). In S309, the calculation unit 11 inputs the endoscopic image acquired in S306 to the instrument detection model 144. The instrument detection model 144 performs calculations in response to the input of the endoscopic image and outputs the position of the surgical instrument 22 in the endoscopic image. The image processing device 1 acquires the position of the surgical instrument 22 (S310). In S310, the calculation unit 11 acquires the position of the surgical instrument 22 in the endoscopic image output by the instrument detection model 144.

[0123] The image processing device 1 then inputs the endoscopic image to the depth estimation model 145 (S311). In S311, the calculation unit 11 inputs the endoscopic image acquired in S306 to the depth estimation model 145. The depth estimation model 145 performs calculations in response to the input endoscopic image and outputs the depth position of each part in the endoscopic image. The image processing device 1 acquires the depth positions of the recurrent laryngeal nerve and the surgical instrument 22 (S312). In S312, the calculation unit 11 acquires the depth positions of the recurrent laryngeal nerve and the surgical instrument 22 in the endoscopic image output by the depth estimation model 145. The image processing device 1 may acquire the depth positions of the recurrent laryngeal nerve and the surgical instrument 22 without using the depth estimation model 145. For example, the endoscope 21 may be equipped with a depth camera, and the image processing device 1 may acquire the depth positions of the recurrent laryngeal nerve and the surgical instrument 22 based on the imaging results obtained by the depth camera.

[0124] The image processing device 1 then determines whether the surgical instrument 22 is close to the recurrent laryngeal nerve based on the positions and depth positions of the recurrent laryngeal nerve and the surgical instrument 22 (S313). In S313, the calculation unit 11 calculates the distance between the recurrent laryngeal nerve and the surgical instrument 22 based on the positions and depth positions of the recurrent laryngeal nerve and the surgical instrument 22 in the endoscopic image, and determines whether the surgical instrument 22 is close to the recurrent laryngeal nerve based on the calculated distance. For example, the calculation unit 11 calculates the shortest distance between the recurrent laryngeal nerve and the surgical instrument 22, and determines that the surgical instrument 22 is close to the recurrent laryngeal nerve if the calculated shortest distance is below a predetermined threshold. If the calculated shortest distance is not below the predetermined threshold, the calculation unit 11 determines that the surgical instrument 22 is not close to the recurrent laryngeal nerve. The calculation unit 11 may also calculate the distance between a specific portion of the surgical instrument 22 and the recurrent laryngeal nerve and make the determination using the calculated distance.

[0125] If the surgical instrument 22 is close to the recurrent laryngeal nerve (S313: YES), the image processing device 1 determines whether the surgical instrument 22 is in an active state in which it generates heat (S314). The surgical instrument 22 is controlled by the control device 31 to switch between an active state and a non-heat-generating state. The image processing device 1 displays a notification image on the display unit 32 superimposed on the endoscopic image to notify the user of the status of the surgical instrument 22, based on a signal from the control device 31. FIG. 19 is a schematic diagram showing an example of a notification image 42 superimposed on the endoscopic image. For example, the status of the surgical instrument 22 is notified by changing the color of the notification image 42 depending on the status of the surgical instrument 22. For example, the notification image 42 includes a message indicating the status of the surgical instrument 22. In S314, the calculation unit 11 determines the status of the surgical instrument 22 based on the notification image 42 superimposed on the endoscopic image. The calculation unit 11 may determine the status of the surgical instrument 22 by receiving direct notification of the status of the surgical instrument 22 from the control device 31.

[0126] If the surgical instrument 22 is in an active state (S314: YES), the image processing device 1 determines that there is a high risk of thermal damage to the recurrent laryngeal nerve (S315). In S315, the calculation unit 11 immediately determines that there is a high risk of thermal damage when the surgical instrument 22 is in an active state. The calculation unit 11 may also determine that there is a high risk of thermal damage when the surgical instrument 22 is close to the recurrent laryngeal nerve and the period during which the surgical instrument 22 is in an active state exceeds a predetermined period. The calculation unit 11 may also determine the level of risk of thermal damage in stages depending on the distance between the recurrent laryngeal nerve and the surgical instrument 22 and the period during which the surgical instrument 22 is in an active state.

[0127] The image processing device 1 may perform processing to determine that there is a high risk of recurrent laryngeal nerve damage when the surgical instrument 22 is close to the recurrent laryngeal nerve, regardless of whether the surgical instrument 22 is in an active state. By this processing, the image processing device 1 determines the risk of the surgical instrument 22 causing damage to the recurrent laryngeal nerve, such as cutting unrelated to heat.

[0128] The image processing device 1 outputs information indicating the risk of thermal damage to the recurrent laryngeal nerve (S316). In S316, the calculation unit 11 generates an image by adding information indicating the position of the recurrent laryngeal nerve and information indicating a high risk of thermal damage to the recurrent laryngeal nerve to the endoscopic image, and displays the generated image on the display unit 32. For example, the calculation unit 11 displays an image including an icon or a warning message indicating a high risk of thermal damage to the recurrent laryngeal nerve on the display unit 32. The calculation unit 11 may output information indicating the risk of thermal damage to the recurrent laryngeal nerve depending on the level of risk of thermal damage. For example, the calculation unit 11 may change the type of icon, the content of the warning message, the color of the recurrent laryngeal nerve 41 in the endoscopic image, or the color of the endoscopic image depending on the level of risk of thermal damage to the recurrent laryngeal nerve. As in the first embodiment, the calculation unit 11 may add a risk bar or a safety bar to the endoscopic image depending on the level of risk of thermal damage to the recurrent laryngeal nerve.

[0129] If the surgical instrument 22 is not close to the recurrent laryngeal nerve (S313: NO), if the surgical instrument 22 is not in an active state (S314: NO), or after S316 is completed, the image processing device 1 measures the time change in the distance between points included in the recurrent laryngeal nerve 41 in the endoscopic image or the time change in the angle formed by multiple lines connecting the points (S317). In S317, the calculation unit 11 determines the current positions and depth positions of multiple points based on the positions and depth positions of the recurrent laryngeal nerve and the surgical instrument 22 in the endoscopic image, and stores the positions and depth positions of the multiple points in the memory unit 14. The calculation unit 11 also measures the time change in the distance between the points or the time change in the angle formed by multiple lines connecting the points based on the stored positions and depth positions of the multiple points at multiple times. For example, the calculation unit 11 measures the time change in the distance between two adjacent points. For example, the calculation unit 11 measures the time change in the angle between two lines connecting each point and its neighboring points.

[0130] The image processing device 1 determines the risk of recurrent laryngeal nerve hyperextension or hyperflexion based on the change over time in the distance between the points or the change over time in the angle between the multiple straight lines (S318). In S318, for example, the calculation unit 11 determines that there is a high risk of recurrent laryngeal nerve hyperextension if the increase in the distance between any two adjacent points from the initial value or the increase from the previous time point exceeds a predetermined threshold. For example, the calculation unit 11 determines that there is a high risk of recurrent laryngeal nerve hyperflexion if the change in the angle between two straight lines connecting any point and both adjacent points exceeds a predetermined threshold.

[0131] The calculation unit 11 may determine that there is a high risk of hyperextension when the distance between two adjacent points exceeds a threshold value at more than a predetermined number of locations. The calculation unit 11 may determine that there is a high risk of hyperflexion when the change in the angle between two straight lines exceeds a threshold value at more than a predetermined number of locations. The calculation unit 11 may determine that there is a high risk of hyperextension or hyperflexion of the recurrent laryngeal nerve when the result of integrating the change in the distance between two points or the change in the angle between two straight lines over the entire length of the recurrent laryngeal nerve exceeds a predetermined threshold.

[0132] The calculation unit 11 may determine the risk of hyperextension or hyperflexion based on a time series of the distance between two points or a time series of the angle between two lines. For example, the calculation unit 11 may determine the risk of hyperextension or hyperflexion based on the integrated value of the time series of the distance or angle, or the integrated value of a graph showing the time series of the distance or angle. The calculation unit 11 may make the determination based on the time series of the distance between any two points or the time series of the angle between any two lines, or may make the determination based on the integrated value or integrated value of the time series of the distance or angle over the entire length of the recurrent laryngeal nerve. The calculation unit 11 may make the determination using a trained model that has been trained to output the risk of hyperextension or hyperflexion of the recurrent laryngeal nerve when a time series of the distance between two points or a time series of the angle between two lines is input.

[0133] The calculation unit 11 may determine the level of risk of hyperextension or hyperflexion of the recurrent laryngeal nerve in stages. For example, the calculation unit 11 may determine the level of risk of hyperextension or hyperflexion based on the increase in the distance between two points or the change in the angle between two lines, or based on the number of points where the distance between two points exceeds a threshold or the number of points where the change in the angle between two lines exceeds a threshold. For example, the calculation unit 11 may determine the level of risk of hyperextension or hyperflexion based on the integrated value of a time series of distance or angle, or the integrated value of a graph showing a time series of distance or angle.

[0134] The image processing device 1 outputs information indicating the risk of recurrent laryngeal nerve overextension or overflexion (S319). In S319, the calculation unit 11 outputs information indicating the risk of recurrent laryngeal nerve overextension or overflexion based on the reactive result of S318. For example, the calculation unit 11 displays an image including an icon or a warning message indicating a high risk of recurrent laryngeal nerve overextension or overflexion on the display unit 32. The calculation unit 11 may output information indicating the risk of recurrent laryngeal nerve overextension or overflexion based on the risk level of recurrent laryngeal nerve overextension or overflexion. For example, the calculation unit 11 may change the type of icon, the content of the warning message, the color of the recurrent laryngeal nerve 41 in the endoscopic image, or the color of the endoscopic image based on the risk level of recurrent laryngeal nerve overextension or overflexion. As in the first embodiment, the calculation unit 11 may add a risk bar or a safety bar to the endoscopic image based on the risk level of recurrent laryngeal nerve overextension or overflexion.

[0135] In S319, if there is even one location on the recurrent laryngeal nerve that is at high risk of hyperextension or hyperflexion, the calculation unit 11 may output information indicating that there is a high risk of hyperextension or hyperflexion of the recurrent laryngeal nerve. The calculation unit 11 may output information indicating the points at high risk of hyperextension or hyperflexion by displaying, among multiple points included in the recurrent laryngeal nerve, points at high risk of hyperextension or hyperflexion on the display unit 32 in the endoscopic image while superimposing the points on the recurrent laryngeal nerve 41. The user 5 can recognize the locations on the recurrent laryngeal nerve that are at high risk of hyperextension or hyperflexion.

[0136] If it is determined in S318 that there is a low risk of hyperextension or hyperflexion of the recurrent laryngeal nerve, the image processing device 1 may output information indicating that there is a low risk of hyperextension or hyperflexion of the recurrent laryngeal nerve. At this time, the calculation unit 11 displays information indicating that there is a low risk of hyperextension or hyperflexion on the display unit 32. If it is determined in S318 that there is a low risk of hyperextension or hyperflexion, the image processing device 1 may omit the processing of S319.

[0137] The image processing device 1 determines whether to end the processing (S320). In S320, if information indicating the end of the endoscopic surgery is input from the control device 31, the calculation unit 11 determines to end the processing of outputting the endoscopic image. If information indicating the end of the endoscopic surgery is not input, the calculation unit 11 determines not to end the processing of outputting the endoscopic image. If it is determined not to end the processing of outputting the endoscopic image (S320: NO), the image processing device 1 returns the processing to S306. The image processing device 1 repeats the processing of S306 to S320 in parallel with the execution of the endoscopic surgery. If it is determined to end the processing (S320: YES), the image processing device 1 ends the information processing.

[0138] As described above, in the second embodiment, the image processing device 1 determines the risk of nerve damage and outputs information indicating the risk of nerve damage using a method different from the method using the risk detection model 143. Compared to the method using the risk detection model 143, the image processing device 1 can determine the risk of thermal damage, hyperextension, or hyperflexion of the recurrent laryngeal nerve in more detail and notify the user 5 of the risk of thermal damage, hyperextension, or hyperflexion.

[0139] In the first and second embodiments, examples have been described in which endoscopic surgery is mainly performed on the neck or esophagus of a patient 4 using the surgical system 100, but it is also possible to perform endoscopic surgery on other parts of the patient 4 using the surgical system 100. In the first and second embodiments, a configuration has been shown in which the recurrent laryngeal nerve of the patient 4 is emphasized in an endoscopic image and the risk of damage is output. The image processing device 1 may also be configured to emphasize and output the risk of damage for nerves other than the recurrent laryngeal nerve when endoscopic surgery is performed on parts of the patient 4 other than the neck or esophagus. For example, the nerve for which emphasis and the risk of damage are output may be the facial nerve, vagus nerve, hypogastric nerve, obturator nerve, or pelvic nerve.

[0140] In the first and second embodiments, endoscopes such as thoracoscopes and laparoscopes that are inserted by drilling holes in the patient's body have been mainly described. However, endoscopes may also be inserted without drilling holes in the patient's body, such as those inserted into the digestive tract. In the first and second embodiments, examples of nerve hyperextension, hyperflexion, and thermal damage have been described as specific examples of nerve damage. However, the image processing device 1 may also be configured to output information indicating the risk of other types of damage. For example, the image processing device 1 may output information indicating the risk of nerve compression or crushing as the risk of nerve damage. For example, the risk detection model 143 may be configured to output the probability of nerve compression or nerve crushing as risk information. Furthermore, the surgical system 100 may be configured to include separate display units for displaying endoscopic images and outputting information indicating the risk of nerve damage. In the first and second embodiments, the surgical instrument 22 is operated by a drive mechanism such as a robot arm and operated using the operation unit 33. However, the surgical instrument 22 may be held and operated by the user 5, who is the surgeon.

[0141] The present invention is not limited to the contents of the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, embodiments obtained by combining technical means modified appropriately within the scope of the claims are also included in the technical scope of the present invention.

[0142] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, the claims use a format in which a claim references two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0143] REFERENCE SIGNS LIST 100 Surgery system 1 Image processing device 10 Recording medium 11 Calculation unit 14 Memory unit 141 Computer program 142 Nerve detection model (second trained model) 143 Risk detection model (first trained model) 21 Endoscope 22 Surgical instrument 31 Control device 32 Display unit 4 Patient 5 User 6 Learning device

Claims

1. A computer program that causes a computer to execute a process of acquiring endoscopic images of the inside of the body using an endoscope, and outputting information indicating the risk of damage to nerves inside the body based on the acquired endoscopic images.

2. The computer program according to claim 1, characterized in that it causes a computer to execute the following process: inputting an acquired endoscopic image into a first trained model that outputs risk information regarding the risk of nerve damage when an endoscopic image is input, acquiring risk information output by the first trained model, and outputting information indicating the risk of damage to the nerve related to the acquired endoscopic image according to the acquired risk information.

3. The computer program of claim 1, wherein the risk includes the risk of overstretching, overbending, or thermal damage to the nerve caused by a surgical instrument.

4. The computer program according to claim 1, characterized in that the computer is caused to execute a process of outputting the information indicating the risk in a form corresponding to the level of the risk.

5. The computer program according to claim 1, characterized in that the computer is caused to execute a process of inputting an acquired endoscopic image into a second trained model that detects the recurrent laryngeal nerve in the endoscopic image when the endoscopic image is input, obtaining a detection result of the recurrent laryngeal nerve output by the second trained model, and coloring the recurrent laryngeal nerve in the acquired endoscopic image according to the detection result and outputting the endoscopic image.

6. The computer program according to claim 5, characterized in that the computer is caused to execute a process of adjusting the display form of the recurrent laryngeal nerve in the endoscopic image according to the information indicating the risk.

7. The computer program according to claim 5, characterized in that the computer is caused to execute a process of individually switching between outputting or ceasing outputting the information indicating the risk and switching between coloring or ceasing coloring of the recurrent laryngeal nerve in an endoscopic image.

8. The computer program according to claim 1, characterized in that the computer is caused to execute a process of adjusting the operation of a surgical instrument in response to the information indicating the risk.

9. The computer program according to claim 1, characterized in that it causes a computer to execute the following processes: acquire the positions of the nerves and surgical instruments in the acquired endoscopic images; acquire the depth positions of the nerves and surgical instruments in the acquired endoscopic images; determine whether the surgical instruments are in a state in which they will generate heat; determine the risk of thermal damage to the nerves based on the positions and depth positions of the nerves and surgical instruments in the acquired endoscopic images and the state of the surgical instruments; and output information indicating the determined risk.

10. The computer program of claim 1, characterized in that it causes a computer to execute the following processes: acquire the position of a nerve in an acquired endoscopic image; acquire the depth position of the nerve in an acquired endoscopic image; set a plurality of points contained in the nerve in the acquired endoscopic image; measure the change in distance between points over time or the change in angle between a plurality of straight lines connecting the points based on the positions and depth positions of the plurality of points; determine a risk of hyperextension or hyperflexion of the nerve based on the change in distance over time or the change in angle over time; and output information indicating the determined risk.

11. The computer program according to claim 10, characterized in that the computer is caused to execute a process of outputting information indicating points among the plurality of points that are at high risk of developing hyperextension or hyperflexion.

12. A method for generating a trained model, comprising: acquiring training data including endoscopic images of the inside of a body captured by an endoscope and risk information regarding the risk of damage to nerves inside the body captured by the images; and generating a trained model that outputs risk information when an endoscopic image is input by learning using the training data.

13. The method for generating a trained model as described in claim 12, characterized in that it comprises acquiring endoscopic images, acquiring risk information using intraoperative neuromonitoring, and generating training data by associating the risk information with the endoscopic images acquired at the time the risk information was acquired.

14. An image processing method comprising: acquiring an endoscopic image of the inside of a body using an endoscope; and outputting information indicating the risk of damage to nerves inside the body based on the acquired endoscopic image.

15. An image processing device comprising a calculation unit, which acquires an endoscopic image of the inside of the body captured by an endoscope, and outputs information indicating the risk of damage to nerves inside the body based on the acquired endoscopic image.

Images