Surgery assistance system and surgery assistance method

Abstract

The embodiments disclosed in this specification and the accompanying drawings relate to a surgical assistance system and a surgical assistance method. Information related to abnormalities occurring during surgery is easily confirmed. The surgical assistance system of the embodiment includes an acquisition unit, a detection unit, and a generation unit. The acquisition unit acquires medical information of a subject undergoing surgery. The detection unit detects items related to the abnormality based on the acquired medical information of the subject. The generation unit generates association information that associates the time at which the item related to the abnormality is detected with the medical information acquired at that time.

Description

[0001] This application enjoys the priority benefit of Japanese patent application No. 2020-111615 filed on June 29, 2020, and the entire contents of that Japanese patent application are incorporated herein by reference. Technical Field

[0002] The embodiments disclosed in this specification and the accompanying drawings relate to a surgery support system and a surgery support method. Background Art

[0003] Various surgical support systems are known for use in surgery. For example, a surgical support system is known that generates virtual endoscopic images from preoperative CT (Computed Tomography) images to avoid damage to blood vessels and internal organs during laparoscopic surgery, and displays these images in conjunction with actual endoscopic images during surgery. Summary of the Invention

[0004] The problem to be solved by the present invention is to facilitate confirmation of information related to abnormalities occurring during surgery.

[0005] A surgical support system according to an embodiment includes an acquisition unit, a detection unit, and a generation unit. The acquisition unit acquires medical information of a subject undergoing surgery. The detection unit detects an abnormality-related event based on the acquired medical information of the subject. The generation unit generates association information that associates the time at which the abnormality-related event was detected with the medical information acquired at that time.

[0006] According to the surgery support system of the embodiment, information related to abnormalities occurring during surgery can be easily confirmed. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a diagram showing an example of the configuration of the surgery support system according to the first embodiment.

[0008] Figure 2 This is a diagram showing an example of display control performed by the control function of the first embodiment.

[0009] Figure 3 This is a diagram showing an example of display control performed by the control function of the first embodiment.

[0010] Figure 4 This is a diagram showing an example of display control performed by the control function of the first embodiment.

[0011] Figure 5 This is a flowchart for explaining the processing steps of the surgical assistance device according to the first embodiment. DETAILED DESCRIPTION

[0012] The following describes in detail embodiments of the surgical assistance system and method with reference to the accompanying drawings. The surgical assistance system and method of the present application are not limited to the embodiments described below. Furthermore, the embodiments can be combined with other embodiments or existing technologies to the extent that they do not conflict with the intended content.

[0013] (First embodiment)

[0014] Figure 1 1 is a diagram showing an example of the configuration of the surgery support system 10 according to the first embodiment. Figure 1 In the present invention, a surgical assistance system 10 including a surgical assistance device for performing surgical assistance is described, but the implementation is not limited thereto. The surgical assistance method described below can also be performed by any device in the surgical assistance system 10.

[0015] For example, Figure 1 As shown, the surgical support system 10 of this embodiment includes a medical image diagnostic device 1, an endoscope system 2, a virtual laparoscopic image system 3, a position sensor 4, and a surgical support device 5. Here, each device and system is connected to each other via a network so that communication is possible. In addition, in the first embodiment, the case of performing laparoscopic surgery as a surgical operation is described as an example, but the surgery is not limited to this and can also be applied to other surgical operations. In addition, the surgical support system 10 may also include systems other than those shown in the figure (for example, HIS (Hospital Information System)), devices (for example, image storage devices), etc.

[0016] Medical image diagnostic apparatus 1 captures images of a subject and collects medical images. It then transmits the collected medical images to virtual laparoscopic imaging system 3, surgical support apparatus 5, and the like. Examples of medical image diagnostic apparatus 1 include an X-ray diagnostic apparatus, an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, an ultrasound diagnostic apparatus, a SPECT (Single Photon Emission Computed Tomography) apparatus, and a PET (Positron Emission Computed Tomography) apparatus.

[0017] Medical image diagnostic apparatus 1 collects medical images of a subject undergoing surgery. Specifically, it collects medical images of the surgical site before and after surgery. It then transmits these images to a virtual laparoscopic imaging system 3, a surgical support system 5, and other devices.

[0018] The endoscope system 2 includes an endoscope 21, a display 22, a processing circuit 23, and a storage circuit 24. The endoscope 21 includes an insertion portion for insertion into a subject and an operating unit for operating the insertion portion. The insertion portion is a treatment unit for treating the surgical target area (affected area) within the subject, and an imaging unit for capturing images of the subject's interior. The operating unit receives operations from the operator on the treatment unit and the imaging unit.

[0019] The treatment unit is, for example, a forceps, an electrocautery, a suturing device, etc. In addition, the imaging unit includes an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a lens, and a light emitting unit, and uses the imaging element to capture an image of the affected part irradiated with light from the light emitting unit.

[0020] The display 22 displays the image (endoscopic image) captured by the imaging unit. The processing circuit 23 is connected to the endoscope 21, the display 22, and the storage circuit 24 to control the entire endoscope system. For example, the processing circuit 23 controls the operation of the treatment unit in the endoscope 21, the collection of endoscopic images by the imaging unit, the display of endoscopic images by the display 22, the storage of endoscopic images in the storage circuit 24, and the storage of endoscopic images. The storage circuit 24 stores endoscopic images 241 captured by the imaging unit of the endoscope 21.

[0021] For example, a doctor or other surgeon inserts an endoscope 21 having a treatment part such as forceps and an electric knife as an insertion part, and an endoscope 21 having an imaging part as an insertion part into the subject, and operates the treatment part while observing the endoscopic image collected by the imaging part and displayed on the monitor 22, thereby treating the surgical target part (affected part) in the subject.

[0022] The virtual laparoscopic image system 3 includes a display 31, a processing circuit 32, and a storage circuit 33. The display 31 displays images generated by the processing circuit 32. Specifically, the display 31 displays virtual laparoscopic images generated based on medical images collected by the medical image diagnostic apparatus 1.

[0023] The processing circuit 32 is connected to the display 31 and the storage circuit 33 and controls the entire virtual laparoscopic image system. Specifically, the processing circuit 32 controls the acquisition of medical images from the medical image diagnostic apparatus 1, the display of virtual laparoscopic images on the display 22, and the storage of virtual laparoscopic images in the storage circuit 33. Furthermore, the processing circuit 32 generates virtual laparoscopic images using medical images by executing a generation function 321. For example, the generation function 321 generates virtual laparoscopic images based on three-dimensional CT images collected from the patient's abdomen by the X-ray CT apparatus serving as the medical image diagnostic apparatus 1 before surgery.

[0024] For example, generation function 321 generates a virtualized laparoscopic image of the abdominal cavity projected from a predetermined viewing direction based on information about the abdominal cavity region contained in a two-dimensional CT image generated using a three-dimensional CT image. Storage circuit 33 stores virtualized laparoscopic image 331 generated by processing circuit 32.

[0025] The position sensor 4 comprises a sensor unit, a magnetic field generator, and a signal receiver. The sensor unit, for example, a magnetic sensor, is positioned at the distal end of the insertion portion of the endoscope 21, within the subject. The magnetic field generator is positioned near the subject and generates a magnetic field centered on the device and directed outward. The signal receiver receives the signal output by the sensor unit.

[0026] The sensor unit detects the three-dimensional magnetic field formed by the magnetic field generating unit. Then, based on the information of the detected three-dimensional magnetic field, the sensor unit calculates the position information (coordinates and angles) of the device in the space with the magnetic field generating unit as the origin, and sends the calculated position information of the device to the signal receiving unit. For example, the position information received from the sensor unit at the front end of the insertion portion installed in the endoscope 21 indicates the position of the front end of the insertion portion in the space with the magnetic field generating unit as the origin. In addition, the position information received from the sensor unit configured in the subject (for example, the internal organs of the affected part) indicates the position of the affected part in the space with the magnetic field generating unit as the origin. The signal receiving unit sends the position information received from the sensor unit to the virtualized laparoscopic image system 3 and the surgical assistance device 5.

[0027] Here, the virtual laparoscopic image system 3 can generate and display a virtual laparoscopic image linked to the endoscopic image by using the position information obtained by the position sensor 4. In this case, first, the three-dimensional coordinates in the space with the magnetic field generating unit as the origin are aligned with the three-dimensional coordinates in the three-dimensional medical image used to generate the virtual laparoscopic image.

[0028] For example, the generation function 321 extracts a position within the three-dimensional medical image (the position of the internal organ where the sensor unit is located) that corresponds to positional information acquired by a sensor unit located within the subject (e.g., an affected internal organ) and performs alignment to position the extracted position identical to the position acquired by the sensor unit. Here, the generation function 321 performs this alignment on positional information from multiple sensor units located within the subject, thereby aligning the three-dimensional coordinates of a space with the magnetic field generator as the origin with the three-dimensional coordinates of the three-dimensional medical image used to generate the virtualized laparoscopic image.

[0029] Furthermore, the alignment between the three-dimensional coordinates of the space with the magnetic field generating unit as the origin and the three-dimensional coordinates of the three-dimensional medical image used to generate the virtualized laparoscopic image is not limited to the above-described method, and other methods may be used. For example, positional information of a part within the subject depicted in an endoscopic image captured by the imaging unit of the endoscope 21 equipped with the sensor unit may be used.

[0030] In this case, for example, the generation function 321 calculates the three-dimensional coordinates of the part (characteristic part, etc.) depicted in the endoscopic image in the space with the magnetic field generating part as the origin based on the position information from the sensor part installed in the imaging part. Then, the generation function 321 extracts the position in the three-dimensional medical image corresponding to the part for which the three-dimensional coordinates are calculated, and performs alignment so that the extracted position and the position for which the three-dimensional coordinates are calculated are set to the same position. Here, the generation function 321 performs the above-mentioned alignment on multiple positions in the subject, thereby aligning the three-dimensional coordinates in the space with the magnetic field generating part as the origin with the three-dimensional coordinates in the three-dimensional medical image used in the generation of the virtualized laparoscopic image.

[0031] Thus, when performing alignment, the generation function 321 generates a virtualized laparoscopic image linked to the endoscopic image based on the positional information of the sensor unit attached to the imaging unit of the endoscope 21. For example, the generation function 321 uses the three-dimensional coordinates obtained by the sensor unit attached to the imaging unit of the endoscope 21 as the viewpoint and the imaging direction derived from the angle of the sensor unit as the projection direction to generate a virtualized laparoscopic image projected onto the interior of the three-dimensional CT image (intraperitoneal cavity).

[0032] Then, the generation function 321 sequentially generates virtual laparoscopic images with varying viewpoints and projection directions based on changes in the position of the imaging unit (changes in the three-dimensional coordinates and angles acquired by the sensor unit). By sequentially displaying the sequentially generated virtual laparoscopic images, a virtual laparoscopic image is displayed that is linked to changes in the endoscopic image.

[0033] Here, the generation function 321 uses positional information acquired by a sensor unit positioned within the subject (e.g., an internal organ in the affected area) to reflect changes in the shape of the internal organs during surgery in the virtual laparoscopic image. For example, each time the positional information acquired by the sensor unit positioned within the internal organ changes, the generation function 321 calculates the amount of change and changes the corresponding position in the three-dimensional medical image by the calculated amount of change. The generation function 321 then uses the changed medical image to generate a virtual laparoscopic image, thereby reflecting changes in the shape of the internal organs during surgery in the virtual laparoscopic image.

[0034] The surgical support device 5 generates information about abnormalities that occurred during surgery based on various information collected during surgery. Specifically, the surgical support device 5 acquires information from medical imaging devices and various medical devices during surgery and generates information about abnormalities that occurred during surgery based on this information. For example, the surgical support device 5 is implemented using a computer device such as a workstation, personal computer, or tablet computer.

[0035] For example, the surgery support device 5 includes an input interface 51, a display 52, a storage circuit 53, and a processing circuit 54. The surgery support device 5 is connected to the medical image diagnosis apparatus 1, the endoscope system 2, the virtual laparoscopic image system 3, and the position sensor 4 via a network.

[0036] The input interface 51 receives input operations of various instructions and various information from the user. Specifically, the input interface 51 is connected to the processing circuit 54, converts the input operation received from the user into an electrical signal and outputs it to the processing circuit 54. For example, the input interface 51 is implemented by a track ball, a switch, a button, a mouse, a keyboard, a touchpad that performs input operations by contacting the operating surface, a touchscreen that integrates the display screen and the touchpad, a non-contact input interface using an optical sensor, and a sound input interface. In addition, in this specification, the input interface 51 is not limited to physical operating components such as a mouse and a keyboard. For example, a processing circuit for an electrical signal that receives an electrical signal corresponding to an input operation from an external input device that is separate from the device and outputs the electrical signal to a control circuit is also included in the example of the input interface 51.

[0037] Display 154 displays various information and data. Specifically, display 154 is connected to processing circuit 155 and displays various information and data output from processing circuit 155. For example, display 154 can be implemented by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

[0038] The storage circuit 53 stores various data and programs. Specifically, the storage circuit 53 is connected to the processing circuit 54 and stores data input from the processing circuit 54, or reads stored data and outputs it to the processing circuit 54. For example, the storage circuit 53 can be implemented by a semiconductor storage element such as RAM (Random Access Memory) or flash memory, a hard disk, an optical disk, or the like.

[0039] For example, the storage circuit 53 stores a determination condition 531 and related information 532. The determination condition 531 and related information 532 will be described in detail later.

[0040] The processing circuit 54 controls the entire surgical support device 5. For example, the processing circuit 54 performs various processes based on input operations received from the user via the input interface 51. For example, the processing circuit 54 stores data sent from another device in the storage circuit 53. Furthermore, for example, the processing circuit 54 outputs data read from the storage circuit 53, thereby transmitting the data to another device. Furthermore, for example, the processing circuit 54 displays data read from the storage circuit 53 on the display 52.

[0041] Here, each processing circuit in the endoscope system 2, the virtualized laparoscopic image system 3, and the surgical assisting device 5 is implemented by, for example, a processor. In this case, each processing function is stored in a storage circuit in the form of a program that can be executed by a computer. Then, each processing circuit reads out each program stored in each storage circuit and executes it, thereby realizing the function corresponding to each program. In other words, each processing circuit has a function when each program is read out. Figure 1 The processing functions shown.

[0042] In addition, each processing circuit can also be constructed by combining multiple independent processors, and each processor executes a program to implement each processing function. In addition, each processing function possessed by each processing circuit can also be appropriately merged or dispersed to a single or multiple processing circuits for implementation. In addition, each processing function possessed by each processing circuit can also be implemented by a mixture of hardware (hardware) such as circuits and software (software). In addition, here, an example is described in which the program corresponding to each processing function is stored in a single storage circuit, but the embodiment is not limited to this. For example, it can also be configured so that the program corresponding to each processing function is dispersed and stored in multiple storage circuits, and the processing circuit reads each program from each storage circuit and executes it.

[0043] The above describes an example of the configuration of the surgery support system 10 of this embodiment. For example, the surgery support system 10 of this embodiment is deployed in an operating room of a medical institution such as a hospital or clinic to assist in identifying abnormalities occurring during surgical procedures performed by a user such as a doctor.

[0044] For example, during laparoscopic surgery, bleeding may occur without the surgeon's knowledge. Examples include instruments coming into contact with blood vessels, damage to underlying blood vessels during electrocautery, increased force from forceps pressing on internal organs causing lacerations in other areas where pressure is applied, and slight stretching of blood vessels during clamping with forceps causing lacerations in other areas where forceps is weakened.

[0045] When such bleeding occurs, the bleeding site is identified and hemostasis is performed while continuing the endoscopic procedure. However, the endoscopic field of view decreases during bleeding, making it difficult to identify the bleeding site. Furthermore, even if the endoscopic image is replayed and confirmed to investigate the bleeding site, it is difficult for the surgeon to determine when and where the procedure was performed, making identification based on the image difficult and time-consuming. Furthermore, while virtual endoscopic images can reveal the structure of internal organs outside the field of view, abnormalities such as bleeding events that occur during surgery are not reflected in the virtual endoscopic images, making identification based on virtual endoscopic images difficult.

[0046] In this way, if it takes time to identify the bleeding site and the bleeding cannot be stopped, the operation will be switched to laparotomy and continued, but the wound will become larger, the switching to laparotomy is more troublesome and the patient may be in a dangerous state, which is a heavy burden on the patient.

[0047] Therefore, the surgery support device 5 of the surgery support system 10 of this embodiment is configured to acquire various information during surgery and generate information in which the acquired information is associated with time information, thereby making it possible to easily confirm information related to abnormalities occurring during surgery.

[0048] Specifically, the surgical support system 5 continuously acquires information from the medical imaging diagnostic device 1, the endoscope system 2, and various other medical devices used during surgery. It analyzes this information and stores information related to items determined to be abnormal, corresponding to the time of acquisition. This allows the surgical support system 10 to display information related to abnormalities when an event (abnormality) such as bleeding occurs during surgery, making it easy to confirm information related to abnormalities that occurred during surgery. The following describes the surgical support system 5 with this configuration in detail.

[0049] For example, Figure 1 As shown, in this embodiment, the processing circuit 54 of the surgical support device 5 executes a control function 541, an analysis function 542, and a generation function 543. Here, the control function 541 is an example of an acquisition unit and a display control unit. Furthermore, the analysis function 542 is an example of a detection unit. Furthermore, the generation function 543 is an example of a generation unit.

[0050] The control function 541 acquires various data (medical information) from other devices connected via a network and stores the acquired medical information in the storage circuit 53. For example, the control function 541 acquires medical images collected by the medical image diagnostic apparatus 1, endoscopic images generated by the endoscope system 2, and virtualized laparoscopic images generated by the virtualized laparoscopic image system 3.

[0051] Here, the control function 541 can acquire medical information about the subject both before and during surgery. For example, the control function 541 can acquire medical images collected before and during surgery. Furthermore, the control function 541 can acquire endoscopic images during surgery. Furthermore, the control function 541 can acquire virtualized laparoscopic images generated before and during surgery.

[0052] Furthermore, the control function 541 can obtain various medical information obtained from the subject during surgery. For example, the control function 541 can obtain vital information obtained from the subject during surgery, position information obtained by the position sensor 4, or various information obtained by various sensors mounted on the endoscope 21. Examples of various sensors mounted on the endoscope 21 include MEMS (Micro Electro Mechanical Systems) sensors that obtain forceps pressure information. If the endoscope 21 is equipped with such a MEMS sensor, the control function 541 can also obtain pressure information obtained by the MEMS sensor.

[0053] Furthermore, the control function 541 causes the display 52 to display various medical information. For example, the control function 541 causes the display 52 to display information generated based on the analysis results of the acquired medical information. The control function 541 can also control the transmission of the generated information to other devices via the network and cause the displays of the other devices to display it. For example, the control function 541 transmits the generated information to the endoscope system 2 and the virtualized laparoscopic image system 3. The endoscope system 2 and the virtualized laparoscopic image system 3 cause their displays to display the information received from the surgical support device 5.

[0054] The analysis function 542 detects items related to abnormalities based on the medical information of the acquired object. Specifically, the analysis function 542 compares the medical information acquired by the control function 541 with the judgment condition 531 stored in the storage circuit 53, and detects the information in the acquired medical information that meets the judgment condition as items related to abnormalities. For example, the analysis function 542 detects items related to abnormalities during surgery based on medical images acquired from the medical image diagnostic device 1, endoscopic images acquired from the endoscope system 2, and vital information. As an example, the analysis function 542 can detect items related to abnormalities based on image feature values ​​in images of the object acquired during surgery (medical images, endoscopic images). In addition, the judgment condition 531 includes various conditions corresponding to the acquired medical information.

[0055] Analysis function 542 detects factors related to abnormalities, such as factors that may have triggered abnormalities. In this case, for example, judgment conditions corresponding to analysis using endoscopic images include "whether the treatment unit is in contact with a blood vessel," "the extent of electrocautery penetration into tissue," "the duration of forceps clamping a blood vessel," "the duration of forceps pressing on an internal organ," and "the degree of deformation of an internal organ."

[0056] For example, the determination condition 531, "Whether the treatment portion is in contact with a blood vessel," indicates whether the treatment portion of the endoscope 21 is in contact with a blood vessel, and further categorizes the degree of contact in stages. Alternatively, the degree of contact can be categorized based on, for example, the amount of movement of the treatment instrument. For example, the degree of contact increases with the amount of movement of the treatment instrument.

[0057] The analysis function 542 determines whether the treatment unit is in contact with a blood vessel by analyzing each endoscopic image sequentially acquired from the endoscope system 2. If the analysis function 542 determines that the treatment unit is in contact with a blood vessel, it calculates the amount of movement of the treatment unit since the previous endoscopic image in a time series and compares the calculated movement with the classification in the determination condition 531 to classify the degree of contact. Here, the analysis function 542 detects, for example, contact between the treatment unit and a blood vessel as a triggering event and determines the classified degree of contact as a risk level. For example, the analysis function 542 determines that a greater degree of contact indicates a higher risk level.

[0058] In addition, the "degree of invasion of the electrocautery into the tissue" in the judgment condition 531 represents a condition that classifies the degree of invasion of the electrocautery in stages. In addition, the degree of invasion can also be classified according to the amount of movement of the electrocautery, for example. When giving an example, the classification is made in such a way that the greater the amount of movement of the electrocautery, the greater the degree of invasion. The analysis function 542 calculates the amount of movement of the electrocautery from the previous endoscopic image in a time series by analyzing the images of each endoscopic image obtained in sequence from the endoscope system 2, and compares the calculated amount of movement with the classification in the judgment condition 531, thereby classifying the degree of invasion. Here, the analysis function 542 detects the situation where the degree of invasion exceeds the specified degree as an inducing event, and determines the classified degree of invasion as a risk level. For example, the analysis function 542 determines that the greater the degree of invasion, the higher the risk level.

[0059] Furthermore, the "duration of clamping a blood vessel with forceps" and "duration of pressing an internal organ with forceps" in judgment condition 531 represent conditions that categorize time lengths into stages. Analysis function 542 calculates "duration of clamping a blood vessel with forceps" and "duration of pressing an internal organ with forceps" by analyzing each endoscopic image sequentially acquired from the endoscope system 2. The calculated times are then compared with the categories in judgment condition 531 to categorize the calculated times. Here, analysis function 542 detects a calculated time exceeding a predetermined time as a triggering event and determines the time length as a risk level. For example, analysis function 542 determines that the longer the calculated time, the higher the risk level.

[0060] Furthermore, the "degree of deformation of the internal organs" in the judgment condition 531 represents a condition that classifies the degree of deformation of the internal organs in stages. Furthermore, the degree of deformation of the internal organs can also be classified, for example, based on the amount of change in the shape of the internal organs. For example, the classification is performed such that the greater the amount of change in the shape of the internal organs, the greater the degree of deformation. The analysis function 542 calculates the amount of change in the shape of the internal organs from the previous endoscopic image in a time series by analyzing the endoscopic images sequentially acquired from the endoscope system 2, and compares the calculated amount of change with the classification in the judgment condition 531, thereby classifying the degree of deformation. Here, the analysis function 542 detects a change exceeding a specified amount as a triggering event and determines the classified degree of deformation as a risk level. For example, the analysis function 542 determines that the greater the degree of deformation, the higher the risk level.

[0061] Furthermore, for example, the internal organs are deformed by being pressed or stretched by forceps. Furthermore, for example, the insertion portion of the endoscope 21 inserted into the body presses the internal organs, thereby causing deformation of the internal organs.

[0062] In addition, the above-mentioned image analysis can also be performed by feature detection based on AI (Artificial Intelligence). In addition, in the above example, the case of calculating the movement amount of the treatment part and the change amount of the internal organs by image analysis is described. However, the embodiment is not limited to this. For example, when the position sensor 4 is used, the position information obtained by the position sensor 4 can also be used. When giving an example, the analysis function 542 calculates the movement amount of the treatment part based on the position information obtained by the sensor part installed at the front end of the treatment part. In addition, for example, the analysis function 542 calculates the deformation amount of the internal organs based on the position information obtained by the sensor part of the internal organs arranged at the affected part.

[0063] Furthermore, analysis function 542 can detect various other abnormality-related issues. For example, a determination condition corresponding to analysis using medical images might include "presence of blood leakage detected by Color Doppler Imaging." For example, determination condition 531, "presence of blood leakage detected by Color Doppler Imaging," indicates whether blood leakage has occurred and further categorizes the degree of blood leakage into stages.

[0064] For example, the analysis function 542 determines the presence or absence of blood leakage by analyzing color Doppler images sequentially acquired from an ultrasonic diagnostic apparatus serving as the medical image diagnostic apparatus 1. The analysis function 542 then detects the presence of blood leakage as an abnormality-related event and determines the degree of blood leakage as a risk level based on the determination condition 531. For example, the analysis function 542 determines that the greater the degree of blood leakage, the higher the risk level.

[0065] For example, judgment conditions corresponding to analysis using vital information include "decreased blood pressure," "abnormal electrocardiogram," and "ischemia." For example, "decreased blood pressure" in judgment condition 531 indicates whether blood pressure has fallen below a predetermined value, and further categorizes the degree of blood pressure reduction into stages.

[0066] For example, during surgery, control function 541 obtains a subject's blood pressure information from a blood pressure monitor. Analysis function 542 determines whether the blood pressure has fallen below a specified value based on the blood pressure information sequentially obtained from the blood pressure monitor. Analysis function 542 then detects the blood pressure falling below the specified value as an abnormality and, based on determination condition 531, determines the degree of blood pressure reduction as a risk level. For example, analysis function 542 determines that a greater decrease in blood pressure corresponds to a higher risk level.

[0067] For example, "electrocardiogram abnormality" in the determination condition 531 indicates whether the rhythm or waveform in the electrocardiogram has changed by more than a predetermined amount, and the degree of change is further classified into stages.

[0068] For example, during surgery, the control function 541 obtains an electrocardiogram (ECG) of the subject from an electrocardiograph. The analysis function 542 determines, based on the ECGs sequentially obtained from the electrocardiograph, whether the rhythm or waveform in the ECG has changed by more than a specified amount. The analysis function 542 then detects changes in the rhythm or waveform in the ECG by more than a specified amount as an abnormality and determines the degree of change as a risk level based on the determination condition 531. For example, the analysis function 542 determines that a greater amount of change indicates a higher risk level.

[0069] For example, "ischemia" in determination condition 531 indicates whether ischemia has occurred and further categorizes the degree of ischemia into stages. Analysis function 542 determines whether ischemia has occurred based on the electrocardiogram waveforms sequentially acquired from the electrocardiograph. Analysis function 542 then detects the occurrence of ischemia as an abnormality-related event and determines the degree of ischemia as a risk level based on determination condition 531. For example, analysis function 542 determines that the greater the degree of ischemia, the higher the risk level.

[0070] As described above, analysis function 542 detects abnormalities based on medical information acquired during surgery and determines the risk level of the detected abnormalities. The above determination conditions are merely examples; abnormalities may be detected based on other information. For example, if a MEMS sensor is installed in the treatment area and pressure information is acquired by the MEMS sensor, abnormalities may be detected based on the acquired pressure information.

[0071] In this case, conditions related to pressure information are stored as determination conditions 531. For example, "pressure" in determination condition 531 indicates whether the acquired pressure value exceeds a predetermined value, and further classifies the degree of pressure into stages.

[0072] For example, control function 541 obtains pressure information from a MEMS sensor. Analysis function 542 determines whether the pressure exceeds a specified value based on the pressure information obtained by control function 541. Analysis function 542 then detects the determination that the pressure exceeds the specified value as an abnormality-related event and determines the pressure level as a risk level based on determination condition 531. For example, analysis function 542 determines that the greater the pressure value, the higher the risk level.

[0073] Above, an example of analysis performed by the analysis function 542 is described. In addition, in the above example, a case where an item related to an abnormality is detected based on one condition is described, but the embodiment is not limited to this, and a plurality of conditions may be combined to detect items related to an abnormality. For example, it may be a case where a plurality of conditions (for example, whether the treatment part is in contact with a blood vessel and the degree of deformation of an internal organ, etc.) included in an analysis based on a single device (for example, an analysis using an endoscopic image) are combined to detect items related to an abnormality, or it may be a case where conditions based on analyses of multiple devices (for example, an analysis using an endoscopic image and an analysis using vital information) are combined to detect items related to an abnormality. In addition, the various conditions of the determination condition 531 for reference by the analysis function 542 can be set arbitrarily. For example, it can be appropriately set according to the type of surgery, the type of internal organ of the affected part, etc.

[0074] The generation function 543 generates association information that associates the time at which an abnormality-related event was detected with the medical information acquired at that time. Specifically, the generation function 543 generates association information that associates the medical information obtained when the abnormality-related event was detected by the analysis function 542 with the time at which the medical information was acquired. For example, when an abnormality-related event (inducing event) is detected in an analysis based on the "degree of deformation of an internal organ," the generation function 543 generates association information that associates the endoscopic image at which the inducing event was detected with the time at which the endoscopic image was captured. Furthermore, when the control function 541 acquires an endoscopic image, the time at which the endoscopic image was captured is also acquired.

[0075] Here, the generation function 543 can generate association information that associates only the medical information at which an abnormality-related event was detected with the time, or it can also generate association information that further associates other medical information with the time at which the medical information at which the abnormality-related event was detected was acquired. For example, if vital information is acquired along with an endoscopic image, and an abnormality-related event is detected during analysis of the "degree of deformation of an internal organ" based on the endoscopic image, the generation function 543 can generate association information that further associates the vital information acquired at the same time. In other words, the generation function 543 generates association information that further associates the vital information with the time at which the endoscopic image at which the abnormality-related event was detected was acquired.

[0076] Furthermore, the generation function 543 can associate the risk level with associated information that links the time at which the medical information related to the abnormality was acquired and the medical information acquired at that time. For example, the generation function 543 can further associate the risk level determined based on the analysis of the "degree of deformation of an internal organ" with the associated information.

[0077] Furthermore, the generation function 543 can also generate association information that associates position information. Specifically, the generation function 543 generates association information that associates the time at which an abnormality-related event was detected, the medical information obtained at that time, and the position information obtained at that time. In this case, the sensor portion of the position sensor 4 is first attached to the distal end of the insertion portion of the endoscope 21 to obtain position information of the distal end of the insertion portion during surgery. The control function 541 obtains the position information obtained by the position sensor 4 and stores the obtained position information in the storage circuit 53, associating it with the time at which it was obtained.

[0078] Generating function 543 obtains from storage circuit 53 the positional information of the time when analysis function 542 acquired the medical information at which the abnormality-related event was detected, and generates association information associating the time at which the medical information at which the abnormality-related event was detected was acquired with the medical information acquired at that time. For example, if an abnormality-related event is detected during analysis based on the degree of deformation of an internal organ, generating function 543 generates association information associating the endoscopic image at which the abnormality-related event was detected, the time at which the endoscopic image was captured, and the positional information at that time.

[0079] Furthermore, the information associated with the related information can be set as appropriate. For example, the generation function 543 associates time, medical information, analysis results, and position information as appropriate as related information based on information acquired during surgery.

[0080] The generation function 543 generates the aforementioned related information each time the analysis function 542 detects an abnormality-related event during surgery and stores it in the storage circuit 53. The related information 532 in the storage circuit 53 is generated and stored by the generation function 543 as described above. Alternatively, the related information 532 may be stored for each surgery and read out and utilized after the surgery.

[0081] When the related information is generated as described above, the control function 541 displays various information using the related information on the display 52. ​​The control function 541 can also display various information using the related information on the display 22 of the endoscope system 2 and the display 31 of the virtual laparoscopic image system 3.

[0082] Next, examples of information displayed by the control function 541 will be described.

[0083] For example, the control function 541 displays timeline information showing related information in chronological order. Figure 2 : is a diagram showing an example of display control performed by the control function 541 of the first embodiment. Figure 2 This shows the display of timeline information relative to the endoscopic image displayed by the endoscope system 2 .

[0084] For example, Figure 2 As shown in the upper diagram of FIG, in the endoscope system 2, based on the endoscopic images obtained during surgery, the display 22 displays real-time images of the abdominal cavity. During this time, the surgical support device 5 obtains various medical information to determine whether an abnormality has occurred.

[0085] Here, the analysis function 542 detects matters associated with the abnormality, and when the generation function 543 generates associated information, such as Figure 2 As shown in the middle diagram of FIG, the control function 541 performs control based on the generated association information so that the timeline information 101 is displayed in the endoscopic image.

[0086] Here, the control function 541 is to display information that can identify at least one of the degree of the detected matter associated with the anomaly and the detection method, as shown in position 1 of the timeline information 101. For example, the control function 541 displays position 1 of the timeline information in a color corresponding to the risk level of the detected matter associated with the anomaly. In addition, for example, the control function 541 displays position 1 of the timeline information in a color corresponding to the detection method (for example, the conditions used in the analysis) for detecting the matter associated with the anomaly. Here, in the case where the risk level and the detection method can be identified, the control function 541, for example, assigns the color of the outer frame and the color of the inner side of position 1 on the timeline information 101 to the detection method and the risk level. When giving an example, the control function 541 uses the color corresponding to the detection method as the color of the outer frame representing position 1, and uses the color corresponding to the risk level as the color of the frame representing position 1.

[0087] Furthermore, the control function 541 can change the display based on changes in the risk level. As described above, the analysis function 542 detects items related to abnormalities based on the medical information sequentially acquired by the control function 541. Therefore, if items related to abnormalities are continuously detected in the sequentially acquired medical information, associated information is continuously generated, and the control function 541 continuously displays the identifiable information relative to the timeline information.

[0088] Here, for example, if the degree of deformation of an internal organ gradually changes, the risk level determined by the analysis function 542 gradually changes. In this case, the risk level associated with the associated information generated by the generation function 543 changes. As a result, for example, Figure 2 As shown in the lower section of , the control function 541 changes the color of position 1 in the timeline information 101 and displays it.

[0089] For example, the control function 541 is, before the associated information is generated, Figure 2 As shown in the upper section of the figure, the timeline information 101 is not displayed, and when the related information is generated, it is automatically displayed. Figure 2 The middle and lower sections of the timeline information 101 are shown. Then, the control function 541 can also control the timeline information not to be displayed when the related information is no longer generated.

[0090] Furthermore, for example, the control function 541 can display a medical image during surgery collected at the time when an abnormality-related event is detected in association with the corresponding time in the timeline information. Figure 3 : is a diagram showing an example of display control performed by the control function 541 of the first embodiment. Figure 3 This shows the display of timeline information relative to the endoscopic images displayed by the endoscope system 2 .

[0091] For example, the control function 541 is based on the operator's operation, such as Figure 3 As shown in FIG. 1 , the timeline information 101 is displayed or the thumbnail 102 is displayed. For example, when the operator performs the display operation of the timeline information 101 during surgery, the control function 541 is as follows: Figure 3 As shown in the middle section of , timeline information 101 based on the related information generated so far is displayed.

[0092] Furthermore, when the operator designates position 6 in the timeline information 101, the control function 541 is as follows: Figure 3 As shown in the lower section of FIG, a thumbnail 102 of the endoscopic image corresponding to the designated position 6 is associated with the position 6 and displayed.

[0093] For example, when an abnormality such as bleeding occurs during surgery, when the operator displays the timeline information, the control function 541 is as follows: Figure 3 As shown in the middle section of , timeline information 101 that can identify the time when an event related to an abnormality was detected is displayed. This allows the operator to easily identify the time when an event that may have caused the abnormality occurred.

[0094] Then, when the operator designates a position in the timeline information 101, the control function 541 is as follows: Figure 3 As shown in the lower section of , a thumbnail 102 of an image acquired at a designated time is displayed. This allows the operator to grasp the treatment status when an abnormality-related event occurs, and to quickly identify the cause of an abnormality such as bleeding.

[0095] Furthermore, for example, the control function 541 displays the position of the medical device corresponding to the moment the abnormality-related event was detected in the medical image of the subject. Specifically, when the associated information includes position information, the control function 541 displays the medical image showing the position information obtained at the moment the abnormality-related event was detected, based on the position information associated with the associated information.

[0096] Figure 4 : is a diagram showing an example of display control performed by the control function 541 of the first embodiment. Figure 4The left side of the figure shows an endoscopic image showing the timeline information 101. Figure 4 The figure on the right side of the middle diagram shows a medical image showing position information.

[0097] For example, Figure 4 As shown, control function 541 displays timeline information 101 on the endoscopic image and displays a medical image showing the position information of the medical device at the time when the associated information was generated. Here, control function 541 displays the medical image showing the position information using a three-dimensional medical image aligned with three-dimensional coordinates in a space with the magnetic field generating portion of position sensor 4 as the origin.

[0098] For example, Figure 4 As shown in the figure on the right, control function 541 displays position information using a virtualized laparoscopic image or CT image based on a three-dimensional medical image aligned with three-dimensional coordinates in space with the magnetic field generator as the origin. In one example, control function 541 uses the position information corresponding to positions 1 through 6 in timeline information 101 and the alignment information to determine the position in the three-dimensional medical image corresponding to each position.

[0099] Then, the control function 541 displays the medical image showing the identification information at each determined position. Figure 4 As shown, control function 541 displays numbers corresponding to positions 1 to 6 in timeline information 101 at specific locations on the CT image. Here, control function 541 can display each number so that the detected risk level and detection method can be identified, similar to timeline information 101. For example, control function 541 can display the risk level and detection method in a identifiable manner using the shape of the frame surrounding the number or the color of the frame's interior.

[0100] Furthermore, the control function 541 is capable of displaying not only the position information of the medical device at the moment when the medical information of the matter associated with the abnormality is detected, but also the current position of the medical instrument (the front end of the endoscope 21) in the medical image of the subject. That is, the control function 541 determines the position in the three-dimensional medical image corresponding to the current position of the medical instrument obtained from the position sensor 4. Then, the control function 541 displays the medical image showing the identification information at the determined position. For example, Figure 4 As shown in the figure on the right, the control function 541 displays the current position 104 of the medical instrument.

[0101] In addition, further, the control function 541 can be based on the operator's operation, such as Figure 4As shown in FIG. 1 , the thumbnail 103 is displayed. For example, when the operator performs the display operation of the timeline information 101 and the designation operation of the position 6 in the timeline information 101 during the operation, as shown in FIG. Figure 4 As shown, the control function 541 reflects the information of the designated position 6 in the virtual laparoscopic image, and displays the thumbnail 103 of the virtual laparoscopic image corresponding to the time of the position 6 in association with the number.

[0102] For example, if an abnormality such as bleeding occurs during surgery, the control function 541 displays an image showing the position of the medical instrument at the time medical information related to the abnormality was acquired. This allows the operator to easily identify the location where the abnormality occurred.

[0103] Then, when the operator designates a position in the timeline information 101, as shown in FIG. Figure 3 As shown in the lower section of FIG, the control function 541 displays the thumbnail 103 of the image at the designated time. This allows the operator to understand the treatment status when an event related to an abnormality occurs and to quickly identify the cause of an abnormality such as bleeding.

[0104] Next, use Figure 5 The processing of the surgery support device 5 according to the first embodiment will be described. Figure 5 This is a flowchart for explaining the processing steps of the surgical assisting device 5 according to the first embodiment. Figure 5 , an example is shown in which time, medical information, and position information are associated as related information.

[0105] Here, Figure 5 Steps S101 to S102 and S106 to S109 are steps implemented by the processing circuit 54 reading out the program corresponding to the control function 541 from the storage circuit 53 and executing it. Figure 5 Step S103 in the above is a step implemented by the processing circuit 54 reading out the program corresponding to the analysis function 542 from the storage circuit 53 and executing the program. Figure 5 Steps S104 to S105 in FIG. 5 are implemented by the processing circuit 54 reading out a program corresponding to the generation function 543 from the storage circuit 53 and executing the program.

[0106] like Figure 5As shown, in the surgical support device 5, the processing circuit 54 first determines whether the surgery has begun (step S101). For example, the processing circuit 54 determines the start of the surgery based on whether it has received an operation to start the surgery. Here, when the surgery has begun (step S101, yes), the processing circuit 54 acquires medical information (step S102). Prior to the start of the surgery, the surgical support device 5 is in a standby state (step S101, no).

[0107] Next, the processing circuit 54 analyzes the medical information (step S103) and determines whether any items associated with the abnormality exist (step S104). If any items associated with the abnormality exist (step S104, yes), the processing circuit 54 generates and stores association information that associates the medical information, time, and location information at which the item associated with the abnormality was detected (step S105). On the other hand, if no items associated with the abnormality are detected in step S104 (step S104, no), the processing circuit 54 proceeds to step S106.

[0108] Then, in step S106, the processing circuit 54 displays medical information. For example, the processing circuit 54 displays an endoscopic image, etc. Here, if associated information is generated, the processing circuit 54 displays the endoscopic image, etc., including timeline information based on the associated information. The processing circuit 54 then determines whether a designated operation has been received (step S107).

[0109] Here, if a designated operation is received (step S107, yes), the processing circuit 54 displays detailed information (e.g., thumbnails, etc.) of the designated medical information (step S108) and determines whether the surgery is complete (step S109). On the other hand, if a designated operation is not received (step S107, no), the processing circuit 54 determines whether the surgery is complete (step S109).

[0110] In step S109, if the operation is not completed (step S109, No), the processing circuit 54 returns to step S102 and continues to acquire medical information. On the other hand, if the operation is completed (step S109, Yes), the processing circuit 54 ends the process.

[0111] As described above, according to the first embodiment, the control function 541 acquires medical information about a subject undergoing surgery. The analysis function 542 detects abnormality-related events based on the acquired medical information. The generation function 543 generates association information that associates the time at which the abnormality-related event was detected with the medical information acquired at that time. Thus, the surgical support device 5 of the first embodiment can provide information related to the time at which the abnormality-related event occurred, making it easy to confirm information related to abnormalities that occurred during surgery.

[0112] For example, even if bleeding occurs during endoscopic surgery, the bleeding site can be quickly identified by prompting the moment when an abnormality (bleeding) is detected. As a result, the endoscopic surgery can be continued without switching to a laparotomy, which can help improve the patient's QOL (Quality of Life) after the operation.

[0113] Furthermore, according to the first embodiment, the analysis function 542 detects a precipitating event that may have caused an abnormality based on the subject's medical information. The generation function 543 generates association information that associates the time at which the precipitating event was detected with the medical information acquired at that time. Thus, the surgical support device 5 of the first embodiment can easily identify precipitating events that occurred during surgery.

[0114] Furthermore, according to the first embodiment, the control function 541 displays timeline information showing related information in chronological order. Therefore, the surgical support device 5 according to the first embodiment can provide easily viewable information on the time when an event related to an abnormality is detected.

[0115] Furthermore, according to the first embodiment, the control function 541 displays medical images collected during surgery at the time an abnormality-related event is detected, in association with the corresponding time in the timeline information. Therefore, the surgical support device 5 of the first embodiment can display images at the time an abnormality-related event is detected, rather than real-time images. This allows, for example, more rapid identification of a bleeding site.

[0116] Furthermore, according to the first embodiment, the control function 541 also acquires positional information indicating the location of medical instruments used during surgery. The generation function 543 generates association information that associates the time at which an abnormality was detected, the medical information acquired at that time, and the positional information acquired at that time. Thus, the surgical support device 5 of the first embodiment can also display positional information, enabling more rapid identification of bleeding sites, for example.

[0117] Furthermore, according to the first embodiment, the control function 541 acquires position information indicating the position of a medical instrument operated within a subject during surgery. Therefore, the surgery support device 5 according to the first embodiment can acquire position information of a medical instrument within a subject.

[0118] Furthermore, according to the first embodiment, the control function 541 also acquires a medical image of the subject. The control function 541 displays the position of the medical instrument corresponding to the time at which an abnormality-related event was detected in the medical image of the subject. Thus, the surgical support device 5 of the first embodiment can display positional information that is easier to see.

[0119] Furthermore, according to the first embodiment, the control function 541 displays the current position of the medical instrument in the medical image of the subject. Thus, the surgical support device 5 of the first embodiment can easily grasp the positional relationship between the current position of the medical instrument and the position at the time when medical information related to the abnormality was detected was acquired.

[0120] Furthermore, according to the first embodiment, the control function 541 displays information that identifies at least one of the degree of the detected abnormality-related event and the detection method. Therefore, the surgical support device 5 of the first embodiment allows the user to clearly understand the degree of the abnormality-related event and the detection method.

[0121] Furthermore, according to the first embodiment, the analysis function 542 detects abnormalities based on image features of an image of the subject acquired during surgery. Therefore, the surgery support device 5 according to the first embodiment can easily detect abnormalities.

[0122] (Other embodiments)

[0123] Furthermore, although the first embodiment has been described so far, the present invention can be implemented in various different forms other than the first embodiment.

[0124] In the above embodiment, the surgical support system 10 includes the surgical support device 5, and the surgical support device 5 performs various processes. However, the embodiment is not limited to this. The various processes of the surgical support method involved in this application can be performed individually by any device in the surgical support system 10, or can be performed in a distributed manner by multiple devices in the surgical support system 10.

[0125] For example, the analysis and processing of each medical information item can be performed by the device that acquires the medical information item. For example, the endoscope system 2 can analyze the endoscopic image. In this case, the processing circuit 23 executes the analysis function 542 to detect abnormalities based on the endoscopic image. Alternatively, an ultrasonic diagnostic device can detect abnormalities based on the ultrasonic image, or an electrocardiograph or other device can detect abnormalities based on vital information.

[0126] Furthermore, the related information generation process and various display processes related to the related information may also be executed in the virtualized laparoscopic image system 3 .

[0127] Furthermore, in the above-mentioned embodiment, the case where the surgery support method of the present application is applied to laparoscopic surgery has been described. However, the embodiment is not limited thereto, and the surgery support method of the present application can also be applied to various other surgeries.

[0128] Furthermore, in the above embodiments, examples are described in which the acquisition unit, detection unit, generation unit, and display control unit in this specification are implemented by the control function, analysis function, generation function, and control function of the processing circuit, respectively. However, the embodiments are not limited to this. For example, in addition to being implemented by the control function, analysis function, generation function, and control function described in the embodiments, the acquisition unit, detection unit, generation unit, and display control unit in this specification may also implement the same functions solely through hardware, solely through software, or a combination of hardware and software.

[0129] In addition, the term "processor" used in the description of the above embodiment means, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, it is also possible to configure the program to be directly incorporated into the circuit of the processor instead of storing it in the storage circuit. In this case, the processor realizes the function by reading the program incorporated into the circuit and executing it. In addition, each processor of the present embodiment is not limited to being configured as a single circuit for each processor, and a plurality of independent circuits can be combined to form a processor and realize its function.

[0130] Here, the program executed by the processor is pre-assembled into a ROM (Read Only Memory), a storage circuit, etc. and provided. In addition, the program can also be provided in the form of a file that can be installed on these devices or in the form of an executable file recorded in a non-temporary storage medium that can be read by a computer, such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), DVD (Digital Versatile Disk). In addition, the program can also be stored on a computer connected to a network such as the Internet and provided or distributed by downloading via the network. For example, the program is composed of modules including the above-mentioned processing functions. As actual hardware, the CPU reads the program from a storage medium such as a ROM and executes it, whereby each module is loaded onto the main storage device and generated on the main storage device.

[0131] Furthermore, in the above-described embodiments and variations, the components of the illustrated devices are conceptual and functional elements and do not necessarily need to be physically configured as shown. Specifically, the specific manner in which the devices are distributed or integrated is not limited to that illustrated; all or part of them can be functionally or physically distributed or integrated in arbitrary units, depending on various loads, usage conditions, and the like. Furthermore, all or any part of the processing functions performed in each device can be implemented by a CPU and programs analyzed and executed by the CPU, or as hardware using wired logic.

[0132] Furthermore, in each of the processes described in the above embodiments and modifications, all or part of the processes described as being automatically performed can also be performed manually, or all or part of the processes described as being manually performed can also be performed automatically using a known method. Furthermore, except where otherwise noted, the processing steps, control steps, specific names, and information including various data and parameters shown in the above text and drawings can be easily modified.

[0133] According to at least one of the embodiments described above, information related to abnormalities occurring during surgery can be easily confirmed.

[0134] Several embodiments have been described, but these embodiments are provided as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope and spirit of the invention and are included in the invention described in the patent claims and their equivalents.

Claims

1. A surgical assistance system comprising: an acquiring unit for acquiring operation information of a medical instrument during surgery on a target part of a subject; a detection unit that detects a triggering event that may be a cause of the abnormality based on the acquired operation information; a generating unit that generates correlation information that correlates a time at which a triggering event that may cause the abnormality is detected, the operation information acquired at that time, and a risk level of the triggering event; and The display control unit displays timeline information indicating the time position of the induced event and the risk level of the induced event based on the associated information, and displays thumbnail information of the induced event corresponding to the time position specified by the timeline information simultaneously with the medical image during surgery.

2. The surgical assistance system according to claim 1, wherein: The display control unit displays the medical image during surgery collected at the time when the triggering event that may cause the abnormality is detected in association with the corresponding time in the timeline information.

3. The surgical assisting system according to claim 1 or 2, wherein: The acquisition unit further acquires position information indicating the position of the medical instrument used in the operation. The generating unit generates associated information that associates a time at which a triggering event that may cause the abnormality is detected, the operation information acquired at that time, and the position information acquired at that time.

4. The surgical assistance system according to claim 3, wherein: The acquiring unit further acquires a medical image of the subject. The display control unit displays, on the medical image of the subject, a position of the medical instrument corresponding to a time when a triggering event that may cause the abnormality is detected.

5. The surgical assistance system according to claim 4, wherein: The display control unit displays the current position of the medical instrument in the medical image of the subject. The surgical assistance system according to claim 1 , wherein: The display control unit displays a method for detecting a triggering event that may cause the abnormality to occur in a identifiable manner.

7. The surgical assistance system according to claim 1, wherein: The detection unit detects a triggering event that may cause the abnormality to occur based on an image feature amount in an image of the subject acquired during surgery.

Images