Surgical assistance system and surgical assistance method

Abstract

Embodiments disclosed in the present specification and drawings relate to a surgical assistance system and a surgical assistance method. The purpose of the present invention is to easily confirm information relating to abnormalities occurring during surgery. A surgical assistance system according to an embodiment includes an acquisition unit, a detection unit, and a generation unit. The acquisition unit acquires medical information of a subject during surgery. The detection unit detects an event associated with an abnormality on the basis of the acquired medical information of the subject. The generation unit generates association information in which a time at which an event associated with the abnormality is detected is associated with the medical information acquired at the time.

Description

[0001] This application is a divisional application of application No. 202110724018.6, filed on June 29, 2021, for "Surgical Assistance System and Surgical Assistance Method".

[0002] This application enjoys priority of Japanese Patent Application No. 2020-111615 filed on June 29, 2020, the entire contents of which are hereby incorporated by reference. TECHNICAL FIELD

[0003] The embodiments disclosed in this specification and the accompanying drawings relate to a surgical assistance system and a surgical assistance method. BACKGROUND

[0004] In the past, various surgical assistance systems used in surgery have been known. For example, a surgical assistance system in which, in laparoscopic surgery, in order to avoid damage to blood vessels and internal organs, a virtual endoscope image is generated from a CT (Computed Tomography) image photographed before surgery, and is presented in conjunction with an actual endoscope image during surgery has been known. SUMMARY

[0005] The problem to be solved by the present application is to make information related to an abnormality occurring in surgery easy to confirm.

[0006] The surgical assistance system of the embodiments includes an acquisition unit, a detection unit, and a generation unit. The acquisition unit acquires medical information of a subject in surgery. The detection unit detects an item associated with an abnormality based on the acquired medical information of the subject. The generation unit generates association information that associates a time at which the item associated with the abnormality is detected with the medical information acquired at the time.

[0007] According to the surgical assistance system of the embodiments, it is possible to make information related to an abnormality occurring in surgery easy to confirm. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a view showing an example of the configuration of a surgical assistance system of Embodiment 1.

[0009] Figure 2 is a view showing an example of display control performed by a control function of Embodiment 1.

[0010] Figure 3 is a view showing an example of display control performed by a control function of Embodiment 1.

[0011] Figure 4 is a view showing an example of display control performed by a control function of Embodiment 1.

[0012] Figure 5 is a flowchart for explaining the processing steps of the surgery assisting apparatus of the first embodiment. DETAILED DESCRIPTION

[0013] Hereinafter, the embodiments of the surgery assisting system and the surgery assisting method will be described in detail with reference to the drawings. In addition, the surgery assisting system and the surgery assisting method of the present application are not limited by the embodiments shown below. Furthermore, the embodiments can be combined with other embodiments, prior art, within a range not conflicting with the processing content.

[0014] (First Embodiment)

[0015] Figure 1 is a diagram showing an example of the configuration of the surgery assisting system 10 of the first embodiment. Here, in the Figure 1 , the surgery assisting system 10 including the surgery assisting apparatus that performs the surgery assistance of the present application is explained, but the embodiments are not limited thereto, and the surgery assisting method explained below can also be performed by any apparatus in the surgery assisting system 10.

[0016] For example, as shown in Figure 1 , the surgery assisting system 10 of the present embodiment includes a medical image diagnostic apparatus 1, an endoscope system 2, a virtualized laparoscope image system 3, a position sensor 4, and a surgery assisting apparatus 5. Here, each of the apparatuses and the systems are communicably connected via a network. Furthermore, in the first embodiment, the case where laparoscopic surgery is performed as the surgery is explained as an example, but the surgery is not limited thereto, and the case where it is applied to other surgeries can also be included. Furthermore, the surgery assisting system 10 can also include systems (for example, a HIS (Hospital Information System) and the like), apparatuses (for example, an image storage apparatus), and the like other than those shown in the diagram.

[0017] The medical image diagnostic apparatus 1 collects medical images by imaging a subject. Then, the medical image diagnostic apparatus 1 transmits the collected medical images to the virtualized laparoscope image system 3, the surgery assisting apparatus 5, and the like. For example, the medical image diagnostic apparatus 1 is an X-ray diagnostic apparatus, an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, an ultrasonic diagnostic apparatus, a SPECT (Single Photon Emission Computed Tomography) apparatus, a PET (Positron Emission computed Tomography) apparatus, and the like.

[0018] The medical image diagnosis apparatus 1 collects medical images related to a subject of surgery. Specifically, the medical image diagnosis apparatus 1 collects medical images of a site as a target of surgery before and after surgery. Then, the medical image diagnosis apparatus 1 transmits the collected medical images to the virtualized laparoscope image system 3, the surgery assisting apparatus 5, and the like.

[0019] The endoscope system 2 has an endoscope 21, a display 22, a processing circuit 23, and a storage circuit 24. The endoscope 21 includes an insertion portion inserted into a subject, and an operation portion that operates the insertion portion. The insertion portion is a treatment portion that treats a site of a subject (a lesion) inside the subject, and an imaging portion that images the inside of the subject. The operation portion receives an operation of the treatment portion and the imaging portion by a surgeon.

[0020] The treatment portion is, for example, a forceps, an electrocautery, a stapler, or the like. Further, the imaging portion has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a lens, and a light emission portion, and images a lesion on which light is irradiated from the light emission portion by the imaging element.

[0021] The display 22 displays a video (an endoscope image) imaged by the imaging portion. The processing circuit 23 is connected to the endoscope 21, the display 22, and the storage circuit 24, and controls the entire endoscope system. For example, the processing circuit 23 controls an operation of the treatment portion in the endoscope 21, a collection of an endoscope image by the imaging portion, a display of the endoscope image by the display 22, a saving of the endoscope image to the storage circuit 24, and the like. The storage circuit 24 stores an endoscope image 241 collected by the imaging portion of the endoscope 21.

[0022] For example, a surgeon or the like inserts the endoscope 21 having a treatment portion such as a forceps or an electrocautery as an insertion portion, and the endoscope 21 having an imaging portion as an insertion portion into a subject, and operates the treatment portion while observing an endoscope image displayed on the display 22 by the imaging portion, thereby treating a site of a subject (a lesion) inside the subject.

[0023] The virtualized laparoscope image system 3 has a display 31, a processing circuit 32, and a storage circuit 33. The display 31 displays an image generated by the processing circuit 32. Specifically, the display 31 displays a virtualized laparoscope image generated on the basis of a medical image collected by the medical image diagnosis apparatus 1.

[0024] The processing circuit 32 is connected to the display 31 and the storage circuit 33, and controls the entire virtualized laparoscope image system. Specifically, the processing circuit 32 controls the acquisition of the medical image from the medical image diagnostic apparatus 1, the display of the virtualized laparoscope image by the display 22, the saving of the virtualized laparoscope image to the storage circuit 33, and the like. Further, the processing circuit 32 generates the virtualized laparoscope image using the medical image by executing a generation function 321. For example, the generation function 321 generates the virtualized laparoscope image from a prescribed line-of-sight direction based on information of a site in the abdominal cavity included in a two-dimensional CT image generated using a three-dimensional CT image collected from the abdominal portion of the subject by the X-ray CT apparatus as the medical image diagnostic apparatus 1 before the surgery.

[0025] When an example is taken, the generation function 321 generates the virtualized laparoscope image obtained by projecting the abdominal cavity from a prescribed line-of-sight direction based on information of a site in the abdominal cavity included in a two-dimensional CT image generated using a three-dimensional CT image. The storage circuit 33 stores the virtualized laparoscope image 331 generated by the processing circuit 32.

[0026] The position sensor 4 has a sensor portion, a magnetic field generating portion, and a signal receiving portion. The sensor portion is, for example, a magnetic sensor, and is disposed in the subject inside the distal end of the insertion portion of the endoscope 21. The magnetic field generating portion is disposed near the subject to form a magnetic field toward the outside with the apparatus as the center. The signal receiving portion receives a signal output from the sensor portion.

[0027] The sensor portion detects a three-dimensional magnetic field formed by the magnetic field generating portion. Then, the sensor portion calculates position information (coordinates and angles) of the apparatus in a space with the magnetic field generating portion as the origin based on information of the detected three-dimensional magnetic field, and transmits the calculated position information of the apparatus to the signal receiving portion. For example, the position information received from the sensor portion mounted to the distal end of the insertion portion of the endoscope 21 indicates the position of the distal end of the insertion portion in the space with the magnetic field generating portion as the origin. Further, the position information received from the sensor portion disposed in the subject (for example, an internal organ of the affected portion) indicates the position of the affected portion in the space with the magnetic field generating portion as the origin. The signal receiving portion transmits the position information received from the sensor portion to the virtualized laparoscope image system 3 and the surgery assisting apparatus 5.

[0028] Here, the virtualized laparoscope image system 3 is able to generate and display the virtualized laparoscope image linked with the endoscope image by using the position information acquired by the above-described position sensor 4. In this case, first, the alignment between the three-dimensional coordinates of the space with the magnetic field generating portion as the origin and the three-dimensional coordinates in the three-dimensional medical image used in the generation of the virtualized laparoscope image is performed.

[0029] For example, the generation function 321 extracts a position in the three-dimensional medical image corresponding to the position information acquired by the sensor section arranged in the subject (for example, an internal organ of the affected part), and performs registration of setting the extracted position to the same position as the position acquired by the sensor section. Here, the generation function 321 performs registration between the three-dimensional coordinates of the space with the magnetic field generating section as the origin and the three-dimensional coordinates of the three-dimensional medical image used in the generation of the virtualized laparoscope image by performing the above registration on the position information from the sensor sections arranged at a plurality of positions in the subject.

[0030] In addition, the registration between the three-dimensional coordinates of the space with the magnetic field generating section as the origin and the three-dimensional coordinates of the three-dimensional medical image used in the generation of the virtualized laparoscope image is not limited to the above method, and other methods can be used. For example, position information of a site in the subject depicted in an endoscope image imaged by the imaging section of the endoscope 21 on which the sensor section is mounted can be used.

[0031] In this case, for example, the generation function 321 calculates the three-dimensional coordinates of a site (a characteristic site or the like) depicted in the endoscope image in the space with the magnetic field generating section as the origin based on the position information from the sensor section mounted to the imaging section. Then, the generation function 321 extracts a position in the three-dimensional medical image corresponding to the site for which the three-dimensional coordinates are calculated, and performs registration of setting the extracted position and the position for which the three-dimensional coordinates are calculated to the same position. Here, the generation function 321 performs registration between the three-dimensional coordinates of the space with the magnetic field generating section as the origin and the three-dimensional coordinates in the three-dimensional medical image used in the generation of the virtualized laparoscope image by performing the above registration on a plurality of positions in the subject.

[0032] Thus, when performing registration, the generation function 321 generates a virtualized laparoscope image linked to the endoscope image based on the position information of the sensor section mounted to the imaging section of the endoscope 21. For example, the generation function 321 generates a virtualized laparoscope image obtained by projecting a three-dimensional CT image (intra-abdominal cavity) by taking the three-dimensional coordinates acquired by the sensor section mounted to the imaging section of the endoscope 21 as a viewpoint and taking the imaging direction derived from the angle of the sensor section as a projection direction.

[0033] Then, the generation function 321 generates virtualized laparoscope images in which the viewpoint and the projection direction are changed in order according to the change in the position of the imaging section (change in the three-dimensional coordinates and the angle acquired by the sensor section). By sequentially displaying the virtualized laparoscope images thus generated in order, a virtualized laparoscope image linked to the change in the endoscope image is displayed.

[0034] Here, the generation function 321 is able to reflect the change in the shape of the internal organ during surgery into the virtualized laparoscope image by using the position information acquired by the sensor section disposed in the subject (for example, the internal organ of the affected part). For example, the generation function 321 calculates the amount of change each time the position information acquired by the sensor section disposed in the internal organ changes, and causes the corresponding position in the three-dimensional medical image to change by the calculated amount of change. Then, the generation function 321 generates the virtualized laparoscope image by using the changed medical image, thereby reflecting the change in the shape of the internal organ during surgery into the virtualized laparoscope image.

[0035] The surgery support apparatus 5 generates information on an item associated with an abnormality generated during surgery, based on various information during surgery. Specifically, the surgery support apparatus 5 acquires information from the medical image diagnostic apparatus, various medical devices during surgery, and generates information on an item associated with an abnormality during surgery, based on the acquired information. For example, the surgery support apparatus 5 is realized by a computer device such as a workstation, a personal computer, a tablet terminal, or the like.

[0036] For example, the surgery support apparatus 5 is provided with an input interface 51, a display 52, a storage circuit 53, and a processing circuit 54. Also, the surgery support apparatus 5 is connected to the medical image diagnostic apparatus 1, the endoscope system 2, the virtualized laparoscope image system 3, and the position sensor 4 via a network.

[0037] The input interface 51 receives various instructions and various information input operations from a user. Specifically, the input interface 51 is connected to the processing circuit 54, converts the input operation received from the user into an electric signal, and outputs the electric signal to the processing circuit 54. For example, the input interface 51 is realized by a track ball, a switch, a button, a mouse, a keyboard, a touch pad that performs an input operation by a contact operation with a touch surface, a touch screen in which a display screen and a touch pad are integrated, a non-contact input interface using an optical sensor, a sound input interface, and the like. In addition, in the present specification, the input interface 51 is not limited to having a physical operation member such as a mouse and a keyboard. For example, an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the apparatus and outputs the electric signal to a control circuit is also included in the example of the input interface 51.

[0038] The display 154 displays various information and various data. Specifically, the display 154 is connected to the processing circuit 155, and displays various information and various data output from the processing circuit 155. For example, the display 154 is realized by a liquid crystal display, a CRT (Cathode Ray Tube) display, an organic EL display, a plasma display, a touch panel, or the like.

[0039] The storage circuit 53 stores various data and various programs. Specifically, the storage circuit 53 is connected to the processing circuit 54, and stores data input from the processing circuit 54, or reads out stored data and outputs to the processing circuit 54. For example, the storage circuit 53 is realized by a RAM (Random Access Memory), a semiconductor storage element such as a flash memory, a hard disk, an optical disk, or the like.

[0040] For example, the storage circuit 53 stores the determination condition 531 and the association information 532. Details of the determination condition 531 and the association information 532 will be described later.

[0041] The processing circuit 54 controls the entire surgery assisting apparatus 5. For example, the processing circuit 54 performs various processes according to an input operation received from a user via the input interface 51. For example, the processing circuit 54 stores data transmitted from other apparatuses in the storage circuit 53. Further, for example, the processing circuit 54 transmits data read out from the storage circuit 53 to other apparatuses by outputting the data. Further, for example, the processing circuit 54 displays data read out from the storage circuit 53 on the display 52.

[0042] Here, each processing circuit in the above-described endoscope system 2, the virtualized laparoscope image system 3, and the surgery assisting apparatus 5 is realized by, for example, a processor. In this case, each processing function described above is stored in the storage circuit as a program executable by a computer. Then, each processing circuit realizes a function corresponding to each program by reading out each program stored in each storage circuit and executing the program. In other words, each processing circuit has each processing function illustrated in the drawing in a state where each program is read out. Figure 1

[0043] ​Further, each processing circuit can also be configured by combining a plurality of independent processors, and each processing function can be realized by each processor executing a program. In addition, each processing function possessed by each processing circuit can be appropriately merged or distributed to a single or a plurality of processing circuits to be realized. Furthermore, each processing function possessed by each processing circuit can be realized by a mixture of hardware and software. In addition, here, an example in which a program corresponding to each processing function is stored in a single storage circuit is described, but the embodiments are not limited thereto. For example, it can be configured such that the program corresponding to each processing function is stored in a plurality of storage circuits, and each processing circuit reads out each program from each storage circuit and executes it.

[0044] The configuration of the surgery assistance system 10 of the present embodiment has been described above. For example, the surgery assistance system 10 of the present embodiment is arranged in an operating room of a medical institution such as a hospital or a clinic, and assists in confirming abnormalities related to an abnormality occurring in surgery performed by a user such as a doctor.

[0045] For example, in laparoscopic surgery, bleeding that is not noticed by the surgeon can occur. When one example is given, there can be given a case in which an instrument comes into contact with a blood vessel, a case in which a blood vessel on the inner side is damaged when a film is peeled off with an electrotome, a case in which a rupture occurs in another part to which pressure is applied due to the force of the internal organs pressed by forceps becoming strong, a case in which a rupture occurs in another part that is weak as a result of slight stretching when a blood vessel is gripped by forceps, and the like.

[0046] In a case in which such bleeding occurs, the bleeding site is determined and hemostasis is performed, and the endoscopic surgery is continued, but the endoscopic field of view becomes poor when bleeding occurs, and determination of the bleeding site becomes difficult. In addition, even if the image based on the endoscopic image is returned and confirmed in order to investigate the bleeding site, it is difficult for the surgeon to grasp when and where this operation was performed, and there is a case in which determination from the image is difficult and time-consuming. Furthermore, although the internal organ structure outside the field of view can be grasped from the virtual endoscopic image, an abnormality such as a bleeding event occurring in surgery is not reflected in the virtual endoscopic image, and therefore determination from the virtual endoscopic image is difficult.

[0047] Thus, in a case in which determination of the bleeding site takes time and hemostasis cannot be performed, a laparotomy surgery is switched to and the surgery is continued, but the wound becomes large, the switching of the laparotomy surgery is troublesome and the patient is in a dangerous state, and the burden on the patient is large.

[0048] Accordingly, the surgery assisting apparatus 5 of the surgery assisting system 10 of the present embodiment is configured to acquire various information during surgery and generate information in which the acquired information is associated with time information, whereby information related to an abnormality occurring during surgery can be easily confirmed.

[0049] Specifically, the surgery assisting apparatus 5 continuously acquires information from the medical image diagnostic apparatus 1, the endoscope system 2, and other various medical devices used during surgery, analyzes the acquired information, and stores information of matters determined to be associated with an abnormality in association with the time of acquisition. Thus, the surgery assisting system 10 is capable of presenting information of matters determined to be associated with an abnormality when an event (abnormality) such as bleeding occurs during surgery, and capable of easily confirming information related to an abnormality occurring during surgery. Hereinafter, the surgery assisting apparatus 5 having such a configuration will be described in detail.

[0050] For example, as shown in FIG. 6, in the present embodiment, the processing circuit 54 of the surgery assisting apparatus 5 executes a control function 541, an analysis function 542, and a generation function 543. Here, the control function 541 is an example of an acquisition unit and a display control unit. Further, the analysis function 542 is an example of a detection unit. Further, the generation function 543 is an example of a generation unit. Figure 1

[0051] The control function 541 acquires various data (medical information) from other apparatuses connected via a network and stores the acquired medical information in the storage circuit 53. For example, the control function 541 acquires a medical image collected by the medical image diagnostic apparatus 1, an endoscope image generated by the endoscope system 2, and a virtualized laparoscope image generated by the virtualized laparoscope image system 3.

[0052] Here, the control function 541 can acquire medical information of a subject before and during surgery. For example, the control function 541 acquires a medical image collected before surgery and a medical image collected during surgery, respectively. Further, the control function 541 acquires an endoscope image during surgery. Further, the control function 541 acquires a virtualized laparoscope image generated before surgery and a virtualized laparoscope image generated during surgery, respectively.

[0053] ​Further, the control function 541 can acquire various medical information acquired from the subject in surgery. For example, the control function 541 acquires vital information acquired from the subject in surgery, position information acquired by the position sensor 4, or various information acquired by various sensors mounted to the endoscope 21. Here, as the various sensors mounted to the endoscope 21, for example, a MEMS (Micro Electro Mechanical Systems) sensor that acquires pressure information of forceps or the like can be given. In a case where the endoscope 21 is mounted with the above-described MEMS sensor, the control function 541 can also acquire the pressure information acquired by the MEMS sensor.

[0054] Further, the control function 541 causes the display 52 to display various medical information. For example, the control function 541 causes the display 52 to display information generated on the basis of an analysis result of the acquired medical information. Here, the control function 541 can also perform control so as to transmit the generated information to another device via a network and cause a display of the other device to display. For example, the control function 541 transmits the generated information to the endoscope system 2, the virtualized laparoscope image system 3. The endoscope system 2 and the virtualized laparoscope image system 3 cause the display of the present device to display the information received from the surgery support device 5.

[0055] The analysis function 542 detects matters associated with abnormalities on the basis of the acquired medical information of the subject. Specifically, the analysis function 542 compares the medical information acquired by the control function 541 with the determination condition 531 stored by the storage circuit 53, and detects information in the acquired medical information that matches the determination condition as a matter associated with an abnormality. For example, the analysis function 542 detects matters associated with abnormalities in surgery on the basis of a medical image acquired from the medical image diagnosis device 1, an endoscope image acquired from the endoscope system 2, and vital information, and the like. When one example is given, the analysis function 542 can detect matters associated with abnormalities on the basis of an image feature amount in an image of the subject acquired in surgery (medical image, endoscope image). In addition, the determination condition 531 includes various conditions corresponding to the acquired medical information.

[0056] The analysis function 542 detects, as a matter associated with an abnormality, for example, a causation matter that causes an abnormality. In this case, for example, as a determination condition corresponding to analysis using an endoscope image, conditions such as "whether a treatment portion is in contact with a blood vessel", "an invasion degree of a tissue by an electrotome", "a time of holding a blood vessel by forceps", "a time of pressing an internal organ by forceps", "a deformation degree of an internal organ", and the like can be given.

[0057] For example, "whether the treatment section is in contact with the blood vessel" in the determination condition 531 indicates a condition in which whether the treatment section of the endoscope 21 is in contact with the blood vessel and further the degree of contact is classified in stages. Also, the degree of contact can be classified, for example, according to the amount of movement of the treatment instrument. When an example is given, the degree of contact is classified in a manner in which the greater the amount of movement of the treatment instrument, the greater the degree of contact.

[0058] The analysis function 542 determines whether the treatment section is in contact with the blood vessel through image analysis of each endoscope image sequentially acquired from the endoscope system 2. Then, the analysis function 542, in a case where it is determined that the treatment section is in contact with the blood vessel, calculates the amount of movement of the treatment section from the previous endoscope image in chronological order and compares the calculated amount of movement with the classification in the determination condition 531, thereby classifying the degree of contact. Here, the analysis function 542 detects a case where the treatment section is in contact with the blood vessel as an inducing event and determines the classified degree of contact as a risk level. For example, the analysis function 542 determines that the greater the degree of contact, the higher the risk level.

[0059] Further, "the degree of invasion of the electrotome into the tissue" in the determination condition 531 indicates a condition in which the degree of invasion of the electrotome into the tissue is classified in stages. Also, the degree of invasion can be classified, for example, according to the amount of movement of the electrotome. When an example is given, the degree of invasion is classified in a manner in which the greater the amount of movement of the electrotome, the greater the degree of invasion. The analysis function 542 calculates the amount of movement of the electrotome from the previous endoscope image in chronological order through image analysis of each endoscope image sequentially acquired from the endoscope system 2 and compares the calculated amount of movement with the classification in the determination condition 531, thereby classifying the degree of invasion. Here, the analysis function 542 detects a case where the degree of invasion exceeds a prescribed degree as an inducing event and determines the classified degree of invasion as a risk level. For example, the analysis function 542 determines that the greater the degree of invasion, the higher the risk level.

[0060] Further, "the time of pinching the blood vessel with the forceps" and "the time of pressing the internal organs with the forceps" in the determination condition 531 indicate conditions in which the length of time is classified in stages. The analysis function 542 calculates the "time of pinching the blood vessel with the forceps", "the time of pressing the internal organs with the forceps" through image analysis of each endoscope image sequentially acquired from the endoscope system 2 and compares the calculated time with the classification in the determination condition 531, thereby classifying the calculated time. Here, the analysis function 542 detects a case where the calculated time exceeds a prescribed time as an inducing event and determines the length of time as a risk level. For example, the analysis function 542 determines that the longer the calculated time, the higher the risk level.

[0061] Further, the "degree of deformation of the internal organ" in the determination condition 531 indicates a condition in which the degree of deformation of the internal organ is classified in stages. In addition, the degree of deformation of the internal organ can also be classified, for example, according to the amount of change in the shape of the internal organ. When an example is given, the degrees of deformation are classified in a manner in which the greater the amount of change in the shape of the internal organ, the greater the degree of deformation. The analysis function 542 classifies the degree of deformation by image analysis of each endoscope image sequentially acquired from the endoscope system 2, calculating the amount of change in the shape of the internal organ from the previous endoscope image in chronological order, and comparing the calculated amount of change with the classification in the determination condition 531, thereby classifying the degree of deformation. Here, the analysis function 542 detects a case in which the amount of change exceeds a prescribed amount as an inducing event, and determines the classified degree of deformation as a risk level. For example, the analysis function 542 determines that the greater the degree of deformation, the higher the risk level.

[0062] In addition, for example, the internal organ is deformed by pressing the internal organ with forceps or stretching the internal organ with forceps. Further, for example, the insertion portion of the endoscope 21 inserted into the body presses the internal organ, thereby deforming the internal organ.

[0063] In addition, the above-described image analysis can also be performed by feature detection based on AI (Artificial Intelligence). Further, in the above-described example, a case in which the amount of movement of the treatment portion and the amount of change in the internal organ are calculated by image analysis is described. However, the embodiments are not limited thereto, and, for example, in a case in which the position sensor 4 is used, the position information acquired by the position sensor 4 can also be used. When an example is given, the analysis function 542 calculates the amount of movement of the treatment portion based on the position information acquired by the sensor portion installed at the distal end of the treatment portion. Further, for example, the analysis function 542 calculates the amount of deformation of the internal organ based on the position information acquired by the sensor portion disposed in the internal organ at the affected portion.

[0064] Further, the analysis function 542 is capable of detecting various other events as events associated with abnormalities. In this case, for example, as determination conditions corresponding to analysis using medical images, conditions such as "presence or absence of blood flow leakage detected by Color Doppler Imaging" can be given. For example, the "presence or absence of blood flow leakage detected by Color Doppler Imaging" in the determination condition 531 indicates a condition in which whether or not blood flow leakage has occurred, and further the degree of blood flow leakage is classified in stages.

[0065] For example, the analysis function 542 determines whether or not there is an outflow of blood flow by image analysis of color Doppler images sequentially acquired from the ultrasonic diagnostic apparatus as the medical image diagnostic apparatus 1. Then, the analysis function 542 detects a case where it is determined that there is an outflow of blood flow as an item associated with an abnormality, and determines a degree of outflow of blood flow as a risk level in accordance with the determination condition 531. For example, the analysis function 542 determines that the greater the degree of outflow of blood flow, the higher the risk level.

[0066] Further, for example, as the determination condition corresponding to analysis using vital information, there are conditions such as "decrease in blood pressure", "ECG abnormality", "ischemia", and the like. For example, "decrease in blood pressure" in the determination condition 531 indicates a condition in which whether or not the blood pressure becomes a prescribed value or less, and further, the degree of decrease in blood pressure is classified in stages.

[0067] For example, the control function 541 acquires blood pressure information of the subject from a blood pressure monitor during surgery. The analysis function 542 determines whether or not the blood pressure becomes a prescribed value or less based on the blood pressure information sequentially acquired from the blood pressure monitor. Then, the analysis function 542 detects a case where it is determined that the blood pressure becomes a prescribed value or less as an item associated with an abnormality, and determines a degree of decrease in blood pressure as a risk level in accordance with the determination condition 531. For example, the analysis function 542 determines that the greater the degree of decrease in blood pressure, the higher the risk level.

[0068] Further, for example, "ECG abnormality" in the determination condition 531 indicates a condition in which whether or not a rhythm, a waveform in an ECG has changed by a prescribed amount of change, and further, the degree of change is classified in stages.

[0069] For example, the control function 541 acquires an ECG of the subject from an ECG instrument during surgery. The analysis function 542 determines whether or not a rhythm, a waveform in the ECG has changed by a prescribed amount of change based on the ECG sequentially acquired from the ECG instrument. Then, the analysis function 542 detects a case where it is determined that the rhythm, the waveform in the ECG has changed by a prescribed amount of change as an item associated with an abnormality, and determines a degree of the amount of change as a risk level in accordance with the determination condition 531. For example, the analysis function 542 determines that the greater the amount of change, the higher the risk level.

[0070] Further, for example, "ischemia" in the determination condition 531 indicates a condition in which whether ischemia occurs and further the degree of ischemia is classified in stages. The analysis function 542 determines whether ischemia occurs on the basis of the waveforms of electrocardiograms successively acquired from an electrocardiograph. Then, the analysis function 542 detects a case where ischemia occurs as a matter associated with an abnormality and determines the degree of ischemia as a risk level in accordance with the determination condition 531. For example, the analysis function 542 determines that the greater the degree of ischemia, the higher the risk level.

[0071] As described above, the analysis function 542 detects a matter associated with an abnormality on the basis of medical information acquired in surgery and determines a risk level of the detected matter associated with an abnormality. Here, the above-described determination condition is only an example, and a matter associated with an abnormality can be detected on the basis of other information. For example, a matter associated with an abnormality can be detected on the basis of acquired pressure information when a MEMS sensor is attached to a treatment portion and pressure information is acquired by the MEMS sensor.

[0072] In this case, as the determination condition 531, a condition related to pressure information is stored. For example, "pressure" in the determination condition 531 indicates a condition in which whether the acquired pressure value exceeds a prescribed value and further the degree of pressure is classified in stages.

[0073] For example, the control function 541 acquires pressure information acquired by a MEMS sensor. The analysis function 542 determines whether the pressure exceeds a prescribed value on the basis of the pressure information acquired by the control function 541. Then, the analysis function 542 detects a case where it is determined that the pressure exceeds the prescribed value as a matter associated with an abnormality and determines the degree of pressure as a risk level in accordance with the determination condition 531. For example, the analysis function 542 determines that the greater the pressure value, the higher the risk level.

[0074] The above describes an example in which the analysis function 542 performs analysis. In addition, in the above-described example, a case in which a matter associated with an abnormality is detected on the basis of one condition is described, but the embodiment is not limited thereto, and a plurality of conditions can be combined to detect a matter associated with an abnormality. For example, it can be a case in which a plurality of conditions (for example, whether a treatment portion is in contact with a blood vessel and the degree of deformation of an internal organ) included in analysis based on a single device (for example, analysis using an endoscope image) are combined to detect a matter associated with an abnormality, or it can be a case in which conditions of analysis based on a plurality of devices (for example, analysis using an endoscope image and analysis using vital information) are combined to detect a matter associated with an abnormality. Further, various conditions of the determination condition 531 referred to by the analysis function 542 can be arbitrarily set. For example, it can be appropriately set in accordance with surgery, the kind of an internal organ of a lesion portion, or the like.

[0075] The generation function 543 generates association information that associates the time when the matter associated with the abnormality is detected and the medical information taken at the time. Specifically, the generation function 543 generates association information that corresponds to the medical information when the matter associated with the abnormality is detected by the analysis function 542 and the time when the medical information is taken. For example, in the case where the matter associated with the abnormality (inducing matter) is detected in the analysis based on "the degree of deformation of the internal organ", the generation function 543 generates association information that associates the endoscope image in which the inducing matter is detected and the time when the endoscope image is taken. In addition, when the endoscope image is taken by the control function 541, the time when the endoscope image is taken is also taken at the same time.

[0076] Here, the generation function 543 can generate association information that corresponds only to the medical information in which the matter associated with the abnormality is detected and the time, or can further generate association information that corresponds to other medical information of the time when the medical information in which the matter associated with the abnormality is detected is taken. For example, the vital information is taken together with the endoscope image, and in the case where the matter associated with the abnormality is detected in the analysis based on the "degree of deformation of the internal organ" of the endoscope image, the generation function 543 can generate association information that further associates the vital information taken at the same time. That is, the generation function 543 generates association information that further associates the vital information of the time when the endoscope image in which the matter associated with the abnormality is detected is taken.

[0077] Further, the generation function 543 can further associate the risk level with the association information that associates the time when the medical information in which the matter associated with the abnormality is detected is taken and the medical information taken at the time. For example, the generation function 543 further associates the risk level determined in the analysis based on "the degree of deformation of the internal organ" with the association information.

[0078] Further, the generation function 543 can further generate association information that associates the position information. Specifically, the generation function 543 generates association information that associates the time when the matter associated with the abnormality is detected, the medical information taken at the time, and the position information taken at the time. In this case, first, the sensor portion of the position sensor 4 is attached to the distal end of the insertion portion of the endoscope 21, and the position information of the distal end of the insertion portion during the surgery is taken. The control function 541 takes the position information taken by the position sensor 4 from the position sensor 4, and saves the taken position information in the storage circuit 53 in correspondence with the time when the position information is taken.

[0079] The generation function 543 generates the association information that associates the time when the medical information that detects the matter associated with the abnormality is acquired and the medical information acquired at that time. For example, when the matter associated with the abnormality is detected in the analysis based on "the degree of deformation of the internal organ", the generation function 543 generates the association information that associates the endoscope image that detects the matter associated with the abnormality, the time when the endoscope image is photographed, and the position information at that time.

[0080] In addition, the information associated with the association information can be appropriately set. For example, the generation function 543 appropriately associates the time, the medical information, the analysis result, and the position information as the association information according to the information acquired in the surgery.

[0081] The generation function 543 generates the above-described association information and stores it in the storage circuit 53 every time the matter associated with the abnormality is detected by the analysis function 542 in the surgery. The association information 532 in the storage circuit 53 is generated and stored by the generation function 543 as described above. In addition, the association information 532 can also be stored for each surgery and read out and used after the surgery.

[0082] When the association information is generated as described above, the control function 541 causes the display 52 to display various information using the association information. In addition, the control function 541 can also display various information using the association information on the display 22 in the endoscope system 2, the display 31 in the virtualized laparoscope image system 3.

[0083] Hereinafter, an example of the information displayed by the control function 541 will be described.

[0084] For example, the control function 541 displays timeline information that represents the association information in chronological order. Figure 2 is a diagram that represents an example of the display control performed by the control function 541 of the first embodiment. Here, Figure 2 represents the display of the timeline information with respect to the endoscope image displayed by the endoscope system 2.

[0085] For example, as shown in the upper diagram of Figure 2 In the endoscope system 2, the display 22 displays a real-time image inside the abdominal cavity based on the endoscope image acquired in the surgery. During this period, the surgery assisting device 5 acquires various medical information to determine whether a matter associated with an abnormality occurs.

[0086] Here, the analysis function 542 detects the matter associated with the abnormality, and when the generation function 543 generates the association information, as shown inFigure 2 As illustrated in the middle of FIG. 6, the control function 541 controls based on the generated association information so as to display the timeline information 101 in the endoscope image.

[0087] Here, the control function 541 displays information capable of identifying at least one of the degree of the detected matter associated with the abnormality and the detection method, as indicated by the position 1 of the timeline information 101. For example, the control function 541 displays the position 1 of the timeline information in a color corresponding to the risk level of the detected matter associated with the abnormality. Further, for example, the control function 541 displays the position 1 of the timeline information in a color corresponding to the detection method (for example, the condition used in the analysis) of the detected matter associated with the abnormality. Here, in a case where the risk level and the detection method can be identified, the control function 541 assigns the color of the outer frame of the position 1 on the timeline information 101 and the color inside to the detection method and the risk level, for example. When an example is taken, the control function 541 displays the color corresponding to the detection method as the color of the outer frame indicating the position 1 and the color corresponding to the risk level as the color inside the frame indicating the position 1.

[0088] Further, the control function 541 can change the display according to the change in the risk level. As described above, the analysis function 542 detects the matter associated with the abnormality based on the medical information sequentially acquired by the control function 541. Thus, in a case where the matter associated with the abnormality is continuously detected in the sequentially acquired medical information, the association information is continuously generated, and the control function 541 continuously displays the identifiable information with respect to the timeline information.

[0089] Here, for example, in a case where the degree of deformation of the internal organ gradually changes, the risk level determined by the analysis function 542 gradually changes. In this case, the risk level associated with the association information generated by the generation function 543 changes. As a result, for example, as indicated by the lower of FIG. 6, the control function 541 displays the color at the position 1 in the timeline information 101 changed. Figure 2

[0090] For example, the control function 541, before the association information is generated, does not display the timeline information 101 as indicated by the upper of FIG. 6, and when the association information is generated, automatically displays the timeline information 101 as indicated by the middle and lower of FIG. 6. Then, the control function 541 can also control so as not to display the timeline information when the association information is not generated. Figure 2 Figure 2 For example, the control function 541, before the association information is generated, does not display the timeline information 101 as indicated by the upper of FIG. 6, and when the association information is generated, automatically displays the timeline information 101 as indicated by the middle and lower of FIG. 6. Then, the control function 541 can also control so as not to display the timeline information when the association information is not generated.

[0091] ​​Further, for example, the control function 541 can display the medical image in the surgery and the corresponding time in the timeline information, which are collected at the time when the matter associated with the abnormality is detected, in association. Figure 3 is a diagram showing an example of the display control performed by the control function 541 of the first embodiment. Here, Figure 3 indicates the display of the timeline information with respect to the endoscope image displayed by the endoscope system 2.

[0092] For example, the control function 541 displays the timeline information 101 or a thumbnail 102 in accordance with the operation of the operator, like Figure 3 When an example is taken, when the operator performs a display operation of the timeline information 101 in the surgery, the control function 541 displays the timeline information 101 based on the association information generated so far, as shown in the middle of Figure 3

[0093] Further, when the operator specifies a position 6 in the timeline information 101, the control function 541 displays the thumbnail 102 of the endoscope image corresponding to the specified position 6 in association with the position 6, as shown in the lower part of Figure 3

[0094] For example, when an abnormality such as bleeding occurs in the surgery, when the operator causes the timeline information to be displayed, the control function 541 displays the timeline information 101 in which the time when the matter associated with the abnormality is detected can be identified, as shown in the middle of Figure 3

[0095] Then, when the operator specifies a position in the timeline information 101, the control function 541 displays the thumbnail 102 of the image taken at the specified time, as shown in the lower part of Figure 3

[0096] Further, for example, the control function 541 displays the position of the medical instrument corresponding to the time when the matter associated with the abnormality is detected in the medical image of the subject. That is, the control function 541, when the position information is included in the association information, displays the medical image indicating the position information taken at the time when the matter associated with the abnormality is detected, based on the position information associated with the association information.

[0097] Figure 4 is a diagram showing an example of the display control performed by the control function 541 of the first embodiment. Here, Figure 4 ​​​​The center left drawing shows an endoscope image in which the timeline information 101 is displayed. Further, Figure 4 The center right drawing shows a medical image in which the position information is displayed.

[0098] For example, as shown in the left drawing, Figure 4 the control function 541 displays the timeline information 101 in the endoscope image, and displays a medical image showing the position information of the medical device at the time when the associated information is generated. Here, the control function 541 displays a medical image showing the position information using a three-dimensional medical image in which the alignment is performed with the three-dimensional coordinates of the space with the magnetic field generating portion in the position sensor 4 as the origin.

[0099] For example, as shown in the right drawing, Figure 4 the control function 541 displays the position information using a virtualized laparoscope image, a CT image of a three-dimensional medical image in which the alignment is performed with the three-dimensional coordinates of the space with the magnetic field generating portion as the origin. When an example is taken, the control function 541 determines the positions in the three-dimensional medical image corresponding to each position information based on the position information corresponding to the times of the positions 1 to 6 in the timeline information 101, and the aligned information.

[0100] Then, the control function 541 displays a medical image showing the identification information at each determined position. For example, as shown in the left drawing, Figure 4 the control function 541 displays the numbers corresponding to the positions 1 to 6 in the timeline information 101 at the determined positions on the CT image. Here, the control function 541 can display each number so that the detected risk level and the detection method can be recognized as well as the timeline information 101. For example, the control function 541 can display the risk level and the detection method in a recognizable manner according to the shape of the outer frame surrounding the number, the color inside the shape, and the like.

[0101] Further, the control function 541 can not only display the position information of the medical device at the time when the medical information detecting the matter associated with the abnormality is acquired, but also display the current position of the medical instrument (the distal end of the endoscope 21) in the medical image of the subject. That is, the control function 541 determines the position in the three-dimensional medical image corresponding to the current position of the medical instrument acquired from the position sensor 4. Then, the control function 541 displays a medical image showing the identification information at the determined position. For example, as shown in the right drawing, Figure 4 the control function 541 displays the current position 104 of the medical instrument.

[0102] Further, further, the control function 541 can display the current position of the medical instrument according to the operation of the operator, as shown in the right drawing, Figure 4Thumbnail 103 is displayed as shown. For example, when an operator performs the display operation of timeline information 101 and the specified operation of position 6 in timeline information 101 during surgery, as shown... Figure 3 As shown, control function 541 reflects the information of the specified position 6 into the virtual laparoscopic image, and displays the thumbnail 103 of the virtual laparoscopic image corresponding to the time of position 6 in association with the numbers.

[0103] For example, in the event of an abnormality such as bleeding during surgery, control function 541 displays an image showing the location of the medical instrument at the moment when medical information related to the abnormality was detected. This allows the operator to easily determine the location of the event that may have caused the abnormality.

[0104] Then, when the operator specifies the position in timeline information 101, such as Figure 5 As shown in the next section, control function 541 displays a thumbnail 103 of the image at the specified moment. This allows the operator to monitor the handling of events associated with abnormalities and quickly determine the cause of abnormalities such as bleeding.

[0105] Next, use Figure 5 The processing of the surgical aid device 5 in the first embodiment will be described. Figure 5 This is a flowchart illustrating the processing steps of the surgical aid device 5 according to the first embodiment. Additionally, in Figure 5 The image shows an example of how time, medical information, and location information are linked as related information.

[0106] Here, Figure 5 Steps S101-S102 and S106-S109 are implemented by the processing circuit 54 reading the program corresponding to the control function 541 from the storage circuit 53 and executing it. Furthermore, Figure 5 Step S103 is implemented by the processing circuit 54 reading the program corresponding to the analysis function 542 from the storage circuit 53 and executing it. Furthermore, Figure 5 Steps S104 to S105 are implemented by the processing circuit 54 reading the program corresponding to the generation function 543 from the storage circuit 53 and executing it.

[0107] like ​As illustrated, in the surgery assisting apparatus 5, first, the processing circuitry 54 determines whether or not surgery has started (step S101). For example, the processing circuitry 54 determines the start of surgery in accordance with whether or not an operation for starting the processing has been accepted. Here, when the surgery has started (step S101, YES), the processing circuitry 54 acquires medical information (step S102). In addition, the surgery assisting apparatus 5 is in a standby state before the surgery has started (step S101, NO).

[0108] Then, the processing circuitry 54 analyzes the medical information (step S103), and determines whether or not there is an item associated with an abnormality (step S104). Here, in a case where there is an item associated with an abnormality (step S104, YES), the processing circuitry 54 generates association information that associates the medical information in which the item associated with the abnormality has been detected, the time, and the position information, and stores the same (step S105). On the other hand, in a case where the item associated with the abnormality has not been detected in step S104 (step S104, NO), the processing circuitry 54 proceeds to step S106.

[0109] Then, in step S106, the processing circuitry 54 displays the medical information. For example, the processing circuitry 54 displays an endoscope image or the like. Here, in a case where the association information has been generated, the processing circuitry 54 displays an endoscope image or the like that includes timeline information or the like based on the association information. Then, the processing circuitry 54 determines whether or not a designation operation has been accepted (step S107).

[0110] Here, in a case where the designation operation has been accepted (step S107, YES), the processing circuitry 54 displays detailed information (for example, a thumbnail or the like) of the medical information designated (step S108), and determines whether or not the surgery has ended (step S109). On the other hand, in a case where the designation operation has not been accepted (step S107, NO), the processing circuitry 54 determines whether or not the surgery has ended (step S109).

[0111] In step S109, in a case where the surgery has not ended (step S109, NO), the processing circuitry 54 returns to step S102, and continues to acquire the medical information. On the other hand, in a case where the surgery has ended (step S109, YES), the processing circuitry 54 ends the processing.

[0112] As described above, according to the first embodiment, the control function 541 acquires medical information of a subject in surgery. The analysis function 542 detects an item associated with an abnormality based on the acquired medical information of the subject. The generation function 543 generates association information that associates a time at which the item associated with the abnormality has been detected, and medical information acquired at the time. Thus, the surgery assisting apparatus 5 of the first embodiment can provide information related to the time at which the item associated with the abnormality has been generated, and can easily confirm information related to the abnormality generated in the surgery.

[0113] For example, even in a case where bleeding occurs in the endoscopic surgery, by prompting the time at which the matter associated with the abnormality (bleeding) is detected, determination of the bleeding site can be performed promptly, as a result of which the endoscopic surgery can be continued without switching to the laparotomy surgery, and improvement in the patient QOL (Quality of Life) after the surgery can be facilitated.

[0114] Further, according to the first embodiment, the analysis function 542 detects an inducing matter that induces an abnormality on the basis of the medical information of the subject. The generation function 543 generates association information that associates the time at which the inducing matter is detected with the medical information taken at the time. Thus, the surgery assistance apparatus 5 of the first embodiment can easily confirm the inducing matter generated in the surgery.

[0115] Further, according to the first embodiment, the control function 541 displays timeline information that represents the association information in chronological order. Thus, the surgery assistance apparatus 5 of the first embodiment can provide information that is easily observable with respect to the time at which the matter associated with the abnormality is detected.

[0116] Further, according to the first embodiment, the control function 541 displays the medical image in the surgery collected at the time at which the matter associated with the abnormality is detected in association with the corresponding time in the timeline information. Thus, the surgery assistance apparatus 5 of the first embodiment, instead of prompting a real-time image, can prompt an image of the time at which the matter associated with the abnormality is detected, and for example, determination of the bleeding site can be performed more promptly.

[0117] Further, according to the first embodiment, the control function 541 further takes position information that represents the position of the medical instrument used in the surgery. The generation function 543 generates association information that associates the time at which the abnormality is detected, the medical information taken at the time, and the position information taken at the time. Thus, the surgery assistance apparatus 5 of the first embodiment can further prompt the position information, and determination of the bleeding site and the like can be performed more promptly.

[0118] Further, according to the first embodiment, the control function 541 takes position information that represents the position of the medical instrument operated in the subject in the surgery. Thus, the surgery assistance apparatus 5 of the first embodiment can take the position information of the medical instrument in the subject.

[0119] Further, according to the first embodiment, the control function 541 further takes the medical image of the subject. The control function 541 displays the position of the medical instrument corresponding to the time at which the matter associated with the abnormality is detected in the medical image of the subject. Thus, the surgery assistance apparatus 5 of the first embodiment can perform display of the position information that is more easily observable.

[0120] Further, according to the first embodiment, the control function 541 displays the current position of the medical instrument in the medical image of the subject. Thus, the surgery support apparatus 5 of the first embodiment can easily grasp the positional relationship between the current position of the medical instrument and the position at the time when the medical information that detects the matter associated with the abnormality is acquired.

[0121] Further, according to the first embodiment, the control function 541 displays information that enables identification of at least one of the degree of the matter associated with the abnormality and the detection method. Thus, the surgery support apparatus 5 of the first embodiment can easily grasp the degree of the matter associated with the abnormality and the detection method.

[0122] Further, according to the first embodiment, the analysis function 542 detects the matter associated with the abnormality based on the image feature amount in the image of the subject acquired in the surgery. Thus, the surgery support apparatus 5 of the first embodiment can easily detect the matter associated with the abnormality.

[0123] (Other Embodiments)

[0124] In addition, the first embodiment has been described so far, but the present application can be implemented in various different ways other than the above-described first embodiment.

[0125] In the above-described embodiments, the surgery support system 10 includes the surgery support apparatus 5, and the surgery support apparatus 5 performs various processes. However, the embodiments are not limited thereto, and various processes of the surgery support method according to the present application can be performed by any apparatus in the surgery support system 10 alone or by a plurality of apparatuses in the surgery support system 10 in a distributed manner.

[0126] For example, the analysis process that takes each medical information as an object can be performed by an apparatus that acquires each medical information. For example, the endoscope system 2 can perform analysis that takes the above-described endoscope image as an object. In this case, the processing circuit 23 performs detection of the matter associated with the abnormality based on the endoscope image by executing the above-described analysis function 542. Further, the ultrasonic diagnostic apparatus can perform detection of the matter associated with the abnormality based on the ultrasonic image, and the electrocardiograph or the like can perform detection of the matter associated with the abnormality based on the vital information.

[0127] Further, the generation process of the association information and various display processes related to the association information can be performed in the virtualized laparoscope image system 3.

[0128] Further, in the above-described embodiments, the case where the surgery assisting method of the present application is applied to laparoscopic surgery has been described. However, the embodiments are not limited thereto, and the surgery assisting method of the present application can be applied to various other surgeries.

[0129] Further, in the above-described embodiments, examples in which the acquisition unit, the detection unit, the generation unit, and the display control unit in the present specification are realized by a control function, an analysis function, a generation function, and a control function of a processing circuit, respectively, have been described, but the embodiments are not limited thereto. For example, the acquisition unit, the detection unit, the generation unit, and the display control unit in the present specification can realize the same functions only by hardware, only by software, or by a mixture of hardware and software, in addition to being realized by the control function, the analysis function, the generation function, and the control function described in the embodiments.

[0130] Further, the term "processor" used in the description of the above-described embodiments means, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)), and the like. Here, instead of saving a program in a storage circuit, the program can be directly incorporated in the circuit within the processor. In this case, the processor realizes the function by reading out and executing the program incorporated in the circuit. Further, each processor of the present embodiments is not limited to the case where a single circuit is configured for each processor, but a plurality of independent circuits can be combined to configure one processor and realize the function thereof.

[0131] Here, the program executed by the processor is provided in advance by a ROM (Read Only Memory), a storage circuit, or the like. In addition, the program can also be recorded in a non-transitory storage medium that can be read by a computer, such as a CD (Compact Disk)-ROM, an FD (Flexible Disk), a CD-R (Recordable), a DVD (Digital Versatile Disk), or the like, in a form that can be installed in these devices or in a form that can be executed. Further, the program can also be saved on a computer connected to a network such as the Internet, and provided or distributed by being downloaded via the network. For example, the program is constituted by modules including the above-described each processing function. As actual hardware, the CPU loads each module onto the main storage device by reading out the program from the storage medium such as a ROM, and thereby generates on the main storage device.

[0132] Further, in the above-described embodiments and modified examples, each constituent element of each device illustrated is a functional conceptual element, and does not necessarily need to be physically constituted as illustrated. That is, the specific manner of dispersing or merging of each device is not limited to the illustrated case, and all or a part of the functionality can be dispersed or merged in any unit, functionally or physically, according to various loads, usage conditions, and the like. Further, all or any part of each processing function performed in each device can be realized by a CPU and a program analyzed and executed by the CPU, or as hardware based on wired logic.

[0133] Further, in each processing described in the above-described embodiments and modified examples, all or a part of the processing described as processing performed automatically can be performed manually, or all or a part of the processing described as processing performed manually can be performed automatically by a known method. Further, except for the case specifically described, the processing steps, control steps, specific names, information containing various data, parameters, and the like shown in the above-described text and the drawings can be easily changed.

[0134] According to at least one of the embodiments described above, it is possible to easily confirm information related to an abnormality generated in surgery.

[0135] Several embodiments were described in connection with the present application, but these embodiments are not intended to limit the scope of the application. These embodiments can be implemented in various ways, and are not limited to the embodiments described herein. Various omissions, substitutions, and changes in form of the method and system described herein can be made without departing from the spirit of the application. The embodiments and their variations include and are meant to cover the application as well as equivalents thereof.

Claims

1. A surgical assistance system, comprising: The department obtains medical information about the patient being examined during surgery. The testing department, based on the medical information obtained from the aforementioned subjects, tests for matters related to abnormalities; and The generation department generates association information that links the time when the event associated with the above-mentioned anomaly was detected with the above-mentioned medical information obtained at that time.

2. The surgical assistance system according to claim 1, wherein, Based on the medical information of the subjects, the aforementioned testing department detected the factors that induced the aforementioned abnormalities. The aforementioned generation unit generates association information that establishes a connection between the time when the aforementioned triggering event was detected and the aforementioned medical information obtained at that time.

3. The surgical assistance system according to claim 1 or 2, wherein, It also includes a display control unit that displays timeline information that shows the aforementioned related information in chronological order.

4. The surgical assistance system according to claim 3, wherein, The aforementioned display control unit enables the display of medical images collected during surgery at the moment when an event related to the aforementioned anomaly is detected, in association with the corresponding moments in the aforementioned timeline information.

5. The surgical assistance system according to claim 1 or 2, wherein, The aforementioned acquisition department also acquires location information indicating the location of medical instruments used in the surgery. The aforementioned generation unit generates association information that establishes a connection between the time when the event associated with the aforementioned anomaly was detected, the aforementioned medical information obtained at that time, and the aforementioned location information obtained at that time.

6. The surgical assistance system according to claim 5, wherein, The aforementioned acquisition section acquires location information indicating the position of medical instruments manipulated within the subject's body during surgery.

7. The surgical assistance system according to claim 5, wherein, The aforementioned department also obtained medical images of the aforementioned subjects. The aforementioned surgical assistance system also includes a display control unit that displays the position of the medical instrument corresponding to the time when an event associated with the aforementioned abnormality is detected in a medical image of the subject.

8. The surgical assistance system according to claim 7, wherein, The aforementioned display control unit displays the current position of the aforementioned medical device in the medical image of the aforementioned subject.

9. The surgical assistance system according to claim 3, wherein, The aforementioned display control unit displays information that can identify the extent of the detected abnormality-related event and at least one of the detection methods.

10. The surgical assistance system according to claim 1 or 2, wherein, The aforementioned detection unit detects items associated with the aforementioned abnormalities based on image feature quantities in images of the subject obtained during surgery.

11. A surgical assistance method, comprising: Obtain medical information of the patient being examined during surgery; Based on the medical information obtained from the aforementioned subjects, the tests were conducted to identify items associated with the abnormalities. as well as Generate association information that links the time when the event associated with the above-mentioned anomaly was detected with the above-mentioned medical information obtained at that time.

Images