Surgery detail evaluation system, surgery detail evaluation method, and computer program

Abstract

To provide a surgery detail evaluation system, a surgery detail evaluation method, and a computer program that can objectively evaluate the details of actual surgery.SOLUTION: A surgery detail evaluation system for evaluating the details of surgery carried out by a surgeon comprises: acquisition means that acquires a surgery image being a picked-up image of the body of a patient in which the surgery is carried out by the surgeon; analysis means that analyzes body information indicating the state of the body in the surgery image; and evaluation means that evaluates the details of the surgery carried out by the surgeon based on the body information analyzed by the analysis means. The analysis means analyzes the body information including information on the anatomical structure of the body. The body information includes information on the state of the anatomical structure. The evaluation means evaluates whether the anatomical structure is in an appropriate state. The evaluation means evaluates the details of the surgery by using an AI learning model generated by being informed with good surgery and / or bad surgery or an evaluation value as correct answer data.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method and computer program for evaluating the content of surgery performed by a surgeon. [Background technology]

[0002] It has been known that the burden on a patient's body varies greatly depending on the surgeon's surgical technique and the way the surgery is performed. For this reason, surgeons are required to improve their surgical skills.

[0003] For example, Patent Document 1 proposes a surgical simulation using a computer simulator, which includes a first process in which the simulated motion calculation device loads and executes an educational scenario sequence from a storage device at the start of a surgical simulation, a second process in which the simulated motion calculation device saves in the storage device information about the simulated surgical tools and the simulation model data during the simulation calculated by the simulated motion calculation device, and a third process in which the simulated motion calculation device evaluates the simulated surgical tools and the simulation model data information saved in the storage device in accordance with evaluation items at the end of the simulation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-71418 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the surgery simulation of Patent Document 1 can evaluate the technique in the simulated surgery simulation, but cannot evaluate the content of the actual surgery. In order to improve surgical skills, it is essential to objectively evaluate the content of the actual surgery. By checking the objective evaluation of the content of the actual surgery, the surgeon can clarify the points for reflection and improvement, and by resolving these points for reflection and improvement, the surgical skill can be further improved.

[0006] The present invention has been made in consideration of the above points, and aims to provide a surgery content evaluation system, a surgery content evaluation method, and a computer program that are capable of objectively evaluating the content of an actual surgery. [Means for solving the problem]

[0007] (1) A surgical content evaluation system for evaluating the content of a surgical procedure performed by a surgeon, comprising: An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing, in the surgical image, body information indicating a state of the body and / or instrument information indicating a state of an instrument operated by a surgeon; A surgical content evaluation system comprising: an evaluation means for evaluating the content of the surgery performed by the surgeon based on the physical information and / or the instrument information analyzed by the analysis means.

[0008] In the invention (1), the surgery content evaluation system includes an acquisition means, an analysis means, and an evaluation means, and evaluates the content of surgery performed by a surgeon. The acquisition means acquires surgical images, which are images of the body of a patient undergoing surgery by a surgeon. The analysis means analyzes, in the surgical image, body information indicating the state of the body and / or instrument information indicating the state of an instrument being operated by the surgeon. The evaluation means evaluates the content of the surgery being performed by the surgeon based on the physical information and / or instrument information analyzed by the analysis means.

[0009] According to the invention of (1), the content of the surgery being performed by a surgeon is evaluated based on physical information indicating the state of the body and / or instrument information indicating the state of the instruments being operated by the surgeon in a surgical image taken of the patient's body while the surgeon is performing surgery, thereby making it possible to objectively evaluate the content of the actual surgery. Therefore, a surgery content evaluation system capable of objectively evaluating the content of an actual surgery can be provided.

[0010] (2) the analysis means analyzes a specific region in the surgical image; The surgical content evaluation system described in (1) is characterized in that the evaluation means evaluates that there has been a leakage of bodily fluids or organ damage if the specific area in the surgical image exceeds a predetermined threshold.

[0011] For example, if an organ is damaged, the blood area will expand in the surgical image if bleeding occurs, the bile area will expand if bile leaks out, and the cauterized area will expand in white or black if damage is caused by an electric scalpel, etc. Such outflow of bodily fluids and organ damage are important subjects for evaluating the contents of the surgery, considering the physical impact on the patient.

[0012] According to the invention of (2), for example, the area that spreads when there is leakage of bodily fluids or organ damage is defined as a specific area, and whether or not this specific area exceeds a predetermined threshold is determined. If the specific threshold is exceeded, it is evaluated that there has been leakage of bodily fluids or organ damage. In this way, it becomes possible to objectively evaluate leakage of bodily fluids and organ damage, which are important evaluation targets.

[0013] (3) The analysis means analyzes the appliance information including information about the appliance, The surgical content evaluation system according to (1) or (2), wherein the evaluation means evaluates the manner in which the instrument is operated during surgery based on information about the instrument.

[0014] Here, if there is unnecessary movement in the operation of the instrument during surgery (for example, the instrument does not move the shortest distance to the treatment site, etc.), the operation time will be longer and the physical burden on the patient will be greater. Also, if the operation of the instrument during surgery is inappropriate (for example, if the orientation of the instrument is inappropriate, etc.), there is a higher possibility that the instrument will cause organ damage. For this reason, the operation of the instrument is important as a subject for evaluating the content of the surgery, considering the reduction of the physical burden on the patient during surgery.

[0015] According to the invention of (3), by evaluating the manner in which an instrument is operated during surgery based on information about the instrument contained in the instrument information, it becomes possible to objectively evaluate the manner in which an instrument is operated, not just based on the surgeon's senses or the results of whether or not there was any organ damage, etc.

[0016] (4) The analysis means analyzing the physical information including information regarding the anatomy of the body; Analyzing the body information and / or the instrument information, including information regarding the position of the instrument relative to the anatomical structure; The surgical content evaluation system according to any one of (1) to (3), characterized in that the evaluation means evaluates the manner in which the instrument is operated relative to the anatomical structure during surgery based on information regarding the position of the instrument relative to the anatomical structure.

[0017] Here, for example, when resecting an affected part of an internal organ, it is necessary to separate the affected part from other organs. In this case, it is necessary to appropriately apply an electric scalpel, which is an example of an instrument, to the connective tissue, which is an example of an anatomical structure that connects the affected part to the other organs. If the electric scalpel is not appropriately positioned relative to the connective tissue, there is a risk that the electric scalpel will come into contact with other organs and cause damage to the other organs. For this reason, the position of the instrument relative to the anatomical structure is important as an evaluation subject for factors that may lead to errors in surgery.

[0018] In the invention of (4), by evaluating the manner in which instruments are operated relative to anatomical structures during surgery based on information regarding the position of the instruments relative to the anatomical structures, it becomes possible to objectively evaluate the manner in which instruments are operated relative to anatomical structures not only based on the surgeon's sense and the results such as no organ damage, but also on the possibility of mistakes leading to organ damage, etc.

[0019] (5) The analysis means analyzes each of the steps in the surgical image of a surgical operation including a plurality of steps, The method further includes a timer for measuring an inter-step time, which is a time from one step to the next step, The surgical content evaluation system according to any one of (1) to (4), wherein the evaluation means evaluates the surgical skill based on the step-to-step time measured by the timing means for a certain step.

[0020] Here, surgery consists of multiple steps, and since each step has different technical difficulty and physical impact on the patient, it is necessary to evaluate the content of the surgery for each step. For this reason, the inter-step time, which is the time from one step to the next, is an important evaluation subject when examining the content of the surgery for each step.

[0021] According to the invention (5), the surgical skill is evaluated based on the inter-step time, which is the time from one step to the next, so that it is possible to objectively evaluate not only the overall time of the surgery but also the surgeon's skill at each step.

[0022] (6) A method implemented by a surgical content evaluation system for evaluating the content of a surgical procedure performed by a surgeon, comprising: acquiring surgical images of a patient's body undergoing surgery by a surgeon; analyzing, in the surgical images, body information indicating a state of the body and / or instrument information indicating a state of an instrument operated by the surgeon; and evaluating the content of the surgery being performed by the surgeon based on the physical information and / or the instrument information.

[0023] (7) A surgical content evaluation system for evaluating the content of surgery performed by surgeons. An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing, in the surgical image, body information indicating a state of the body and / or instrument information indicating a state of an instrument being operated by a surgeon; A program that functions as an evaluation means for evaluating the content of the surgery being performed by the surgeon based on the physical information and / or the instrument information analyzed by the analysis means.

[0024] The inventions (6) and (7) have the same effects as the invention (1).

[0025] (8) A surgical content evaluation system for evaluating the content of a surgical procedure performed by a surgeon, comprising: An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing physical information indicating a physical condition in the surgical image; A surgery content evaluation system comprising: an evaluation means for evaluating the content of the surgery performed by the surgeon based on the physical information analyzed by the analysis means.

[0026] According to the invention of (8), the content of the surgery being performed by a surgeon is evaluated based on physical information indicating the condition of the body in a surgical image taken of the patient's body while the surgeon is performing surgery, thereby making it possible to objectively evaluate the content of the actual surgery. Therefore, a surgery content evaluation system capable of objectively evaluating the content of an actual surgery can be provided.

[0027] (9) The analysis means calculates a recognition degree indicating a degree of recognition of the physical information in the surgical image, The surgery content evaluation system according to (8), wherein the evaluation means evaluates the content of the surgery performed by the surgeon according to the degree of recognition.

[0028] According to the invention of (9), it is possible to evaluate the content of the surgery performed by the surgeon according to the recognition degree indicating the degree of recognition of the physical information in the surgical image.

[0029] (10) The analysis means Calculating a confidence level indicating the reliability of the analysis result of the physical information; The surgery content evaluation system according to (9), wherein the degree of recognition is calculated based on the degree of certainty.

[0030] According to the invention (10), the analysis means can calculate the degree of recognition based on the degree of certainty that indicates the accuracy of the analysis result of the physical information.

[0031] (11) A surgical content evaluation system as described in either (9) or (10), characterized in that the evaluation means evaluates the content of the surgery being performed by the surgeon in accordance with changes in the degree of awareness over time.

[0032] According to the invention of (11), it is possible to evaluate the content of the operation being performed by the surgeon according to the change in the degree of awareness over time.

[0033] (12) A surgical content evaluation system as described in any one of (8) to (11), characterized in that the evaluation means evaluates the difficulty of the surgery based on information regarding the anatomical structure of the body contained in the physical information.

[0034] According to the invention of (12), it is possible to evaluate the difficulty of surgery according to information on the anatomical structure of the body contained in the physical information.

[0035] (13) The analyzing means analyzes a trajectory of a tip position of an instrument operated by a surgeon, The surgical content evaluation system according to any one of (8) to (12), wherein the evaluation means evaluates the manner in which the instrument is operated during surgery based on the trajectory.

[0036] According to the invention of (13), it is possible to evaluate the manner in which an instrument is operated during surgery based on the trajectory of the tip position of the instrument being operated by the surgeon.

[0037] (14) The analyzing means analyzes a positional relationship between a point of action, which is a portion of an instrument operated by a surgeon that is in contact with an anatomical structure, and a portion of the anatomical structure where the instrument is in contact; The surgical content evaluation system according to any one of (8) to (13), wherein the evaluation means evaluates the manner in which the instruments are operated during the surgery based on the positional relationship.

[0038] According to the invention of (14), it is possible to evaluate the manner in which an instrument is operated during surgery based on the positional relationship between the point of action, which is the part of the instrument operated by the surgeon that contacts the anatomical structure, and the part of the anatomical structure where the instrument contacts.

[0039] (15) The acquisition means acquires a plurality of the surgical images continuously in time series, The surgical content evaluation system according to any one of (9) to (14), characterized in that the evaluation means identifies the surgical images in which the degree of recognition of information relating to a specific anatomical structure contained in the physical information is determined to be equal to or higher than a specific threshold value in the analysis means.

[0040] According to the invention of (15), it is possible to identify surgical images from a plurality of consecutive surgical images in a chronological order, the surgical images having a degree of recognition of information relating to a specific anatomical structure contained in the physical information that is determined to be equal to or higher than a specific threshold value.

[0041] (16) A method implemented by a surgical content evaluation system for evaluating the content of a surgical procedure performed by a surgeon, comprising: acquiring surgical images of a patient's body undergoing surgery by a surgeon; analyzing physical information indicative of a physical condition in the surgical image; A method for evaluating surgical content, comprising: a step of evaluating the content of the surgery performed by the surgeon based on the physical information analyzed by the analysis means.

[0042] (17) A surgical content evaluation system for evaluating the content of surgery performed by a surgeon. An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing physical information indicating a physical condition in the surgical image; A program that functions as an evaluation means for evaluating the content of the surgery performed by the surgeon based on the physical information analyzed by the analysis means.

[0043] The inventions (16) and (17) have the same effects as the invention (8). Effect of the Invention

[0044] According to the present invention, it is possible to provide a surgery content evaluation system, a surgery content evaluation method, and a computer program capable of objectively evaluating the content of an actual surgery. [Brief description of the drawings]

[0045] [Figure 1] 1 is a diagram illustrating an overview of a surgery content evaluation system according to an embodiment of the present invention. FIG. [Diagram 2] 1 is a diagram showing the functional configuration of a surgery content evaluation system according to an embodiment of the present invention. [Diagram 3] 13 is a diagram illustrating the processing of a timing means in the surgery content evaluation system according to an embodiment of the present invention. FIG. [Figure 4]4A and 4B are diagrams for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 4A is a surgery image in a state where no blood flow has occurred. Fig. 4B is a surgery image in a state where blood flow has occurred. [Diagram 5] 1 is a diagram illustrating the processing of an evaluation means in a surgery content evaluation system according to an embodiment of the present invention. FIG. [Figure 6] 6A and 6B are diagrams for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 6(a) shows a state in which the instrument is not operated appropriately with respect to the anatomical structure. Fig. 6(b) shows a state in which the instrument is operated appropriately with respect to the anatomical structure. [Figure 7] 7A and 7B are diagrams for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 7(a) shows a state in which the instrument is not operated appropriately with respect to the anatomical structure. Fig. 7(b) shows a state in which the instrument is operated appropriately with respect to the anatomical structure. [Figure 8] 1 is a diagram showing a surgical content evaluation process flow executed by the surgical content evaluation system according to an embodiment of the present invention. FIG. [Figure 9] 13 is a diagram showing an example of evaluation information generated by a surgery content evaluation system relating to an application example of an embodiment of the present invention. FIG. [Figure 10] 13 is a diagram illustrating the processing of an evaluation means in a surgery content evaluation system according to an application example of an embodiment of the present invention. FIG. [Figure 11] 13 is a diagram showing an example of evaluation information generated by a surgery content evaluation system relating to an application example of an embodiment of the present invention. FIG. [Figure 12] FIG. 13 is a diagram showing an example of evaluation criteria in the evaluation means of the surgery content evaluation system according to an application example of an embodiment of the present invention. [Figure 13] 13 is a diagram showing an example of evaluation information generated by a surgery content evaluation system relating to an application example of an embodiment of the present invention. FIG. [Figure 14] 13 is a diagram showing an example of evaluation information generated by a surgery content evaluation system relating to an application example of an embodiment of the present invention. FIG. [Figure 15]13 is a diagram showing an example of evaluation information generated by a surgery content evaluation system relating to an application example of an embodiment of the present invention. FIG. [Figure 16] 16A and 16B are diagrams for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 16A shows a surgery image of a thin patient. Fig. 16B shows a surgery image of an obese patient. [Figure 17] 1 is a diagram illustrating the processing of an evaluation means in a surgery content evaluation system according to an embodiment of the present invention. FIG. [Figure 18] FIG. 13 is a diagram showing an example of evaluation criteria in the evaluation means of the surgery content evaluation system according to an application example of an embodiment of the present invention. [Figure 19] 13A to 13C are diagrams illustrating processing in an image editing means of the surgery content evaluation system according to an application example of an embodiment of the present invention. [Figure 20] 13A to 13C are diagrams illustrating processing in an image editing means of the surgery content evaluation system according to an application example of an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] Hereinafter, a detailed description will be given of an embodiment of the present invention with reference to the accompanying drawings. In the following drawings, the same elements are designated by the same numbers or symbols throughout the description of the embodiment.

[0047] (Basic concept / basic configuration) FIG. 1 is a diagram for explaining an overview of a surgery content evaluation system according to an embodiment of the present invention. The surgery content evaluation system 1 evaluates the content of the surgery performed by a surgeon.

[0048] The surgery content evaluation system 1 acquires surgery images, which are images of the state of surgery by a surgeon at a medical institution (for example, an institution such as a hospital where surgery is performed by a surgeon). The surgery images are images of the body of a patient undergoing surgery and the instruments (for example, energy devices such as forceps, electric scissors, electric scalpels, ultrasonic coagulation and incision devices, etc.) operated by a surgeon or assistant. The surgery images may be videos or, for example, still images captured continuously in a time series. In this embodiment, the surgery images are images captured by a camera inserted from an access port during endoscopic surgery, but are not limited to this. The surgery images may be images captured by a surgery support robot or images captured from above the patient, for example, as long as they capture the body of a patient undergoing surgery and the instruments operated by a surgeon or assistant, etc.

[0049] The surgery content evaluation system 1 analyzes, in a surgery image, physical information indicating the state of the body and / or instrument information indicating the state of an instrument operated by a surgeon, using AI (Artificial Intelligence).

[0050] Then, the surgery content evaluation system 1 uses AI to evaluate the content of the surgery performed by the surgeon based on the physical information and / or instrument information analyzed by the analysis means, and transmits evaluation information indicating the evaluation results to, for example, a terminal at a medical institution where the surgeon can check the evaluation results.

[0051] According to this type of surgery content evaluation system 1, the content of the surgery being performed by the surgeon is evaluated using AI based on physical information indicating the condition of the body and / or instrument information indicating the state of the instruments being operated by the surgeon in surgical images taken of the patient's body while the surgeon is performing surgery, making it possible to objectively evaluate the content of the actual surgery. Therefore, it becomes possible to objectively evaluate the content of the actual surgery.

[0052] (Functional configuration) FIG. 2 is a diagram showing the functional configuration of the surgery content evaluation system according to the embodiment of the present invention. The surgical content evaluation system 1 is connected to multiple medical institution terminals 2 via a network, and comprises an acquisition means 11, an analysis means 12, a timing means 13, an evaluation means 14, a transmission means 15, and a memory means 20.

[0053] In this embodiment, the medical institution terminal 2 may be a terminal managed by the medical institution, or it may be a terminal installed in the medical institution and managed by the operator of the surgery content evaluation system 1, and capable of transmitting surgical images captured by an imaging means (e.g., a camera) to the surgery content evaluation system 1 and receiving evaluation information from the surgery content evaluation system 1.

[0054] The acquiring means 11 acquires surgical images, which are images of the body of a patient undergoing surgery by a surgeon. The acquiring means 11 may receive surgical images from the medical institution terminal 2, may receive surgical images from another device (e.g., a server that accumulates surgical images), or may read surgical images stored in advance in the storage means 20 from the storage means 20. The acquiring means 11 may also receive voice data during surgery corresponding to the surgical images. In addition to the analysis results of the surgical images, such voice data may be used to evaluate, for example, the state of communication between the surgeon and assistants, the surgeon's mental state such as the surgeon's initiative and impatience, and the surgeon's ability to lead assistants.

[0055] In addition, when instrument information indicating the state of an instrument being operated by a surgeon (e.g., position information detected over the timeline of a surgery) is detected by various sensors (e.g., a gyro sensor) equipped on the instrument, the acquisition means 11 may acquire such instrument information, associate it with the corresponding surgical image, and store it in the memory means 20.

[0056] The storage means 20 stores surgical images, learned information for analysis used in analysis by the analysis means 12 described below, and learned information for evaluation used in evaluation by the evaluation means 14 described below.

[0057] The trained information for analysis is generated by having an AI learn from images of multiple surgical procedures that have been performed in the past, and includes a step recognition model for recognizing the multiple steps that make up a surgery, a specific area analysis model for analyzing specific areas in a surgical image (e.g., areas identified as bodily fluids (e.g., red areas formed by outflow of blood, yellow areas formed by outflow of bile, etc.), areas of burned organs, etc.), an anatomical structure identification model for identifying the patient's anatomical structure (organs, blood vessels, fat, connective tissue, etc.), and an instrument identification model for identifying instruments used by surgeons (operators, assistants, etc.) (e.g., forceps, energy devices such as electric scissors and electric scalpels, etc.). It is desirable that the detection threshold for the trained information for analysis be changeable.

[0058] The trained information for evaluation is generated by having the AI ​​learn from multiple surgical images performed in the past, and includes a specific area threshold model that indicates a predetermined threshold for the size of a specific area in a surgical image, and an anatomical structure / instrument state judgment model for judging the state of the anatomical structure for each of the multiple steps that make up the surgery and the state of the instruments (instrument type, instrument position, instrument orientation, etc.). It is desirable that the trained information for analysis allows the detection threshold at the time of judgment to be changed.

[0059] The learned information for analysis and the learned information for evaluation are generated by any learning method such as known machine learning, deep learning, reinforcement learning, etc., and are stored in advance in the storage means 20. In addition, the learned information for analysis and the learned information for evaluation may be generated by the surgery content evaluation system 1, or may be generated by an external device and stored in the storage means 20.

[0060] The analysis means 12 analyzes, in the surgical image acquired by the acquisition means 11, body information indicating the state of the body and / or instrument information indicating the state of the instrument being operated by the surgeon.

[0061] Specifically, the analysis means 12 compares the surgical image acquired by the acquisition means 11 with the step recognition model stored in the storage means 20, and analyzes which of the multiple steps constituting the surgery the state shown in the surgical image is. The analysis means 12 analyzes which step each frame in the video or each of multiple consecutive still images acquired in time series (hereinafter also referred to as "frames, etc.") is. The analysis means 12 may adjust the fps (frames per second) of the video before performing the analysis. For example, by reducing the fps of the video, it is possible to reduce the processing load and processing time of the surgical content evaluation system 1.

[0062] Furthermore, the analysis means 12 compares the surgical image acquired by the acquisition means 11 with the specific region analysis model stored in the storage means 20, and analyzes the specific region in the surgical image. The analysis means 12 analyzes the specific region for each frame, etc. Here, in this embodiment, the specific region is a region created by the outflow of body fluids (e.g., blood, bile, etc.) in the surgical image, and is analyzed by the analysis means 12 to be a specific region based on characteristics such as color (e.g., red which is the color of blood, yellow which is the color of bile, white or black when an organ is burned, etc.) and edges of the region (body fluids have a higher viscosity than water and have distinctive edges).

[0063] Furthermore, the analysis means 12 compares the surgical image acquired by the acquisition means 11 with the anatomical structure identification model stored in the storage means 20, and analyzes the anatomical structure captured in the surgical image, which is an example of body information. The analysis means 12 analyzes the anatomical structure for each frame, etc.

[0064] Furthermore, the analysis means 12 compares the surgical image acquired by the acquisition means 11 with the instrument identification model stored in the storage means 20, and analyzes the state of the instrument captured in the surgical image, which is an example of instrument information (for example, the type of instrument and the orientation of the instrument). The analysis means 12 analyzes the state of the instrument captured for each frame, etc.

[0065] The timing means 13 measures an inter-step time, which is the time from one step to the next step, among the multiple steps constituting the surgery captured in the surgery image acquired by the acquisition means 11.

[0066] FIG. 3 is a diagram illustrating the processing of the timing means in the surgery content evaluation system according to the embodiment of the present invention. The example shown in FIG. 3 is a surgical image of a cholecystectomy, and from the top of FIG. 3, it shows step X (exposure of the gallbladder), the next step X+1 (resection of the cystic duct), and the next step X+2 (removal of the gallbladder).

[0067] The analysis means 12 uses the step recognition model to identify frames or still images in which step X, step X+1, and step X+2 are captured, from among a plurality of consecutive frames in a moving image or from among a plurality of consecutive still images acquired in time series.

[0068] The timing means 13 measures, for example, an inter-step time TX+1 which is the time difference between a frame of step X specified by the analysis means 12 and a frame of step X+1.

[0069] Returning to FIG. 2, the evaluation means 14 evaluates the content of the surgery being performed by the surgeon based on the physical information and / or instrument information analyzed by the analysis means 12.

[0070] The evaluation means 14 evaluates the skill of the surgery based on the inter-step time between certain steps measured by the timing means 13. Specifically, the storage means 20 stores a judgment table in which a reference time for the surgery is associated with each inter-step time, and the evaluation means 14 compares the reference time for the surgery with the inter-step time of each step measured by the timing means 13, and for example, if the inter-step time is shorter than the reference time, the evaluation value for the surgery skill is set to a higher value, and if the inter-step time is longer than the reference time, the evaluation value for the surgery skill is set to a lower value.

[0071] Furthermore, the evaluation means 14 evaluates that an outflow of bodily fluids has occurred when a specific color component in the surgical image has increased. Fig. 4 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to an embodiment of the present invention. Fig. 4(a) is a surgical image in a state where no bleeding has occurred. Fig. 4(b) is a surgical image in a state where bleeding has occurred. In the example shown in Fig. 4(b), the red component, which is an example of a specific color, has increased throughout the image, making it a darker image than Fig. 4(a).

[0072] The evaluation means 14 compares the analysis result of the specific region of the surgical image by the analysis means 12 with a specific region threshold model indicating a predetermined threshold for the size of the specific region stored in the storage means 20, judges whether or not the size of the specific region in the surgical image exceeds the predetermined threshold, and evaluates that there has been outflow of body fluids or organ damage if the size of the specific region exceeds the predetermined threshold. Furthermore, the evaluation means 14 may evaluate the outflow of body fluids or organ damage in stages according to the size of the specific region, regardless of whether the size of the specific region is the predetermined threshold. For example, the evaluation means 14 may assign a lower evaluation value for outflow of body fluids or organ damage to a larger specific region.

[0073] Furthermore, the evaluation means 14 may determine whether or not the size of the specific region exceeds a predetermined threshold for each frame, etc., and identify the time of the frame, etc., from which the size of the specific region exceeded the predetermined threshold (for example, if the surgical image is a video, the elapsed time from the start of the video or the elapsed time from the start of the surgery). By identifying such a time, it becomes easy to check the surgical images around this time, and to identify the operation of the instrument that caused bleeding, etc., or the position of bleeding, etc., in the organ.

[0074] Furthermore, the evaluation means 14 evaluates the manner in which the instrument is operated during surgery based on the information about the instrument contained in the instrument information analyzed by the analysis means 12.

[0075] Specifically, the evaluation means 14 compares the analysis results of the instrument state for each frame, etc. by the analysis means 12 with the anatomical structure / instrument state judgment model stored in the memory means 20, and evaluates the operation manner of the instrument during surgery.

[0076] Here, the anatomical structure / instrument state determination model specifies the standard type of instrument, standard direction, standard route, standard speed, etc. for each of the multiple steps constituting the surgery. The evaluation means 14 compares such an anatomical structure / instrument state determination model with the analysis results by the analysis means 12 to evaluate, for example, whether an appropriate instrument has been selected, whether an instrument has been used appropriately, whether ligation has been performed properly, whether suturing has been performed appropriately, whether blood vessels have been properly treated, etc.

[0077] Fig. 5 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. In the example shown in Fig. 5, forceps D1 being operated by the surgeon's left hand (non-dominant hand) and an energy device D2 being operated by the surgeon's right hand are imaged.

[0078] The analysis means 12 analyzes the trajectory of the tool (energy device D2 in the example shown in Fig. 5) that changes over time by analyzing the state of the tool captured in each frame. In Fig. 5, a dot is typically added to the tip position of the tool in each frame, and the trajectory of the tool is indicated by a black line connecting these dots.

[0079] The evaluation means 14 evaluates whether the trajectory analyzed by the analysis means 12 is straight or not, and evaluates whether the instrument is used appropriately based on the total distance traveled along the trajectory. The longer the total distance of the trajectory of such an instrument is than a reference value (a reference value defined in the anatomical structure / instrument state judgment model), the lower the evaluation value of whether the instrument is used appropriately.

[0080] The evaluation means 14 compares the trajectory analyzed by the analysis means 12 with the anatomical structure / instrument state judgment model, and can evaluate the presence or absence of tremors during the operation of the instrument, the constancy of the moving speed (low acceleration), whether the instrument moves the shortest distance from when it is inserted to the position where it is used, whether the instrument is able to remain still during use if it is an energy device, whether the use of the instrument is well-balanced, whether blunt dissection and sharp dissection are appropriately used, whether the dissection line is straight, etc. Such evaluation can be used to evaluate the mental state of the surgeon (decline in concentration) due to the occurrence of a change over time in the nature of the trajectory during the course of the surgery.

[0081] Furthermore, the evaluation means 14 evaluates the manner in which the instrument is operated relative to the anatomical structure during surgery, based on the physical information including information on the anatomical structure of the body analyzed by the analysis means 12 and information on the position of the instrument relative to the anatomical structure included in the instrument information. Note that the information on the position of the instrument relative to the anatomical structure may be included not only in the instrument information, but also in the physical information and / or the instrument information.

[0082] Specifically, the evaluation means 14 compares the analysis results of the anatomical structure and instrument state for each frame, etc., by the analysis means 12 with the anatomical structure / instrument state judgment model stored in the memory means 20, and evaluates the manner in which the instrument is operated relative to the anatomical structure during surgery.

[0083] Here, the anatomical structure / instrument state judgment model prescribes, for each of the multiple steps constituting the surgery, the appropriate exposed area of ​​connective tissue, the appropriate tension on blood vessels and connective tissue, the appropriate situation around the blood vessel to be ligated (for example, a situation in which no unnecessary organs are wrapped around the blood vessel to be ligated), etc. The evaluation means 14 compares such an anatomical structure / instrument state judgment model with the analysis results by the analysis means 12, thereby evaluating whether the instrument is being operated at an appropriate position and with an appropriate amount of force with respect to the anatomical structure.

[0084] In addition to the above, the anatomical structure / instrument state determination model specifies the appropriate shape and color of each organ. The evaluation means 14 compares such an anatomical structure / instrument state determination model with the analysis results by the analysis means 12, and if there are differences, evaluates that an inappropriate instrument has been operated on the organ. For example, if the color of the organ is different from the appropriate color, it is evaluated as a burn mark caused by an instrument (such as an electric scalpel), and that an inappropriate instrument has been operated.

[0085] Fig. 6 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 6(a) shows a state in which the instrument is not operated appropriately with respect to the anatomical structure. Fig. 6(b) shows a state in which the instrument is operated appropriately with respect to the anatomical structure.

[0086] In the example shown in Figure 6(a), the instrument (forceps D4 in the example shown in Figure 6(a)) being operated by the surgeon's left hand (non-dominant hand) is not applying appropriate tension to the connective tissue, which is an example of an anatomical structure (the area circled in the figure), and the part to be cut by the energy device D4 is close to other organs. If the energy device D4 is brought close in this state, there is a risk of damaging the other organs.

[0087] When the analysis means 12 analyzes the state illustrated in FIG. 6(a), the evaluation means 14 reduces the evaluation value for the manner in which the instrument is operated relative to the anatomical structure during surgery.

[0088] In the example shown in Figure 6(b), appropriate tension is applied to the connective tissue, which is an example of an anatomical structure (the area surrounded by a white circle in the figure), by the forceps D4, and the part to be cut by the energy device D4 is separated from other organs.

[0089] When the state illustrated in Fig. 6(b) is analyzed by the analysis means 12, the evaluation means 14 increases the evaluation value of the operation mode of the instrument against the anatomical structure in surgery. The evaluation means 14 can increase the evaluation value of the operation mode of the instrument against the anatomical structure in surgery as the operation mode approaches the appropriate tension for the connective tissue defined in the anatomical structure-instrument state determination model.

[0090] Fig. 7 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 7(a) shows a state in which the instrument is not operated appropriately with respect to the anatomical structure. Fig. 7(b) shows a state in which the instrument is operated appropriately with respect to the anatomical structure.

[0091] In the example shown in FIG. 7(a), the fat, which is an example of an anatomical structure, is not properly grasped by the instrument (forceps D5 in the example shown in FIG. 7(a)) operated by the assistant, causing slack (white line in the upper center of FIG. 7(a)). In this state, the fat cannot be properly cut with the energy device.

[0092] When the analysis means 12 analyzes the state illustrated in FIG. 7(a), the evaluation means 14 lowers the evaluation values ​​for the manner of operation of the instruments with respect to the anatomical structures during surgery and the cooperation with the assistant.

[0093] In the example shown in Figure 7(b), the upper part of the fat, which is an example of an anatomical structure, is held in a straight line by forceps D5 operated by the assistant, and the lower part is grasped by forceps D6 operated by the surgeon's left hand (non-dominant hand), applying appropriate tension to the fat, and an appropriate triangular shape (white line part in Figure 7(b)) is formed in the fat as the cutting area.

[0094] When the state illustrated in Fig. 7(b) is analyzed by the analysis means 12, the evaluation means 14 increases the evaluation value of the operation mode of the instrument against the anatomical structure during surgery and the cooperation with the assistant. The closer the shape of the fat cutting position in the analysis result is to the shape of the fat cutting position defined in the anatomical structure / instrument state judgment model (for example, a triangular shape stretched with appropriate tension like a sail of a yacht), the higher the evaluation value of the operation mode of the instrument against the anatomical structure during surgery and the cooperation with the assistant can be.

[0095] The evaluation means 14 associates the evaluation value determined based on the analysis results by the analysis means 12 with each item for evaluating the content of the surgery, and generates evaluation information that can be displayed on the medical institution terminal 2 in any form, such as an arranged table, graph, or radar format.

[0096] Furthermore, the evaluation means 14 may extract proposal information that is a proposal for encouraging technical improvement according to the evaluation value, and an ideal procedure image for each evaluation item according to the evaluation value. In this case, the surgery content evaluation system 1 stores, for example, in the storage means 20, proposal information and procedure images corresponding to the evaluation value for each evaluation item, and the evaluation means 14 refers to the storage means 20 and extracts proposal information and procedure images corresponding to the determined evaluation value. Furthermore, the evaluation means 14 may determine an evaluation value relative to the ideal procedure image for the analysis result.

[0097] The surgery content evaluation system 1 may also include an image editing means for generating a digest version image of the surgery image acquired by the acquisition means 11. The image editing means extracts, for example, a video of a predetermined time including frames of each step in the surgery specified by the analysis means 12 for each step from the surgery image acquired by the acquisition means 11, and synthesizes these to generate a digest version image. The image editing means may also extract, for example, a video of a predetermined time including frames evaluated by the evaluation means 14 as having an evaluation value lower than a preset reference value from the surgery image acquired by the acquisition means 11, and synthesizes these to generate a digest version image.

[0098] Returning to FIG. 2, the transmission means 15 transmits the evaluation information generated by the evaluation means 14 to the medical institution terminal 2. The transmission means 15 may also transmit the proposed information, ideal procedure image, or digest image extracted by the evaluation means 14 to the medical institution terminal 2. The transmission means 15 may also transmit to the medical institution terminal 2 an evaluator (e.g., a surgeon who can evaluate the content of the surgery in the digest image) who is different from the surgeon of the surgery in the digest image. In this case, the surgery content evaluation system 1 may receive human evaluation information indicating the evaluator's evaluation based on the transmitted digest image from the medical institution terminal 2. The transmission means 15 may then transmit the human evaluation information to the medical institution terminal 2 together with the evaluation information generated by the evaluation means 14.

[0099] The above-mentioned functional configuration of the present system is merely an example, and one functional block (database and functional processing unit) may be divided, or multiple functional blocks may be combined into one functional block. Each functional processing unit is realized by a computer program executed by a CPU (Central Processing Unit) built into the device or terminal, which reads out a computer program (e.g., a core software or an application that causes the CPU to execute the above-mentioned various processes) stored in a storage device (storage unit) such as a ROM (Read Only Memory), a flash memory, an SSD (Solid State Drive), or a hard disk. Each functional processing unit may be configured with an FPGA (Field-Programmable Gate Array). That is, each functional processing unit is realized by the computer program reading and writing necessary data such as a table from a database (DB; Data Base) stored in the storage device or a storage area on the memory, and, in some cases, controlling related hardware (e.g., a GPU (Graphics Processing Unit), an input / output device, a display device, a communication interface device). Furthermore, the database (DB) in the embodiment of the present invention may be a commercial database, but may also refer to a simple collection of tables and files, and the internal structure of the database itself is not an issue.

[0100] (Processing flow) FIG. 8 is a diagram showing a flow of a surgery content evaluation process executed by the surgery content evaluation system according to the embodiment of the present invention.

[0101] In step S1, the acquisition means 11 acquires from the medical institution terminal 2 or the storage means 20 surgical images, which are images captured of a patient's body undergoing surgery by a surgeon.

[0102] In step S2, the analysis means 12 analyzes physical information indicating the state of the body and / or instrument information indicating the state of an instrument operated by a surgeon in the surgical image acquired by the acquisition means 11 in step S1. Also, in this step, the analysis means 12 analyzes color components of specific colors (e.g., red, which is the color of blood, yellow, which is the color of bile, etc.) in the surgical image acquired by the acquisition means 11 in step S1.

[0103] In step S3, the timing means 13 measures an inter-step time, which is the time from one step to the next step, among the multiple steps constituting the surgery captured in the surgery image acquired by the acquisition means 11 in step S1.

[0104] In step S4, the evaluation means 14 evaluates the skill of the surgery based on the step-to-step time measured by the timing means 13 between certain steps. Also, in this step, the evaluation means 14 judges whether or not the component of the specific color has increased in the surgery image based on the analysis result of the color component of the specific color of the surgery image by the analysis means 12 in step S2, and evaluates that there has been an outflow of body fluids when it is judged that the component of the specific color has increased. Also, in this step, the evaluation means 14 evaluates the operation mode of the instrument in the surgery based on the information on the instrument included in the instrument information analyzed by the analysis means 12 in step S2. Also, in this step, the evaluation means 14 evaluates the operation mode of the instrument with respect to the anatomical structure in the surgery based on the body information including the information on the anatomical structure of the body and the information on the position of the instrument with respect to the anatomical structure included in the instrument information analyzed by the analysis means 12 in step S2. And, in this step, the evaluation means 14 generates evaluation information that can display the evaluation on the medical institution terminal 2.

[0105] In step S5, the transmission means 15 transmits to the medical institution terminal 2 the evaluation information generated by the evaluation means 14 in step S4.

[0106] According to such a surgery content evaluation system 1, the content of the surgery being performed by a surgeon is evaluated based on physical information indicating the state of the body and / or instrument information indicating the state of the instruments being operated by the surgeon in a surgical image taken of the patient's body while the surgeon is performing surgery, thereby making it possible to objectively evaluate the content of the actual surgery. Therefore, a surgery content evaluation system capable of objectively evaluating the content of an actual surgery can be provided.

[0107] Furthermore, according to the surgical content evaluation system 1, if a specific color component in a surgical image increases, it is evaluated as indicating that bodily fluids have leaked out, making it possible to objectively evaluate organ damage, which is an important evaluation target.

[0108] Furthermore, according to the surgery content evaluation system 1, by evaluating the manner in which instruments are operated during surgery based on the information about the instruments contained in the instrument information, it becomes possible to objectively evaluate the manner in which instruments are operated, rather than merely based on the surgeon's senses or results such as the absence of organ damage.

[0109] In addition, according to the surgical content evaluation system 1, by evaluating the manner in which instruments are operated relative to anatomical structures during surgery based on information regarding the position of the instruments relative to the anatomical structures, it is possible to objectively evaluate not only the surgeon's sense and the results such as no organ damage, but also the possibility of the instrument's operation relative to the anatomical structures leading to mistakes such as organ damage.

[0110] Furthermore, according to the surgery content evaluation system 1, the surgical skill is evaluated based on the inter-step time, which is the time from one step to the next, so that it is possible to objectively evaluate not only the overall time of the surgery but also the surgeon's skill at each step.

[0111] [Application example] Next, application examples of the embodiment of the present invention will be described. In the following description, the same components as those in this embodiment are denoted by the same reference numerals, and the description thereof will be omitted or simplified. In addition to the configuration of this embodiment, the surgical content evaluation system 1 of the application example mainly has an analysis means 12 (see Figure 2) that calculates a degree of recognition indicating the degree of recognition of physical information, and an evaluation means 14 (see Figure 2) that evaluates the content of the surgery performed by the surgeon based on this degree of recognition.

[0112] In the application example, "degree of recognition" refers to the number of pixels or area where the degree of certainty indicating the accuracy of the analysis results by the analysis means 12 for the surgical image is above a certain threshold, the total amount of certainty, or the speed or acceleration of their increase or decrease, or the degree of continuity (absence of interruptions, etc.) of structures such as organs.

[0113] FIG. 9 is a diagram showing an example of evaluation information generated by the surgery content evaluation system according to an application example of the embodiment of the present invention. The example shown in FIG. 9 is an example of evaluation information that can be generated by the evaluation means 14 and displayed on the medical institution terminal 2, and the evaluation value determined according to the analysis result by the analysis means 12 is shown in a graph format.

[0114] For example, in the example shown in FIG. 9, the graph on the right side of the figure shows the data transition during surgery, with the horizontal axis being the elapsed time during surgery (in the example shown in FIG. 9, the start of surgery is set to 0 and the end of surgery is set to 100), and the vertical axis being the degree of each item. In this graph, graphs showing the transition of items such as confidence level, device movement amount, blood loss amount, surgery time between steps, and relative positions of instruments and structures are arranged on the same time axis. With such evaluation information, it becomes possible to evaluate the contents of the surgery from the relationship between the probability (confidence level) indicating the likelihood of the analysis result calculated when the analysis means 12 (AI) performs the analysis and the analysis result by the analysis means 12 (device movement amount, blood loss amount, surgery time between steps, relative positions of instruments and structures, etc.).

[0115] The evaluation information may also include information showing the progression of events during surgery (the elapsed time of each event) in a pie chart, as shown on the left side of Figure 9, information showing the usage rate of the AI ​​model used by the analysis means 12 and the evaluation means 14, and information showing the surgery time.

[0116] (Evaluation based on the degree of recognition of anatomical structures) The analysis means 12 calculates a degree of recognition indicating the degree of recognition of physical information (e.g., anatomical structure) in the surgical image. In detail, the analysis means 12 calculates a certainty indicating the likelihood of the analysis result of the physical information, and calculates the degree of recognition based on the certainty. Then, the evaluation means 14 evaluates the content of the surgery being performed by the surgeon according to this degree of awareness.

[0117] Fig. 10 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to an application example of an embodiment of the present invention. Fig. 10 shows surgical images of a step of dissecting fatty tissue and exposing nerves. In the surgical image before fatty tissue dissection (upper figure), the nerve tissue is not exposed. Then, as the surgery progresses, in the surgical image after fatty tissue dissection (lower figure), the nerve tissue is exposed, and it can be confirmed that the nerve tissue is not interrupted but is continuous.

[0118] In the surgical image after adipose tissue dissection in Fig. 10 (the lower diagram), the left side is the original image captured by the camera, and the right side is this original image in which the nerve tissue portion is identified and colored by the analysis by the analysis means 12. The analysis means 12 analyzes the portion of the surgical image in which the nerve tissue is displayed, and colors the portion of the analysis result whose confidence exceeds a predetermined threshold value.

[0119] For example, if the adipose tissue is not sufficiently removed, or if there is a cauterization scar where the nervous tissue has been burned by an electric scalpel or the like, the exposed area of ​​the nervous tissue (the area where the confidence exceeds a certain threshold) will be smaller than the example shown in the surgical image after adipose tissue removal in Figure 10. Such surgical content is rated low. In such a case, the degree of recognition of the nervous tissue, which is an example of physical information calculated by the analysis means 12, decreases compared to a case in which the nervous tissue is sufficiently exposed and not interrupted. Then, when the degree of recognition decreases, the evaluation means 14 accordingly lowers the content of the surgery performed by the surgeon.

[0120] The analysis means 12 may calculate the exposed area of ​​the target organ, which is an example of physical information (the total number of pixels whose certainty exceeds a threshold), and use this as the degree of recognition, or may calculate the total amount of certainty of the target organ and use this as the degree of recognition, or may calculate the degree of recognition from the continuity of the target organ, or may calculate the degree of recognition by combining these.

[0121] In addition to nerves, important organs that should be preserved, such as blood vessels, ureters, vas deferens, and pancreas, should be exposed cleanly (separated from surrounding tissue and without interruption), which is one of the criteria for evaluating a successful surgery. Therefore, if the rate at which the exposed area of ​​the preserved organ increases is high, it indicates that the organ was exposed smoothly, and is evaluated as good. Based on such evaluation criteria, the evaluation means 14 may evaluate the content of the surgery performed by the surgeon according to the change over time in the degree of awareness. Specifically, the evaluation means 14 may evaluate the content of the surgery according to the speed or acceleration of the increase or decrease in the degree of awareness. For example, the evaluation means 14 may evaluate that the surgery is progressing quickly if the speed of increase or decrease in the degree of awareness is fast, and may evaluate that the surgery is progressing stably if the acceleration of increase or decrease in the degree of awareness is constant.

[0122] In addition, the evaluation means 14 may generate a learning model that uses AI to learn good evaluation examples (examples evaluated as good surgeries) and bad evaluation examples (examples evaluated as bad surgeries) that reflect the analysis results of the analysis means 12, and use this learning model to make evaluations. Specifically, a good surgery is one that has no surgical complications, is performed by an experienced doctor, is judged to be a good surgery by an experienced doctor, etc. A bad surgery is one that has surgical complications, is performed by an inexperienced doctor, is judged to be a bad surgery by an experienced doctor, etc. The evaluation means 14 assigns these as correct answer data, learns the evaluation using AI, and gives a good or bad evaluation.

[0123] FIG. 11 is a diagram showing an example of evaluation information generated by the surgery content evaluation system according to an application example of the embodiment of the present invention. Fig. 11 is an example of evaluation information that is generated by the evaluation means 14 and can be displayed on the medical institution terminal 2, and the evaluation value determined according to the analysis result by the analysis means 12 is shown in a graph form. The example shown in Fig. 11 shows the transition of the confidence level of a resected organ in an operation to resect the organ, and shows examples with higher and lower evaluation points by the evaluation means 14 overlapping each other.

[0124] As shown in FIG. 11, the evaluation means 14 assigns a higher evaluation point the higher the maximum confidence level for the analysis results of the resected organ (good exposure, little camera shake or bleeding) and the longer the period during which the confidence level is below 0.1 (the resection is performed appropriately, so there is no or very little part analyzed as the resected organ).

[0125] The evaluation index for the evaluation in the evaluation means 14 may not be the degree of certainty, but may be the amount of pixels with a certain degree of certainty or more, the rate of increase or decrease of the amount of pixels (fast processing time), or the rate of increase or decrease may be constant (no fluctuation in the procedure). In this way, the evaluation means 14 may evaluate the contents of the surgery performed by the surgeon according to the degree of recognition based on other factors, not limited to the degree of recognition based on the degree of certainty.

[0126] FIG. 12 is a diagram showing an example of evaluation criteria in the evaluation means of the surgery content evaluation system according to an application example of the embodiment of the present invention. In surgery, there are a mixture of organs to be removed and organs to be preserved. Figure 12 shows the change over time in the degree of recognition of organs to be removed and organs to be preserved in such a case.

[0127] The evaluation means 14 recognizes a surgical scene (recognizes an event of resecting a resected organ from a preserved organ) based on the analysis results of the anatomical structure and the state of the instruments (such as the type of instrument being used) by the analysis means 12. The evaluation means 14 evaluates the content of the surgery being performed by the surgeon based on the change in the recognition level of the resected organ (e.g., connective tissue, etc.) and the change in the recognition level of the preserved organ (nerves) after the event recognition. For example, if the recognition level of the resected organ decreases and the recognition level of the preserved organ increases after the event recognition, the evaluation means 14 assigns a higher evaluation point.

[0128] (Evaluation of Instrument Recognition) FIG. 13 is a diagram showing an example of evaluation information generated by the surgery content evaluation system according to an application example of the embodiment of the present invention. Fig. 13 is an example of evaluation information that is generated by the evaluation means 14 and can be displayed on the medical institution terminal 2, and shows in a graph form the amount of movement calculated according to the analysis results by the analysis means 12. The example shown in Fig. 13 shows the transition of the amount of movement in the trajectory of the tip position of the instrument operated by the surgeon, and shows examples with higher and lower evaluation points by the evaluation means 14 superimposed on each other.

[0129] As shown in FIG. 13, the evaluation means 14 estimates that the surgery is more efficient the less the movement of the instrument, and therefore gives a higher evaluation point.

[0130] Moreover, the analysis means 12 analyzes the trajectory of the tip position of the instrument (see FIG. 5) in the same manner as in this embodiment. Moreover, the evaluation means 14 evaluates whether the trajectory analyzed by the analysis means 12 is straight or there is no deviation, and evaluates whether the instrument is being used appropriately based on the total distance operated on the trajectory in the same manner as in this embodiment. The total distance and deviation in the trajectory of the tip position of the instrument become larger in proportion to the unnecessary movement in the operation of the instrument, and therefore serve as an evaluation of the surgeon's technique. Moreover, when the instrument is a grasping forceps, the evaluation means 14 may evaluate whether the instrument is being used appropriately based on the frequency of opening and closing the tip. Furthermore, the evaluation means 14 may evaluate the manner of operation of the instrument in surgery based on the speed calculated from the interval between the tip positions of the instrument between adjacent frames, the acceleration measured by the gyro center (forceps side), etc. In addition, the analysis means 12 analyzes the positional relationship between the point of action (e.g., the tip of forceps or a resection instrument) which is the part of the instrument operated by the surgeon that contacts the anatomical structure, and the part of the anatomical structure that the instrument contacts (e.g., the part grasped by the forceps or the part cut by the resection instrument). Specifically, the part that grasps the tissue including the anatomical structure and the instrument is image-recognized, and the tissue is appropriately grasped or not is recorded as correct answer data, and an AI learning model (anatomical structure / instrument state determination model) is created for evaluation. For example, a good evaluation is a good surgical operation when the certainty of the grasping state is high. By having the AI ​​learn the state in which some processing is being performed, it is possible to apply it to resection instruments, etc.

[0131] (Evaluation using location information of instruments and structures) As in the present embodiment, the evaluation means 14 evaluates the manner in which an instrument is operated relative to an anatomical structure during surgery, based on the body information including information on the anatomical structure of the body analyzed by the analysis means 12 and information on the position of the instrument relative to the anatomical structure included in the instrument information. Furthermore, the evaluation means 14 may evaluate the content of surgery using the anatomical structure of the body and position information of an instrument (e.g., forceps, etc.) analyzed by the analysis means 12 (for example, the positional relationship between the point of action, which is the part of the instrument operated by the surgeon that contacts the anatomical structure, and the part of the anatomical structure where the instrument contacts). Specifically, the evaluation means 14 may evaluate whether an organ is grasped at an appropriate position with the forceps, based on the anatomical structure and position information on the tip of the forceps.

[0132] In this case, the evaluation means 14 can evaluate the content of the surgery based on the three-dimensional absolute positions of the anatomical structures and instruments, the two-dimensional absolute positions of the anatomical structures and instruments, the three-dimensional relative positions of the anatomical structures and instruments, or the two-dimensional relative positions of the anatomical structures and instruments.

[0133] FIG. 14 is a diagram showing an example of evaluation information generated by the surgery content evaluation system according to an application example of the embodiment of the present invention. FIG. 14 is an example of evaluation information that can be generated by the evaluation means 14 and displayed on the medical institution terminal 2, and shows the anatomical structure and the position of the tip of the surgical instrument calculated according to the analysis result by the analysis means 12 in a graph format. The example shown in FIG. 14 shows the transition of the three-dimensional absolute position of the center of gravity of the structure (anatomical structure) and the three-dimensional absolute position of the tip position of the surgical instrument. In detail, the transition of each of the positions of the center of gravity of the structure in the x, y, and z directions and the position of the tip position of the surgical instrument in the x, y, and z directions is shown. The center of gravity of the structure is the center of gravity position (1 / 2 point, 1 / 3 point, etc.) of the area of ​​the part analyzed by the analysis means 12 as the target structure (for example, pancreas, etc.). Note that instead of the center of gravity of the structure, the position of the outline (upper edge, lower edge, left edge, right edge) of the part may be used.

[0134] FIG. 15 is a diagram showing an example of evaluation information generated by the surgery content evaluation system according to an application example of the embodiment of the present invention. Fig. 15 is an example of evaluation information that is generated by the evaluation means 14 and can be displayed on the medical institution terminal 2, and shows in a graph form the relative positions between the anatomical structure and the tip of the surgical instrument calculated according to the analysis results by the analysis means 12. The example shown in Fig. 15 shows the transition of the relative position between the center of gravity of the structure (anatomical structure) and the tip of the surgical instrument. In detail, the transition of the relative positions in the x, y, and z directions between the center of gravity of the structure and the tip of the surgical instrument is shown.

[0135] The evaluation means 14 evaluates the content of the surgery from the transition of the positional relationship between the anatomical structure and the instrument according to the type of anatomical structure and the content of the surgery. For example, when the anatomical structure is a preserved organ, the evaluation means 14 evaluates the content of the surgery higher if the instrument used for the resection (resection instrument) is farther away from the preserved organ. Also, when the anatomical structure is a resected organ, the evaluation means 14 evaluates the content of the surgery higher if the resection instrument is closer to the resected organ. In this case, the evaluation means 14 may evaluate the content of the surgery higher if the speed at which the relative positions of the resection instrument and the resected organ become closer is fast or if the acceleration is constant.

[0136] 14 and 15 may be cumulatively recorded in the storage means 20, and this evaluation information may be divided into appropriate surgeries (e.g., surgeries in which no adverse events occurred, surgeries performed by experienced surgeons, etc.) and inexperienced surgeries (surgeries in which adverse events occurred, surgeries performed by inexperienced surgeons, etc.) and the average of each may be calculated to determine the boundary between skilled and inexperienced surgeries, or an AI may be trained to generate a learning model for evaluating the content of the surgery from the transition of the positional relationship between the anatomical structures and the instruments. This allows the evaluation means 14 to use such boundary points and learning models to evaluate the content of the surgery from the transition of the positional relationship between the anatomical structures and the instruments.

[0137] (Assessment of surgical difficulty based on awareness of anatomical structures) The evaluation means 14 evaluates the difficulty of the surgery according to the information on the anatomical structure of the body contained in the physical information.

[0138] Fig. 16 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. Fig. 16(a) is a surgery image of a thin patient. Fig. 16(b) is a surgery image of an obese patient.

[0139] As shown in Figure 16(a), if the patient is thin, the amount of fat is small, so the blood vessels can be easily seen, making the surgery less difficult. On the other hand, if the patient is obese, fat is attached to the target organ, making the blood vessels invisible, making the surgery more difficult. The evaluation means 14 evaluates the area of ​​adhesions, etc., which are composed of fat, blood vessels, and connective tissue, at the early stage of the surgery, and evaluates the difficulty of the surgery. The evaluation means 14 may also correct the final evaluation value of the contents of the surgery based on such an evaluation at the early stage of the surgery. If the evaluation means 14 evaluates the surgery as being highly difficult at the early stage of the surgery, it may send warning information indicating a warning or suggestion such as a high risk of bleeding to a terminal of the medical institution.

[0140] (Evaluation of bleeding frequency and bleeding area) FIG. 17 is a diagram for explaining the processing of the evaluation means in the surgery content evaluation system according to the embodiment of the present invention. As in this embodiment, the analysis means 12 compares the surgical image acquired by the acquisition means 11 with the specific area analysis model stored in the memory means 20, analyzes the specific area (the area created by the outflow of body fluids (e.g., blood, bile, intestinal fluids, etc.)) in the surgical image, and colors the specific area (the area marked as bleeding in the example shown in Figure 17) and the outline of the specific area.

[0141] The evaluation means 14 quantifies the number of times the specific region is analyzed by the analysis means 12 (the number of times the body fluid flows out), and evaluates the content of the surgery performed by the surgeon according to this number. The analysis means 12 may also determine the type of body fluid (blood, bile, intestinal fluid, etc.) that forms the specific region. In this case, the evaluation means 14 may evaluate the content of the surgery performed by the surgeon according to the number of times for each type of body fluid that forms the specific region. For example, when the analysis means 12 determines that the type of body fluid is blood, the evaluation means 14 may gradually lower the evaluation as the number of times increases, and when the analysis means 12 determines that the type of body fluid is a specific body fluid (bile, intestinal fluid, etc.), the evaluation means 14 may significantly lower the evaluation if the outflow of the specific body fluid is analyzed even once.

[0142] Furthermore, the analysis means 12 analyzes, as the tool information, a body fluid treatment tool (for example, gauze used to wipe bleeding) that treats leaked body fluids. The evaluation means 14 may evaluate the content of the surgery being performed by the surgeon according to the number of times the body fluid treatment tool has been analyzed by the analysis means 12 and the distance moved by the body fluid treatment tool (area per frame). If the number of times a body fluid treatment tool (for example, gauze) has been inserted or the distance moved is large, it is possible that treatment for bleeding has been performed frequently. For this reason, the evaluation means 14 may lower the evaluation the greater the value of the number of times the body fluid treatment tool has been analyzed or the distance moved by the body fluid treatment tool.

[0143] Incidentally, a body fluid processing tool (e.g., gauze, etc.) may also be used to gently restrain the preserved tissue. For this reason, the evaluation means 14 may correct the evaluation of the content of the surgery using the anatomical structure of the body and the position information of the body fluid processing tool. For example, when the positions of the preserved tissue and the body fluid processing tool are close to each other and the movement of the body fluid processing tool is not analyzed (when it is presumed that the body fluid processing tool is used to gently restrain), the evaluation means 14 does not have to lower the evaluation even if the body fluid processing tool is analyzed.

[0144] Fig. 18 is a diagram showing an example of an evaluation criterion in the evaluation means of the surgery content evaluation system according to the application example of the embodiment of the present invention. Fig. 18 shows the change over time in the area of ​​a specific region (e.g., blood) and the moving distance of an instrument (e.g., a resection instrument).

[0145] In the example shown in Fig. 18, the area of ​​the specific region does not decrease (the amount of bleeding does not decrease), and the resection instrument continues to move. In such a case, it is estimated that the operation is proceeding without stopping the bleeding, and the evaluation means 14 lowers the evaluation.

[0146] (Image extraction and video editing using anatomical structure recognition) As in the present embodiment, the surgery content evaluation system 1 may include an image editing means for generating a digest version image of the surgery image acquired by the acquisition means 11. The image editing means extracts, for each step, a video of a predetermined time including frames of each step in the surgery identified by the analysis means 12 from the surgery image acquired by the acquisition means 11, and synthesizes these to generate a digest version image.

[0147] In such a case, the evaluation means 14 identifies surgical images (frames) for which the analysis means 12 has determined that the degree of recognition of information relating to a specific anatomical structure included in the physical information is equal to or greater than a specific threshold value.

[0148] Fig. 19 is a diagram for explaining the processing in the image editing means of the surgery content evaluation system according to the application example of the embodiment of the present invention. Fig. 19 shows the change over time in the recognition degree of the target organ and the range of the extracted moving image.

[0149] The evaluation means 14 sets a surgical image (frame) for which the analysis means 12 has determined that the degree of recognition of information relating to a specific anatomical structure (e.g., a target organ such as nervous tissue) contained in the physical information is equal to or above a specific threshold (area of ​​nervous tissue (pixel amount or total degree of certainty)) as a video extraction point. Then, the image editing means automatically extracts a predetermined period before and after the video extraction point and edits the video. The predetermined period is, for example, from when the analysis means 12 analyzes the anatomical structure (from when the degree of recognition of the anatomical structure occurs) to when the anatomical structure is no longer analyzed (until the degree of recognition of the anatomical structure becomes 0).

[0150] The evaluation means 14 may specify a surgical image (frame) in which the certainty of the target organ is equal to or higher than a certain level. The evaluation means 14 may set a moving image extraction point based on the relationship between the certainty of the operating tool and the structure, or the graph of the area of ​​the structure.

[0151] This makes it possible to automatically extract video from the video of the entire surgery, from the point where anatomical structures (e.g., nerve tissue, etc.) begin to appear to the point where the anatomical structures disappear. In this way, by using a video from which a specific scene during surgery is extracted to evaluate the contents of the surgery, it becomes possible to perform the evaluation more efficiently than when the evaluation is based on a video of the entire surgery.

[0152] Furthermore, for example, when extracting a specific scene during surgery, the image editing means may display, in the extracted video, the trajectory of an instrument (e.g., a resection line, etc.) presented by an AI model that has learned the resection lines of each expert doctor. Furthermore, the image editing means may display, in the extracted video, a predicted surgical field and a predicted structure after resection using a generative adversarial network (GAN) or the like.

[0153] Fig. 20 is a diagram for explaining processing in an image editing means of a surgery content evaluation system according to an application example of an embodiment of the present invention. Fig. 20 shows a transition of a surgery image by operation of an imaging means (for example, a camera) that captures the surgery image.

[0154] A camera for capturing images inside the body is passed through a camera port that is inserted into the body from outside. The upper part of Fig. 20 shows the transition of surgical images when the camera placed inside the body is pulled out of the body, and the lower part of Fig. 20 shows the transition of surgical images when the camera is inserted from outside the body into the body.

[0155] For example, when a camera is placed inside the body and a structure is analyzed by the analysis means 12, when the camera is pulled out of the body, the certainty becomes 0. After that, when the camera is inserted into the body from outside, the structure is analyzed by the analysis means 12 and the certainty increases. The evaluation means 14 may identify a first point where the certainty of the target organ becomes 0 and a second point where the certainty increases again thereafter. The analysis means 12 may also analyze a fixed structure outside the camera (e.g., a camera port) as the instrument information. In this case, the evaluation means 14 may identify the time when such a structure is analyzed after starting the analysis of the body information as the first point. Based on the first point and the second point thus identified by the evaluation means 14, the image editing means may automatically delete the video between the first point and the second point.

[0156] According to the surgery content evaluation system 1 relating to such an application example, it becomes possible to evaluate the content of the surgery performed by the surgeon according to the recognition degree indicating the degree of recognition of the physical information in the surgery image.

[0157] For example, if the target organ of the surgery is exposed and separated from other organs, the physical information of the target organ is clear in the surgical image, the degree of recognition of the physical information is higher, and the degree of recognition is higher. On the other hand, if the target organ of the surgery is not clearly separated from other organs, the physical information of the target organ is unclear in the surgical image, the degree of recognition of the physical information is lower, and the degree of recognition is lower. Naturally, the more the target organ of the surgery is separated from other organs and exposed, the smoother the operation on the target organ will be, the less likely it is that damage will be caused to other organs, and the better the surgery will be with less strain on the patient's body (the higher the evaluation of the content of the surgery will be).

[0158] Moreover, according to the surgery content evaluation system 1 of the application example, the analysis means can calculate the degree of recognition based on the degree of certainty that indicates the accuracy of the analysis result of the physical information. This makes it possible to calculate the degree of recognition that indicates the degree of recognition of the physical information from a more objective point of view, and to evaluate the contents of the surgery performed by the surgeon according to such degree of recognition, so that the contents of the actual surgery can be evaluated more objectively.

[0159] Furthermore, according to the surgery content evaluation system 1 according to the application example, it is possible to evaluate the content of surgery performed by a surgeon according to changes over time in the degree of recognition.

[0160] For example, if the target organ of the surgery (physical information to be analyzed) is a preserved organ, the faster the rate of increase in the degree of recognition, the more skillfully the target organ is exposed. Also, if the target organ of the surgery (physical information to be analyzed) is an organ to be resected, the more skillfully the target organ is resected, the lower the degree of recognition remains. Also, if there is relatively little change over time in the degree of recognition for a certain organ, there is no inconsistency in the technique. In these cases, the surgery will be better with less strain on the patient's body (the content of the surgery will be evaluated higher).

[0161] Furthermore, according to the surgery content evaluation system 1 according to the application example, it is possible to evaluate the difficulty of surgery according to information on the anatomical structure of the body included in the physical information.

[0162] For example, in the case of gastrectomy, if there is a relatively small amount of fat (an example of an anatomical structure), the blood vessels can be easily seen, making the surgery less difficult. On the other hand, if there is a relatively large amount of fat (an example of an anatomical structure), the blood vessels will be covered by the fat and will not be visible, and adhesions will prevent the fat from being lifted sufficiently, resulting in a poor field of vision and making the surgery more difficult.

[0163] Furthermore, the surgery content evaluation system 1 according to the application example makes it possible to evaluate the operation mode of an instrument in a surgery based on the trajectory of the tip position of the instrument operated by the surgeon. This improves the analysis accuracy of the trajectory, and since it is not necessary to recognize all of the instruments, it is possible to reduce the load of the analysis process.

[0164] In addition, according to the application example of the surgical content evaluation system 1, it is possible to evaluate the manner in which an instrument is operated during surgery based on the positional relationship between the point of action, which is the part of the instrument operated by the surgeon that contacts the anatomical structure, and the part of the anatomical structure where the instrument contacts.

[0165] When resecting an anatomical structure, the part of the anatomical structure grasped by the forceps may apply appropriate tension to the anatomical structure, making the cutting position easier to cut, or the anatomical structure may become loose, making the cutting position not clearly exposed. For example, if the tip (the part that grasps), which is the point of action of the forceps, is positioned at a position where the surgeon can properly grasp the anatomical structure intended by the surgeon and apply appropriate tension, the possibility of resecting the resected organ without damaging the preserved organ increases (the operating manner of the instrument in the surgery is evaluated more highly).

[0166] Furthermore, according to the surgical content evaluation system 1 of the application example, it is possible to specify surgical images, among a plurality of surgical images consecutively arranged in a time series, for which the degree of recognition of information on a specific anatomical structure included in the body information is determined to be equal to or higher than a specific threshold. This makes it possible to specify the time point at which a specific anatomical structure is exposed in a video consisting of a plurality of surgical images consecutively arranged in a time series. For example, by attaching a marker to the specified time point in the video, it becomes possible to quickly play back the scene at which the specific anatomical structure is exposed, or to edit the video based on the specified time point.

[0167] Although the present invention has been described above using the embodiments and application examples, it goes without saying that the technical scope of the present invention is not limited to the scope described in the above embodiments and application examples. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiments and application examples. In addition, it is clear from the claims that the forms to which such modifications or improvements are made can also be included in the technical scope of the present invention. In the above embodiments and application examples, the present invention has been described as an invention of a product, namely, a surgery content evaluation system, but it can also be considered as an invention of a method executed by the surgery content evaluation system in the present invention, or a program that causes the surgery content evaluation system to function as various parts. [Explanation of symbols]

[0168] 1. Surgery content evaluation system 2 Medical institution terminals 11 Acquisition method 12 Analysis methods 13 Timekeeping means 14 Evaluation methods 15 Transmission Method 20 Memory means

Claims

1. A surgery content evaluation system for evaluating the content of a surgery performed by a surgeon, comprising: An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing, in the surgical image, body information indicating a state of the body and / or instrument information indicating a state of an instrument being operated by a surgeon; and an evaluation means for evaluating the content of the surgery being performed by the surgeon based on the physical information and / or the instrument information analyzed by the analysis means, The analysis means analyzing said physical information, including information regarding anatomy or specific regions of the body; analyzing the body information and / or the instrument information, including information regarding the anatomical structure or specific region and the position and status of the instrument; The evaluation means identifies the anatomical structure or the specific region, and evaluates the operation mode of the instrument with respect to the anatomical structure or the specific region. Surgery content evaluation system.

2. The operation mode of the tool is evaluated by evaluating at least one of a trajectory, a moving distance, a speed, a frequency of opening and closing, a number of attempts of the operation, and a number of occurrences. The surgical content evaluation system according to claim 1 .

3. A method implemented by a surgical content evaluation system for evaluating the content of a surgical procedure performed by a surgeon, comprising: acquiring surgical images of a patient's body undergoing surgery by a surgeon; analyzing, in the surgical image, body information indicating a state of the body and / or instrument information indicating a state of an instrument operated by the surgeon; and evaluating the content of the surgery being performed by the surgeon based on the physical information and / or the instrument information analyzed in the analyzing step, In the analyzing step, analyzing said physical information, including information regarding anatomy or specific regions of the body; analyzing the body information and / or the instrument information, including information regarding the anatomical structure or specific region and the position and status of the instrument; In the step of evaluating, the anatomical structure or the specific region is identified, and an operation manner of the instrument is evaluated with respect to the anatomical structure or the specific region. Methods for evaluating surgical procedures.

4. A system for evaluating the content of surgery performed by a surgeon, An acquisition means for acquiring surgical images, which are images of a patient's body undergoing surgery by a surgeon; an analysis means for analyzing, in the surgical image, body information indicating a state of the body and / or instrument information indicating a state of an instrument being operated by a surgeon; an evaluation means for evaluating the content of the surgery being performed by the surgeon based on the physical information and / or the instrument information analyzed by the analysis means; Furthermore, the analysis means analyzing said physical information, including information regarding anatomy or specific regions of the body; analyzing the body information and / or the instrument information, including information regarding the anatomical structure or specific region and the position and status of the instrument; The evaluation means functions as a means for identifying the anatomical structure or the specific region and evaluating the operation mode of the instrument with respect to the anatomical structure or the specific region. program.

Images