Surgical support device, surgical support method, and surgical support program

Abstract

It is impossible to know the thought process of a skilled surgeon during surgery. [Solution] A surgical support device according to one embodiment of the present invention comprises: an image acquisition unit that acquires surgical images; an information acquisition unit that acquires queries and reference information from a user; a thought estimation unit that performs image recognition on the surgical images and estimates at least one of a situational judgment, a decision, and a thought process leading to the situational judgment or the decision using the queries and reference information from the user; and an output unit that changes the output format according to the results of the image recognition, the queries and reference information from the user, and at least one of the situational judgment, the decision, and the thought process, and outputs at least one of the situational judgment, the decision, and the thought process to the user.

Description

TECHNICAL FIELD

[0001] The present invention relates to a surgical assistance device, a surgical assistance method, and a surgical assistance program. BACKGROUND ART

[0002] Intraoperative complications remain common, with a mortality rate of 2% and a complication rate of over 50% particularly observed after esophageal cancer surgery. Notably, the complication rate varies depending on the surgeon's experience and the medical facility: while experienced surgeons and large facilities tend to achieve better outcomes, less experienced surgeons show a tendency toward higher complication rates. Causes of complications include intraoperative misrecognition, lack of attention, and communication errors. PRIOR ART DOCUMENTS PATENT DOCUMENTS

[0003] PATENT DOCUMENT 1 Japanese Unexamined Patent Application Publication No. 2020-194439 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] In surgical education, it is important to learn in real time while observing a surgery by clarifying "What is the surgeon thinking and what is the surgeon doing?", "What is the surgeon seeing?", "Why is the surgeon performing such an action?", and "What will the surgeon do next?". That is, it is important to learn the situational judgment and decision-making of the surgeon during surgery. However, since surgeons are concentrated on the surgery, students and others find it difficult to speak to them, and as a result, students and others cannot learn effectively. In addition, even during surgery, there are many situations where decision-making needs to be shared within the same team, but communication errors may lead to the occurrence of complications.

[0005] Accordingly, an object of the present invention is to understand the thinking of a skilled physician during surgery. MEANS FOR SOLVING THE PROBLEMS

[0006] A surgical support device according to one embodiment of the present invention includes: an image acquisition unit that acquires surgical images; an information acquisition unit that acquires queries and reference information from a user; a thought estimation unit that performs image recognition on the surgical images and estimates at least one of a situational judgment, a decision, and a thought process leading to the situational judgment or the decision, using the queries from the user and the reference information; and an output unit that changes the output format according to the results of the image recognition, the queries from the user and the reference information, and at least one of the situational judgment, the decision, and the thought process, and outputs at least one of the situational judgment, the decision, and the thought process to the user. [Effects of the Invention]

[0007] According to this invention, it is possible to understand the thought process of a skilled surgeon during surgery. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating an overview of one embodiment of the present invention. [Figure 2] This diagram shows the overall configuration of one embodiment of the present invention. [Figure 3] This is a functional block diagram of a surgical support device according to one embodiment of the present invention. [Figure 4] This is a flowchart of the estimation process for situational judgment, decision-making, and thought processes related to one embodiment of the present invention. [Figure 5] This diagram illustrates the estimation of situational judgment, decision-making, and thought processes related to one embodiment of the present invention. [Figure 6] This diagram illustrates the estimation of situational judgment, decision-making, and thought processes related to one embodiment of the present invention. [Figure 7] This is a diagram illustrating image recognition of surgical images related to one embodiment of the present invention. [Figure 8] This is a diagram illustrating image recognition of surgical images related to one embodiment of the present invention. [Figure 9]This is a diagram illustrating the creation of a prompt related to one embodiment of the present invention. [Figure 10] This diagram illustrates the classification of queries and answers related to one embodiment of the present invention. [Figure 11] This diagram illustrates the classification of queries and answers related to one embodiment of the present invention. [Figure 12] This is an example of an interactive display relating to one embodiment of the present invention. [Figure 13] This is an example of an interactive display relating to one embodiment of the present invention. [Figure 14] This is a hardware configuration diagram of a surgical assistance device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] Embodiments of the invention will be described below with reference to the drawings.

[0010] <Explanation of Terms> In this specification, "surgical image" refers to an image of a patient undergoing surgery, and may be a video or a still image. In this specification, “surgery” may refer to any type of surgery. For example, surgery is performed in general surgery (digestive system, vascular, mammary gland, thyroid), cardiovascular surgery, respiratory surgery, neurosurgery, orthopedic surgery, plastic surgery, urology, obstetrics and gynecology, ophthalmology, otolaryngology, oral surgery, pediatric surgery, transplant surgery, psychiatry, trauma surgery, etc. For example, (1) and (2) below are performed. (1) Surgical procedures according to the surgical method (grasping, incision, and dissection of organs and other objects in surgical procedures, determination of the extent of malignant tumor resection, securing safe margins, dissection of nearby lymph nodes, resection of digestive organs (gastrectomy, intestinal resection, hepatectomy, cholecystectomy, etc.), reconstructive procedures (anastomosis, bypass surgery), use of staplers (automatic staplers), clipping of aneurysms and replacement of artificial blood vessels in vascular surgery, reduction and fixation of fractures in orthopedic surgery (plates, screws, intramedullary nails), replacement of artificial joints, removal of brain tumors in neurosurgery, clipping of cerebral aneurysms, submucosal dissection in endoscopic treatment, etc.) (2) Suturing and ligation operations (clipping, ligation, hemostasis by bipolar and monopolar methods, use of hemostatic agents (gelatin sponge, fibrin glue), suturing of tissues such as digestive tract and blood vessels, use of surgical staplers, insertion of drains) · As used herein, a "skilled person" refers to a medical worker who satisfies predetermined conditions (for example, a doctor such as a surgeon, a resident, a medical student, a nurse, a nursing student, a clinical engineer, etc.). For example, the skilled person is a person who has been engaged in medical practice for a period equal to or longer than a threshold, a person who has performed a number of surgeries equal to or more than a threshold, or the like.

[0011] <Overview> Figure 1 is a diagram for explaining an overview according to an embodiment of the present invention. In the present invention, a surgery support device 10 (which will be described in detail later) performs image recognition on a surgical image, and uses a query and reference information from a user to estimate at least one of a situation judgment, a decision-making, and a thinking process (specifically, a thinking process leading to the situation judgment and a thinking process leading to the decision-making) of a skilled person (for example, a skilled doctor or the like) for the surgical image. The surgery support device 10 changes an output mode according to the result of the image recognition, the query and reference information from the user, and at least one of the situation judgment, the decision-making, and the thinking process, and outputs at least one of the situation judgment, the decision-making, and the thinking process to the user.

[0012] For example, the surgical assistance device 10 performs image recognition on a surgical image to recognize the type and position of a site (for example, an anatomical structure such as an organ or a nerve), the type and position of an instrument, the presence or absence of bleeding, and the surgical process. The surgical assistance device 10 inputs the image recognition result (for example, assuming that the site recognition result is the right recurrent laryngeal nerve, the instrument recognition result is bipolar forceps, the bleeding recognition result is no bleeding, and the surgical process recognition result is the right recurrent laryngeal nerve lymph node), a query from the user (for example, "Please tell me the situation of this scene and the future policy"), and reference information (for example: new staff, no surgical experience, surgical procedure (esophagus)) into a generative AI (artificial intelligence) that generates text, to output the situation judgment, decision-making, and thought process of a skilled doctor (for example, "You are dissecting the lymph node outside the right recurrent laryngeal nerve with bipolar forceps. Please perform the dissection carefully, as there is a risk of paralysis due to recurrent laryngeal nerve injury."), and presents it to the user.

[0013] <System Configuration> Figure 2 is a diagram showing the overall configuration according to an embodiment of the present invention. The surgical assistance system 1 can include a surgical assistance device 10, an input device 11, an output device 12, and an imaging device 20. Each of these will be described below.

[0014] <<Surgical Assistance Device>> The surgical assistance device 10 is a device that estimates at least one of the situation judgment, decision-making, and thought process (specifically, the thought process leading to situation judgment and the thought process leading to decision-making) of a skilled person (for example, a skilled doctor) for a surgical image. When the surgical assistance device 10 is operating during surgery, the surgical image being captured by the imaging device 20 is used, and when the surgical assistance device 10 is operating outside of surgery, a recorded surgical image captured by the imaging device 20 in the past is used. The surgical assistance device 10 consists of one or a plurality of computers.

[0015] The surgical support device 10 may be a computer (local server) installed in the operating room, or a computer installed outside the operating room (for example, a cloud server, where a terminal installed in the operating room accesses the cloud server).

[0016] <<Input Device>> The input device 11 is a microphone, keyboard, mouse, etc., used to give various instructions to the surgical support device 10.

[0017] <<Output Device>> The output device 12 includes a monitor or display for displaying images and text acquired from the surgical support device 10, and a speaker for playing back audio acquired from the surgical support device 10.

[0018] <<Imaging device>> The imaging device 20 is a device (for example, an endoscope camera) that takes pictures of the patient undergoing surgery. The images taken by the imaging device 20 are called surgical images (they may be videos or still images).

[0019] Furthermore, at least two of the surgical support device 10, input device 11, output device 12, and imaging device 20 may be implemented in a single device.

[0020] <Function Block> Figure 3 is a functional block diagram of a surgical support device 10 according to one embodiment of the present invention. The surgical support device 10 may include an image acquisition unit 101, an information acquisition unit 102, a knowledge database processing unit 103, a knowledge database storage unit 104, a thought estimation unit 105, an output unit 106, an interaction processing unit 107, and a data storage unit 108. By executing a program, the surgical support device 10 can function as the image acquisition unit 101, the information acquisition unit 102, the knowledge database processing unit 103, the thought estimation unit 105, the output unit 106, and the interaction processing unit 107. Each of these will be described below.

[0021] The image acquisition unit 101 acquires surgical images.

[0022] [[Surgical Images]] Now, let's explain surgical images. The image acquisition unit 101 may acquire surgical images being taken by the imaging device 20 in real time, or it may acquire surgical images taken in the past that are stored in the data storage unit 108 or the like.

[0023] The information acquisition unit 102 acquires queries (e.g., questions) and reference information from the user (i.e., someone who wants to know the situational judgment, decision-making, and thought process of an expert). For example, the information acquisition unit 102 acquires queries and reference information entered by the user using the input device 11, etc.

[0024] The knowledge database processing unit 103 retrieves knowledge data from the knowledge database (for example, medical books, reports from skilled physicians, etc.) stored in the knowledge database storage unit 104.

[0025] The knowledge database storage unit 104 stores the knowledge database.

[0026] The thought estimation unit 105 performs image recognition on the surgical images acquired by the image acquisition unit 101 and uses user queries and reference information acquired by the information acquisition unit 102 to estimate at least one of the following regarding the surgical images: the expert's "situational judgment," "decision-making," and "thought process leading to the situational judgment or decision-making." Specifically, the thought estimation unit 105 uses the visual language described in Figures 5 and 6 for estimation.

[0027] The output unit 106 changes the output format according to the image recognition results from the thought estimation unit 105, the user queries and reference information acquired by the information acquisition unit 102, and at least one of the "situation judgment," "decision-making," and "thought process" estimated by the thought estimation unit 105, and outputs at least one of the "situation judgment," "decision-making," and "thought process" estimated by the thought estimation unit 105 to the user. The output unit 106 may also output at least one of the "situation judgment," "decision-making," and "thought process" estimated by the thought estimation unit 105 along with the image recognition results from the thought estimation unit 105.

[0028] The interaction processing unit 107 changes the output format of the information output by the output unit 106 in accordance with the user's operation (instructions entered by the user using the input device 11, etc.).

[0029] Various types of data are stored in the data storage unit 108.

[0030] Furthermore, the surgical support device 10 may further include a learning unit for learning a language model (also called a language generation model; hereinafter referred to as LM), an auxiliary information addition determination unit for determining whether or not to add auxiliary information to the output, and an emergency state determination unit for determining whether the surgical image is in an emergency state.

[0031] <Processing method> Figure 4 is a flowchart of the estimation process for situational judgment, decision-making, and thought processes in one embodiment of the present invention.

[0032] In step 1 (S1), the information acquisition unit 102 of the surgical support device 10 acquires queries and reference information from the user. In this way, by acquiring reference information, the output format of the natural language output in S8 (i.e., the situation judgment, decision-making, and thought process of an expert such as a skilled physician) can be changed according to the reference information.

[0033] In step 2 (S2), the knowledge database processing unit 103 of the surgical support device 10 retrieves knowledge data from the knowledge database stored in the knowledge database storage unit 104. By referring to this knowledge data (for example, medical textbooks, reports from experienced physicians, etc.), it is possible to estimate the situational judgment, decision-making, and thought processes of experienced physicians and other experts.

[0034] In step 3 (S3), the image acquisition unit 101 of the surgical support device 10 acquires surgical images.

[0035] In step 4 (S4), the thought estimation unit 105 of the surgical support device 10 generates query features and reference information features using the query and reference information obtained from the user in S1.

[0036] In step 5 (S5), the thought estimation unit 105 of the surgical support device 10 performs image recognition on the surgical image acquired in S3.

[0037] In step 6 (S6), the thought estimation unit 105 of the surgical support device 10 generates image features using the surgical image acquired in S3 or the image recognition result in S5.

[0038] The thought estimation unit 105 may generate image features using a machine learning model that has undergone representation learning (where arbitrary information is associated with an image). The thought estimation unit 105 may also generate image features using the results of image recognition specialized for a specific task (for example, recognition of body parts (organs, etc.), recognition of instruments, recognition of bleeding, recognition of processes, etc.).

[0039] In step 7 (S7), the thought estimation unit 105 of the surgical support device 10 estimates at least one of the expert's situational judgment, decision-making, and thought process regarding the surgical image. The thought estimation unit 105 may use information about the thought process in image recognition (specifically, annotation data from a doctor or the like is used to generate the image recognition model), information about the thought process output by the generating AI, information about the thought process acquired by the generating AI repeatedly generating answers (specifically, image information obtained from the results of image recognition, language information previously output by the LM (language model), and information from the knowledge database are used as input to the LM (language model)), or information about the thought process linked by knowledge data from the knowledge database (specifically, knowledge data stored in the knowledge database is acquired by RAG and said knowledge data is input to the LM (language model)).

[0040] In step 8 (S8), the output unit 106 of the surgical support device 10 changes the output format according to the image recognition results, user queries and reference information, and at least one of the situation judgment, decision-making and thought process, and outputs at least one of the situation judgment, decision-making and thought process estimated in S7. The output unit 106 can also output the image recognition results.

[0041] In step 9 (S9), the interaction processing unit 107 of the surgical support device 10 changes the output format of the information output by the output unit 106 in accordance with the user's operation (instructions entered by the user using the input device 11, etc.).

[0042] In step 10 (S10), the data storage unit 108 of the surgical support device 10 records the user's input queries and reference information, image recognition results, situation assessment, decision-making, and thought process information.

[0043] The following describes the details of each step in Figure 4.

[0044] [S1 (Query - Retrieving Reference Information)] The information acquisition unit 102 of the surgical support device 10 acquires queries and reference information from the user. For example, the voice of a query is acquired via a headset with a microphone worn by the user (surgeon). For example, the text of a query is acquired via a keyboard. If the query is voice, arbitrary speech recognition (specifically, speech recognition processing that uses the frequency of the voice waveform as a feature, for example, GMM-HMM or deep learning) is performed to convert the voice into text. Voiceprint authentication may also be performed to acquire role information (surgeon, assistant, nurse, or individual), and this role information may be added to the text output by speech recognition.

[0045] [[Query]] Now, let's explain queries. A query is a prompt (for example, a question) for the generating AI. Queries may also contain specific, fixed words or phrases.

[0046] [[Reference information]] Here, we will explain the reference information. The reference information includes at least one of the following: information about the user (i.e., someone who wants to know the situational judgment, decision-making, and thought process of an expert), information about the surgery, and information about the patient. For example, the reference information may include the user's profile such as their medical history, surgical history, and age, pre-operative information such as the surgical procedure, and the patient's medical record information.

[0047] [S2 (Knowledge Data Acquisition)] The knowledge database processing unit 103 of the surgical support device 10 retrieves knowledge data from the knowledge database stored in the knowledge database storage unit 104. The knowledge database processing unit 103 can retrieve knowledge data from the knowledge database stored in the knowledge database storage unit 104 using RAG (Retrieval-Augmented Generation) methods (vector search, graph search).

[0048] [[Knowledge Database]] Now, let's explain the knowledge database. The knowledge database stores data from medical textbooks, doctors' surgical reports, and doctors' annotations. Specifically, it stores data from medical textbooks, surgical reports, and doctors' annotations in a hierarchical structure for each surgical procedure. Therefore, it is possible to acquire only the knowledge data corresponding to the reference information (for example, if the pre-operative information is the surgical procedure: esophageal surgery, then only knowledge data related to esophageal surgery will be acquired). Furthermore, the knowledge data is not limited to text; it may also be images.

[0049] Alternatively, the above-mentioned reference information may be stored in the knowledge database storage unit 104, and the information acquisition unit 102 may acquire the reference information stored in the knowledge database storage unit 104.

[0050] [S3 (Acquisition of surgical images)] The image acquisition unit 101 of the surgical support device 10 acquires surgical images.

[0051] [S4 (Generation of query and reference information features)] The thought estimation unit 105 of the surgical support device 10 generates query features and reference information features using queries and reference information from the user.

[0052] The S4 process will be explained while referring to the generation of query and reference information features in Figure 5. Using a tokenizer, the query and reference information texts are decomposed into tokens, and these tokens are input to the Embedding Layer of the Language Model, outputting Embedding (query features and reference information features (embedded representations)). Similarly, using a tokenizer, the knowledge data texts in the knowledge database are decomposed into tokens, and these tokens are input to the Embedding Layer of the Language Model, outputting Embedding (knowledge data features (embedded representations)).

[0053] The thought estimation unit 105 may generate the Embedding after explicitly adding a flag indicating that it is specific reference information, or a parameter that controls the generated text.

[0054] In the case of specific reference information, the thought estimation unit 105 may add prompts to modify the text (i.e., sentences describing the situation, decision-making, and thought process). For example, surgical experience may be defined as reference information to be entered in advance, and if the surgical experience is 0-3 years, the prompt may be given a prompt to provide a concise and easy-to-understand explanation. A user interface for entering surgical experience may also be provided.

[0055] In the case of specific reference information, the thought estimation unit 105 may assign a special token. For example, if surgical experience is defined as reference information to be entered in advance, and the surgical experience is 0-3 years, the Tokenizer assigns a special token (e.g., a beginner token) to the reference information token. When a beginner token is assigned, the text generated by the LM (language model) (i.e., the sentences describing the situation, decision-making, and thought process) will be concise and easy to understand (i.e., learned during the LM (language model) training). A user interface for inputting surgical experience may also be provided.

[0056] In the case of specific reference information, the thought estimation unit 105 may change the output control parameters (parameters that control the generated text). For example, if surgical experience is defined as reference information to be entered in advance, and the surgical experience is 0-3 years, the output control parameters of the LM (language model) may be changed to change the length or variety of the generated text (i.e., sentences describing situational judgment, decision-making, and thought processes). A user interface for entering surgical experience may also be provided. For example, you can change the Temperature of the LM (Language Model) output or the distribution of attention during processing. For example, change the number of tokens in the output of the LM (Language Model). For example, change the parameters for the Top K samples of the output tokens of the LM (Language Model).

[0057] [S5 (Image Recognition)] The thought estimation unit 105 of the surgical support device 10 performs image recognition on surgical images. For example, the thought estimation unit 105 uses semantic segmentation to recognize the location (x-coordinate and y-coordinate) of specific parts (organs, etc.), instruments, or bleeding detected on the surgical image at the pixel level. Alternatively, for example, the thought estimation unit 105 uses classification to recognize the surgical process from the surgical image.

[0058] Figure 7 is a diagram illustrating image recognition of surgical images according to one embodiment of the present invention. As shown in Figure 7 [Recognition of body part (organ, etc.), instrument, and bleeding], the location and likelihood of detecting a body part (organ, etc.) on the surgical image, the location and likelihood of detecting an instrument on the surgical image, and the location and likelihood of detecting bleeding on the surgical image are recognized. As shown in Figure 7 [Recognition of surgical process], the surgical process is recognized.

[0059] The thinking estimation unit 105 can perform image recognition and recognize the incision line, margins relative to the incision line, and direction of incision, as shown in Figure 8. As shown in Figure 8, the thinking estimation unit 105 can recognize the incision line (the area to be incised), the margins relative to the incision line, the direction of incision (the direction to be incised), and the direction of traction (specifically, it performs regression estimation in the x and y directions relative to the incision line).

[0060] [S6 (Image Feature Generation)] The thought estimation unit 105 uses the surgical image or the result of image recognition to generate image features (i.e., at least one feature that serves as the basis for situational judgment, decision-making, and thought processes).

[0061] The process of S6 will be explained with reference to the generation of surgical image features in Figure 5. Note that the visual-language model used in this invention is not limited to Figure 5, but may also be Example 1 or Example 2 in Figure 6, as any module capable of handling visual and linguistic information is acceptable.

[0062] [[When generating embeddings from surgical images]] The surgical image is input into a Vision Encoder (where arbitrary information is associated with the image), and the feature vectors of the surgical image are output. Next, the feature vectors of the surgical image are input into a Projector, and the Embedding (the feature vectors of the surgical image (embedded representation)) which is input to the Language Model (LM) is output.

[0063] [[When generating an embedding (embedded representation) from the image recognition results]] First, as described above, an Image Recognition Model (IMR) is used to perform image recognition of surgical images. This image recognition is specialized for specific tasks (e.g., recognition of body parts (organs, etc.), recognition of instruments, recognition of bleeding, recognition of processes, etc.). Next, a prompt is created based on the results of the image recognition, and the prompt is broken down into tokens using a Tokenizer. These tokens are then input into the Embedding Layer of the Language Model (LM), and Embedding (features of the surgical image (embedded representations)) is output.

[0064] For example, if an image recognition model that detects nerves recognizes a nerve as a result of image recognition, a prompt such as "A nerve is present on the right side" is created, and a token is generated using a Tokenizer. For example, if an image recognition model that recognizes surgical procedures recognizes a dissection procedure as a result of image recognition, a prompt such as "Dissection procedure is being performed" is created, and a token is generated using a Tokenizer. The prompts are predefined sentences (template sentences) such as "b is present on side a," into which the information from the image recognition results is applied.

[0065] Furthermore, when multiple objects are detected or when multiple image recognition models are used, multiple prompts may be input to the Tokenizer to create tokens. Also, in Example 1 of Figures 5 and 6, instead of Prompt+Tokenizer, a Projector may be used to create the Embedding that will be input to the Language Model (LM) (in which case, the intermediate features of the output of the image recognition model's backbone will be used). In addition, the output of the image recognition model may be used as input to the Vision Encoder.

[0066] [S7 (Thought and Estimation Processing)] The thought estimation unit 105 estimates the situational judgment, decision-making, and thought processes of an expert regarding surgical images. Specifically, the thought estimation unit 105 inputs embeddings into a language model (LM) to generate text (i.e., sentences describing the situational judgment, decision-making, and thought processes) (for example, a Decoder-only Transformer is used).

[0067] The process of S7 will be explained while referring to the thought estimation in Figure 5.

[0068] [[Annotation]] In this invention, when training a language model (LM), the situational judgments, decision-making, and thought processes of doctors and other medical professionals are used as annotation data for the training data, so that the language model (LM) outputs the situational judgments, decision-making, and thought processes of doctors and other medical professionals. Specifically, the annotation data used when training the language model (LM) is categorized and trained by providing text for Object (visible objects), Action (actions of visible objects, etc.), Situation (current situation), Prediction (predicted next situation), Plan (next action to be taken), and Reason (reasons for arriving at Situation, Prediction, and Plan). By creating annotation text so that the same words appear between Object, Action, Situation, Prediction, Plan, and Reason, each category becomes linked and easier to learn. For example, an image recognition model or Vision Encoder capable of generating Embeddings that contribute to annotation data (i.e., categorized annotation data) as shown below is used. Image recognition models that recognize body parts (organs, etc.) or instruments generate features to output an Object. • An image recognition model that recognizes a process generates features to output Action and Situation. • Image recognition models that recognize bleeding generate features to output Situation and Prediction. The Vision Encoder generates features for all of these categories.

[0069] The following are examples of annotations. Object: Right recurrent laryngeal nerve, bipolar Action: Bipolar delamination Situation: Dissection of the tissue around the right recurrent laryngeal nerve for right recurrent laryngeal nerve dissection. • Prediction: There was no bleeding, and the dissection could be carried out without any problems. Plan: Right recurrent larynx dissection Reason: Since there has been no bleeding and the surgery is progressing smoothly, we will proceed with the right recurrent laryngeal nerve dissection. However, we must proceed with caution as there is a risk of paralysis due to recurrent laryngeal nerve damage.

[0070] The annotation example above is for the esophageal procedure. In this case, if the image recognition specific to this task can use the following method, the language model (LM) will be able to output answers for Object, Action, and Situation with high accuracy, and will also be able to output answers for Prediction, Plan, and Reason, taking these circumstances into consideration. The Vision Encoder generates arbitrary image information that includes all of these and inputs it into the language model (LM). Image recognition techniques: site recognition (recurrent laryngeal nerve), instrument recognition (bipolar), action recognition (dissection, instrument action), surgical process recognition (nerve dissection), bleeding recognition (presence or absence of bleeding)

[0071] For example, an image recognition model is an object detection model (this could be object detection that estimates the bounding box of the object to be detected from an image, or segmentation that estimates the position of the object to be detected at the pixel level from an image (the segmentation case will be explained below)). In the case of segmentation, an image is input to the image recognition model (object detection model), and the position of the object to be detected is estimated at the pixel level from the image.

[0072] [[Create a Reach]] This section explains how to create prompts. For example, if a nerve is detected as a result of image recognition, a prompt such as "A nerve is located in the lower center of the screen" can be created. Prompts can be expressed in various template sentences, such as "A nerve is visible near the lower center of the screen." The prompt text is created based on the coordinate position and likelihood of the object obtained as a result of image recognition.

[0073] As shown in [Example 1] in Figure 9, prompts can be created based on the approximate location of the object to be detected. First, divide the surgical image vertically into two sections and horizontally into three sections to create six divided regions. Next, based on the image recognition results, it is determined which of the six divided regions each segmentation class (e.g., the right recurrent laryngeal nerve class) belongs to. Then, the region containing the most segments (e.g., the right recurrent laryngeal nerve class) (e.g., the lower center) is determined to be the region where the object (in this example, the right recurrent laryngeal nerve) was detected. Based on the determination of which region it belongs to, a prompt (for example, "The right recurrent laryngeal nerve is visible near the bottom center of the screen") is created.

[0074] As shown in Figure 9 [Example 2], prompts can be created based on the relative positional relationship between the objects to be detected. First, randomly select an object from the detected objects (or select an important organ). For example, let's say the right recurrent laryngeal nerve is selected. Next, the system calculates the coordinates of the selected object (in this example, the right recurrent laryngeal nerve) and the coordinates of another detected object (e.g., bipolar forceps), as well as the closest distance between the two objects and the direction from one object to the other. A threshold is set for the distance between such two objects, and if the distance between the two objects is less than the threshold, it is determined that the objects are in close proximity. When objects are close together, a prompt (e.g., "The bipolar forceps is visible to the right of the right recurrent laryngeal nerve") is generated based on the relative positional relationship between the two objects (i.e., distance and direction).

[0075] For example, prompts can be created based on the likelihood of an object in object detection. Prompts can be created based on the maximum (or minimum or average) likelihood of the detected object. For example, the likelihood (0-1) can be divided into high (1.0-0.66), medium (0.66-0.33), and low (0.33-0), and the prompts can be changed accordingly. High: Nerves are present. Middle: Nerves may be present. Low: There is a high probability that nerves are not present.

[0076] For example, prompts can be created based on the likelihoods of multiple surgical procedures in surgical procedure recognition. By applying Softmax to predefined classes of surgical procedures (for example, surgical procedure A (likelihood: 0.38), surgical procedure B (likelihood: 0.4), surgical procedure C (likelihood: 0.12), surgical procedure D (likelihood: 0.1)), the likelihood of each class is output so that the sum of the likelihoods of each class equals 1.0. Normally, "procedure B," which has the highest likelihood, will be the recognized surgical procedure, but since the difference in likelihood between surgical procedure A and surgical procedure B is very small, a prompt with the message "Performing either procedure A or procedure B" is created.

[0077] Furthermore, it is also possible to create a message by combining the aforementioned prompts (for example, "A bipolar forceps is visible near the right recurrent laryngeal nerve, located near the bottom center of the screen. There is no bleeding.").

[0078] As mentioned above, in Example 1 of Figures 5 and 6, instead of Prompt+Tokenizer, you can use Projector to create Embeddings that will be input to the language model (LM) (in this case, the intermediate features of the output of the image recognition model's backbone are used (input to Projector)). The Projector and language model (LM), which have been trained based on the information of the image recognition results and their captions, interpret the image recognition results, which is more expressive than the method of fitting them to template prompts.

[0079] The following is a concrete example of creating a prototype from the results of image recognition.

[0080] For example, if a tumor has infiltrated the blood vessels more than expected, a wider excision or vascular reconstruction may be necessary. In this case, (1) Use the results of image recognition to represent the situation in which the tumor has invaded blood vessels. (2) Detect the location (coordinates) of tumors and blood vessels and calculate the proximity of each location. (3) Using the information from (1) and (2) above, create an Embedding that will be the input to the Language Model (LM).

[0081] For example, if unexpected and serious complications such as massive bleeding, tissue fragility, or damage to surrounding organs are encountered, the most appropriate repair method will be selected. In this case, (1) Use the results of image recognition to represent the situation in which complications such as massive bleeding, tissue fragility, and damage to surrounding organs have occurred. (2) Calculate the amount of bleeding (area of ​​bleeding detection result (segmentation)), the amount and magnitude of tissue damage (area of ​​tissue damage range (segmentation), estimation of damage amount (regression)), and the recognition of damage to surrounding organs of the target organ (presence or absence of surrounding organs other than the target organ). (3) Using the information from (1) and (2) above, create an Embedding that will be the input to the Language Model (LM).

[0082] [Reference Information / Utilization of Knowledge Data] In one embodiment of the present invention, the content to be output is changed according to the situation, using at least one of the following: reference information (e.g., pre-operative information, medical record information) and knowledge data.

[0083] [[Model Training]] Here, we will explain how to train the visual language model used in this invention. The image recognition model, Vision Encoder, and Language Model (LM) in Figure 5 are each trained independently. 1. Train an image recognition model for a specific task. This is done using supervised learning. 2. Train the Vision Encoder. This will be done through image-to-text comparison learning, such as SigLIP or CLIP. 3. Train a language model (LM). This involves pre-training using token prediction tasks, additional training using QA tasks, and reinforcement learning using RLHF (Reinforcement Learning from Human Feedback), similar to Llama. 4. The Vision Encoder is frozen to prevent training, and an image is input into the Vision Encoder. The obtained image features are input into the Projector to obtain the Embedding as output. The text annotated to the image is input into the Embedding layer of the Tokenizer and the Language Model (LM) to obtain the true image caption Embedding. Loss calculation is performed using the Projector's output Embedding and the true Embedding to perform training. 5. Perform training on the entire visual language model. An image is input into an Image Recognition Model to obtain the result of image recognition for a specific task. The image recognition result is then applied to a predefined prompt, and tokens are created from the prompt via a Tokenizer. The image is input into a Vision Encoder to obtain image features. Embeddings are then created from these features via a Projector. Query and reference information text is input into a Tokenizer to create tokens. Embeddings are created from the tokens of the query and reference information text, and the tokens created from the image recognition results, via the Embedding Layer of the Language Model (LM). The embeddings of the query and reference information text, the image recognition result embeddings, and the Vision Encoder embeddings are combined and used as input to the Language Model (LM). Only the Projector and Language Model (LM) are trained; that is, the Image Recognition Model and Vision Encoder are trained in a frozen state.

[0084] Alternatively, instead of using a generative AI (language model (LM)) to generate text as described above, an image generation model or a Grounding model that solves image recognition tasks can be used, taking arbitrary language and image information as input. In other words, by using these models, the thought estimation unit 105 can output not language, but an arbitrary generated image or the result of image recognition, instead of language.

[0085] For example, by using an image generation model in the thought estimation process, it is possible to generate images such as displaying an ideal unfolded diagram or showing incision lines.

[0086] For example, by using a Visual Grounding model in the thought estimation process, image recognition tasks can be performed using instructions in any language. In the configuration shown in Figure 5, a specific image recognition model was placed before the language model (LM), but in this configuration, the Visual Grounding model is added after the language model (LM). Language and images are input to the Visual Grounding model, which then interprets the language and images to perform image recognition (for example, displaying incision lines).

[0087] [S8 (Output Processing)] The output unit 106 changes the output format according to the image recognition results, user queries and reference information, and at least one of the situation judgment, decision-making, and thought process, and outputs at least one of the estimated situation judgment, decision-making, and thought process (for example, by visually displaying it on a screen such as a monitor or display, or by outputting it as audio to a headset, etc.).

[0088] The output unit 106 can change the output format by classifying the category of the query or the answer to the query (for example, determining whether or not it is a category related to an object (including organs, other body parts, instruments, gauze, etc.)). The output unit 106 can change the display format of the text output by the language model (LM model) according to the image recognition results. For example, the output unit 106 may prevent the text from being output at locations on the surgical image that overlap with an object included in the query or answer, or at the contour of an object on the surgical image, or conversely, it may output the text at a location close to an object. The output unit 106 can change the display format of the image recognition results depending on the content of the query or answer. For example, when the query or answer relates to an object, the output unit 106 may light up, blink, or display text on the object in the surgical image.

[0089] The method for classifying the categories of the above queries or answers (for example, determining whether they are related to an object (including organs, other body parts, instruments, gauze, etc.)) will be explained with reference to Figures 10 and 11.

[0090] Figure 10 is a diagram illustrating the classification of queries and answers according to one embodiment of the present invention. As shown in Figure 10, the similarity between the Embedding (vector) of a text collection (query database) of queries related to an object is calculated. A common method for calculating similarity between texts is to calculate the distance between texts by converting the text into Embedding (vectors) and calculating the similarity. Using this, if the similarity between the text collection (query database) of queries related to an object and the user's query is above a threshold, it can be determined that the query is related to an object, and the display format can be changed.

[0091] Figure 11 is a diagram illustrating the classification of queries and answers according to one embodiment of the present invention. As shown in Figure 11, the language model (LM) is trained to output tokens, etc., that determine the category of a query or answer. As described above, when training the language model (LM), annotation data used during the training of the language model (LM) is categorized and trained by providing text for Object (visible object), Action (action of visible object, etc.), Situation (current situation), Prediction (predicted next situation), Plan (next action to be taken), and Reason (reason for arriving at Situation, Prediction, and Plan). This annotation data is used to output answers according to each category during the training of the language model (LM), by adding special tokens or specific words (prefixes) to the output. In other words, when the language model (LM) outputs an answer, a flag indicating the category can be added to the answer. The output unit 106 can refer to special tokens attached to the answer and change the display format if the answer is one that has a token ([OBJ]) indicating an object attached to it.

[0092] If a method for determining whether a query or answer relates to such an object determines that the query or answer relates to "where to cut", the output unit 106 can display the cutting location or direction (cutting line, margin relative to the cutting line, cutting direction).

[0093] The output unit 106 may change the text depending on whether the surgical support device 10 is operating during surgery (i.e., assumed to be operating for surgeons) or whether the surgical support device 10 is operating outside of surgery (i.e., assumed to be operating for student education). The reference information may also indicate whether the surgical support device 10 is operating during surgery or outside of surgery.

[0094] The output format will consist of the surgeon's situational assessments, decision-making processes, and thought processes during surgery, as shown below. An example is provided below. These situational assessments, decision-making processes, and thought processes will be used as annotation data for the language model (LM) training data. 1. Situation assessment during incision and field of view expansion. • Determine the appropriate port location and incision (optimize the incision area considering the possibility of adhesions. Adjust the incision line based on the tumor location and surrounding anatomy). • Procedures and methods for exposing the surgical field (e.g., choosing between sharp or blunt dissection in cases of strong adhesions; determining which tissue to pull, in what direction, and with what force using forceps). 2. Situation assessment during primary operations Identification of anatomical structures (confirmation and estimation of the location and type of exposed or unexposed blood vessels, nerves, and organs covered by fat or membranes; how to deal with abnormal anatomy such as mutated blood vessels; differentiation and judgment when similar anatomical structures exist; adjustment of strategy if the extent of cancer invasion is wider than expected; evaluation of blood flow and tumor location using ICG fluorescence, ultrasound, and endoscopy; and whether or not to perform cholangiography). • Decision-making regarding resection (whether to incise certain tissues or lymph nodes, how much to resect, whether the resection or lymph node dissection is sufficiently completed, whether to preserve or resect normal tissue, whether additional resection is necessary, whether to preserve or resect important nerves (e.g., autonomic nerves in rectal cancer surgery), and how to choose the surgical technique (sharp or blunt dissection, whether to use staples or hand-sewn anastomosis)). 3. Decision-making in case of bleeding or injury • Management of bleeding (identification of the source of bleeding; determination of hemostasis method (whether to stop bleeding with direct pressure, clamp, clip hemostasis, suture hemostasis, or use an electrosurgical unit); decision-making in cases of massive bleeding (whether to immediately block major blood vessels and prioritize bleeding control, or to use an autologous blood recovery device (cell saver), etc.)) 4. Decision-making regarding reconstruction and anastomosis • Selection of anastomosis method (end-to-end, end-to-side, or side-to-side anastomosis; hand-sewn or stapled anastomosis; whether to assess blood flow with ICG fluorescence and expand the resection area if blood flow at the anastomosis site is poor). • Assessment of leakage risk at the anastomosis site (whether to perform reanastomosis, switch reconstruction routes, place prophylactic drains, or consider bypasses or mitigation measures if there are signs of insufficient blood flow) 5. Decision-making in unexpected situations • Strategic changes based on intraoperative discoveries (If unexpected metastases such as peritoneal dissemination are discovered during surgery, should the radical surgery be continued, the tumor left unresected, or a palliative surgery be performed? What should be done if another disease is discovered?) • Equipment malfunctions and unforeseen circumstances (For example, if a surgical robot or laparoscope cannot be used, should the procedure be changed to open surgery or should the surgery be continued with equipment replacement? Should an X-ray be taken when the gauze runs out?) 6. Thought processes during surgery • The thought process leading to situational assessment and decision-making (for example, recognizing the anatomy during surgery, such as "the branching of blood vessels is abnormal," making inferences such as "it looks like it will bend further ahead," making a judgment such as "continuing in this direction will cause massive bleeding," and taking action such as "changing the dissection line"—a series of processes from observation → inference → judgment → action).

[0095] [Other output formats] • Enhanced detailed labeling and interpretation In one embodiment of the present invention, an AI confidence score (for example, ambiguity when the recognized object belongs to multiple categories) is displayed for each object, allowing the user to make corrections. • Support for time-series data In one embodiment of the present invention, the movement of an object is tracked against a series of surgical images (e.g., frames of a video), and a description of the changes is displayed. • Integration with generative AI One embodiment of the present invention automatically generates related images and stories based on recognized objects and scenes, enabling their use in augmented reality (AR) and virtual reality (VR) environments. • Voice support One embodiment of the present invention provides audio descriptions of surgical images, improving accessibility for visually impaired individuals. • Addition of background knowledge One embodiment of the present invention provides information from Wikipedia and scientific papers related to the recognized object, enabling a deeper understanding of the context.

[0096] [S9 (Interaction Processing)] The interaction processing unit 107 changes the output format of the information output by the output unit 106 in response to user operations (instructions entered by the user using the input device 11, etc.). For example, the interaction processing unit 107 generates explanations in multiple languages ​​such as Japanese, English, and French, and allows switching between them with a click (and the prompt is configured to translate into multiple languages). For example, the interaction processing unit 107 reads aloud the explanation of the surgical image when a button is clicked.

[0097] [S10 (Data Recording)] The data storage unit 108 records user-entered queries and reference information, image recognition results, situational judgments, decision-making processes, and thought processes. The recorded information is used in a series of processes from the next time the user enters a query until the image recognition results and answers are output. For example, it may be used as reference information or in image recognition that requires time-series processing.

[0098] <Interactive display> In one embodiment of the invention, the interaction processing unit 107 of the surgical support device 10 displays detected objects (detected objects (e.g., organs or other body parts, instruments)) or text obtained from the image recognition results (i.e., text describing the situational judgment, decision-making, and thought process) output by the output unit 106, highlighting them in response to user operations.

[0099] For example, as shown in Figure 12 under [Object Description], when a user clicks on a specific area of ​​an image on the screen (e.g., a body part (organ, etc.), an object such as a device), the corresponding description is highlighted. Specifically, when the location information of an object is obtained through image recognition, and the user clicks on an area of ​​an object such as a body part (organ, etc.) or an object such as a device on the screen (e.g., clicking on the area of ​​the right recurrent laryngeal nerve), the language model (LM) is prompted with "Please describe the right recurrent laryngeal nerve," and the output is displayed (e.g., "This is the right recurrent laryngeal nerve. It is part of the vagus nerve and is a nerve that circles the subclavian artery in the thoracic cavity").

[0100] For example, you can provide a description of the object in the current situation, as shown in Figure 12 [Description of the object in the current situation]. You can input the prompt "Describe the current situation of the right recurrent laryngeal nerve" into the language model (LM) and display the output (for example, "This is the right recurrent laryngeal nerve. There is a risk of paralysis because dangerous behavior was observed when the surrounding tissue was detached").

[0101] Furthermore, the interaction processing unit 107 of the surgical support device 10 may prioritize outputting "explanation of the current situation" during surgery, and prioritize outputting "general explanation" when playing back recorded surgical images.

[0102] For example, as shown in Figure 13, when a user clicks on a specific part of the text on the screen, the corresponding area of ​​the image is highlighted. Specifically, when the position information of an object has been obtained through image recognition, when a user clicks on a part of the text on the screen (such as an organ or an object), for example, clicking on the part "right recurrent laryngeal nerve," the image recognition results are used to display the position of that object in the image. For example, a transparent mask may be displayed to highlight the area of ​​the right recurrent laryngeal nerve, or its outline may be colored, or these transparent masks and outlines may be made to blink. The text of the clicked object and the highlighted display of the object in the image may also be displayed in correspondence with each other and colored (for example, displayed in the same color).

[0103] <Text Style> In one embodiment of the present invention, the thought estimation unit 105 of the surgical support device 10 changes the style of text (i.e., sentences of situational judgment, decision-making, and thought processes) to provide at least one of a general explanation, a specialized (medical, engineering, etc.) explanation, and an emotional (artistic, psychological, etc.) explanation. Specifically, the thought estimation unit 105 can generate multiple interpretations for the same surgical image, such as "general explanation (e.g., lymph node dissection around the right recurrent laryngeal nerve is being performed.)", "specialized explanation (e.g., bipolar dissection is being performed on the lymph nodes lateral to the right recurrent laryngeal nerve. Please perform the dissection carefully.)", and "emotional explanation (e.g., nerves are susceptible to complications from damage, so extreme care is required during surgery.)". A user interface is provided for inputting the desired explanation method, and prompts indicating how the explanation should be provided are input to the language model (LM) to generate a response. Template prompts are defined in advance to change the input to the language model (LM). Specifically, during the training of the language model (LM), a style transfer module is added after the output of the language model (LM) to train it to solve the task of changing the style of text.

[0104] For example, if the user wants a technical explanation, the prompt "What are you doing?" can be changed to "What are you doing? Please explain it in technical terms." For example, if the user wants an emotional explanation, the prompt "What are you doing?" can be changed to "What are you doing? Please also explain how a surgeon feels during surgery."

[0105] <Adding supplementary information to the answer> In one embodiment of the present invention, supplementary information such as disclaimers is added before and after the answer (leaving the final judgment to the operator). When using AI, its answer (output) may make plausible mistakes such as hallucination, and such answers can sometimes be fatal. By adding supplementary information such as disclaimers before and after the answer in this way, the operator can treat the answer as merely a reference.

[0106] For example, the following supplementary information is available. (1) When describing an irreversible (cannot be redone) procedure, provide supplementary information in your response. (2) Provide at least one answer with a score above the threshold, and include the score as supporting information. (3) Provide supplementary information instructing the user to repeat the same question (query) using different wording.

[0107] Specifically, the surgical support device 10 determines whether a surgical procedure is irreversible based on surgical images, queries, and reference information, and if it determines that it is an irreversible procedure, it adds supplementary information to the answer. The surgical support device 10 also calculates a score (confidence score) for each answer and adds the score as supplementary information to at least one answer whose score is above a threshold. In addition, the surgical support device 10 adds supplementary information to any answer that instructs the system to ask the same question repeatedly using different wording.

[0108] This section explains when to provide supplementary information in response to irreversible (cannot be corrected) procedures. For example, it explains the case where the question is "What should be done next?" (1) Proposals such as "removing the target object" are irreversible and can be fatal if they result in a false notification, therefore, if such reasoning is made, no notification will be sent. (2) Proposals such as "excise the target object" are irreversible and can be fatal if they are misreported. Therefore, if such reasoning is made, a disclaimer should be generated and notified at the beginning or end. This will allow the surgeon to make a final confirmation before excision.

[0109] For example, in right recurrent laryngeal nerve lymph node dissection during robotic esophageal cancer surgery, the resection of the right recurrent laryngeal nerve itself must be avoided; only the surrounding tissue must be resected. Once "resection" is performed in surgery, it is impossible to reverse the procedure, and irreversible complications may occur. Therefore, if a response includes a suggestion regarding the procedure, such as "resection," we will always notify the patient to check the surrounding anatomical structure, that there is no going back, and that the final decision rests with the surgeon.

[0110] [Determining whether supplementary information needs to be provided] Here, we will explain how to determine whether or not to provide supplementary information. One case where supplementary information is necessary is to determine whether the instructions relate to irreversible procedural actions (you may use either method a or b below). a. Utilization of knowledge databases • Pre-register words and phrases related to irreversible surgical procedures in the knowledge database. • Searching between generated text and irreversible manual manipulation of words and sentences in the knowledge database. • If a search yields a result, it is determined that the operation is an irreversible procedure. b. Other (variant) 1. Obtain the score of words related to irreversible procedural operations from the attention map of the language model's generated documents, and determine whether the procedural operation is irreversible based on the score. 2. When training a language model (LM), special tokens are used to generate the output of irreversible procedural operations. <information>Train the system to assign the necessary information. 3. The generated text is then analyzed using a language processing model that determines whether it refers to irreversible surgical procedures.

[0111] [Generating responses with supplementary information] If the surgical support device 10 determines that an answer should be supplemented with auxiliary information, it will add the auxiliary information and regenerate the answer sentence. For example, if the sentence generated by the language model (LM) is "Cut outside the nerve," it will add the auxiliary information as follows and present the regenerated sentence (you may use any of the following methods a, b, or c). a. Prepare a template statement and add it to the beginning or end of the generated document. Cut the outside of the nerve. However, the procedure described in the answer is dangerous, so please use your own judgment. Please use my answer as a reference only and perform the procedure with caution. Cut outside the nerve. b. Prompt the language model (LM) to regenerate the sentence by adding auxiliary information. For example, prompt the language model (LM) to regenerate the sentence "Cut the outside of the nerve" by adding auxiliary information. • Cut outside the nerve. However, be sure to confirm the nerve's exact location. The surgeon should make the decision to remove the nerve. After carefully confirming the location of the nerve, the surgeon should cut outside the nerve at their discretion. c. Notify as an audio message providing supplementary information. • Information audio • Beep sound d. Use a pre-trained language model (LM) to generate supplementary text. Based on a method for determining irreversible procedural operations, a language model (LM) trained to output supplementary text when it is determined that input related to an irreversible procedural operation has been made is used. In this case, the determination of whether to provide supplementary information and the processing of adding supplementary information are included in the response generation process.

[0112] This section describes a scenario where at least one answer with a score above a threshold is provided (for example, by reading the answer aloud), and the score is provided as supplementary information (for example, by displaying the score). The questioner cannot ascertain the degree of confidence with which the answer was given. By notifying the degree of confidence as supplementary information, the questioner can ascertain the degree of confidence with which the answer was given. In robotic esophageal cancer surgery, during lymph node dissection around the right recurrent laryngeal nerve, there are situations where the lateral side of the right recurrent laryngeal nerve should be cut, and situations where the medial side should be cut. Depending on the degree of confidence with which the answer was given, the surgeon can interpret the answer and proceed with the surgery accordingly. Furthermore, even if the answer was incorrect, if the degree of confidence was displayed, the surgeon could interpret the incorrect answer and make the correct decision.

[0113] [Output of multiple answer candidates and their likelihood] 1. The output of the language model (LM) is calculated using the Softmax function between all the tokens in the dictionary the language model (LM) possesses and the single token of the language model (LM)'s output, for each output (1 token (1 word)). In other words, the likelihood is calculated for all the words included in the dictionary at this time. 2. To generate multiple sentences, the top K tokens are obtained from the dictionary tokens calculated by Softmax, starting with the tokens with the highest likelihood. 3. The language model (LM) infers the next token based on the token it has inferred. It predicts the top K of the next token based on the top K token it has obtained. By repeating this process and generating documents, it is possible to generate multiple sentences. 4. The language model (LM) calculates the likelihood of a sentence as a whole by using the average of the log-likelihoods of all the output tokens.

[0114] [Generating responses with supplementary information] The surgical support device 10 regenerates the response text by adding auxiliary information. It then adds likelihood information and presents the regenerated text as follows (you may use any of the following methods a, b, or c). a. Prepare a template document and add likelihood information to the beginning or end of the generated document. • Cut the outside of the nerve. However, make sure you know the nerve's exact location. The score is "***". b. Prompt the language model (LM) to regenerate the sentence by adding likelihood information. For example, prompt the language model (LM) to regenerate the sentence "Cut the outside of the nerve." by adding a confidence level of "***" to the sentence. Cut outside the nerve. However, make sure you know the nerve's exact location. The confidence level of this answer is "***". c. Prompt the language model (LM) to regenerate the sentence taking likelihood information into account. For example, prompt the language model (LM) to regenerate the sentence "Cut the outside of the nerve." by adding the confidence level "***" for nerve detection (image recognition result). • Cut outside the nerve. However, the likelihood of detecting the nerve is low ("***"). The accuracy of the response may be low.

[0115] This section explains how to provide supplementary information instructing the same question (query) to be asked repeatedly using different wording. The questioner, such as the surgeon, is instructed to ask the same question again using different wording if the question or answer relates to the procedure being performed. This allows the surgeon to confirm consistency in the answers. (Example: In response to the question, "What will you do next?", the answer might be, "Please perform a gastrectomy. For confirmation, please ask the same question using different wording.")

[0116] [Generating responses with supplementary information] If the surgical support device 10 determines that a question or answer relates to the target action, it regenerates the answer text to prompt the surgeon to ask a follow-up question. The regenerated text is presented with a sentence prompting a follow-up question added as follows (either method a or b below may be used): a. Prepare a template text and add a sentence prompting a follow-up question at the beginning or end of the generated document. Please perform a gastrectomy. However, please ask the same question using different wording. • Perform a gastrectomy. However, ask the same question using different wording to ensure consistency in the answer. b. Prompt the language model (LM) to regenerate the sentence by adding a sentence that prompts a follow-up question. For example, prompt the language model (LM) to regenerate the sentence "Please perform a gastrectomy" by adding a sentence that asks the same question again in a different way.

[0117] Examples of supplementary information are shown below. (1) Explanation of points to be noted when performing the procedure in question (e.g., "Please pay close attention to the location of the nerve" when performing a perineurectomy) (2) Explanation of adverse events that may occur if the procedure in question is performed incorrectly (e.g., when performing a perineurectomy, "If the wrong site is removed, nerve paralysis may occur") (3) Explanation that the surgeon will bear ultimate responsibility (e.g., "The use of this system in this surgery is intended to assist existing standard procedures, and the final decision on diagnosis and treatment will be left to the surgeon's judgment.") (4) An explanation that there are certain limitations to the accuracy and safety of the system (e.g., "The algorithm and measurement accuracy of this system may contain certain errors, and the operator should be fully aware of these limitations and use it as an auxiliary tool"). (5) An explanation that the system will be used in conjunction with standard procedures and will not be entirely dependent on the system (e.g., "The use of this system in this surgery is intended to supplement existing standard procedures," etc.) (6) Explanation of how to provide appropriate explanations to patients and their families and obtain their consent (e.g., "Regarding the use of this system, we will provide a thorough explanation to the patient or their family in advance and obtain their consent before proceeding.") (7) Explanation of compliance with laws, regulations and ethical guidelines (e.g., "Use of this system will be in compliance with applicable laws, regulations and ethical guidelines, and patient safety will be the top priority")

[0118] <Changes to how to respond in emergencies> In one embodiment of the present invention, the method of answering urgent questions (queries) is changed so that the surgeon or other medical personnel can immediately understand and recognize them. During surgery, the surgeon's hands are occupied, and especially in emergencies, they cannot take their eyes off the patient, so there is no time to read or listen to long generated sentences during surgery. By changing the method of answering in emergencies in this way, medical personnel can immediately understand and recognize the situation and how to deal with it in an emergency.

[0119] Specifically, the surgical support device 10 determines whether or not it is an emergency based on the surgical image, query, and reference information, and if it determines that it is an emergency, it changes the way it responds.

[0120] This section explains the criteria for making decisions in emergency situations. 1. Assign special tokens to annotation data to indicate danger. To add special tokens to training data, for example, data containing sentences in the annotation text such as "It is dangerous, so do this immediately" should be assigned special tokens. <warning>This assigns a special token to the annotation data. Specifically, by calculating the similarity between the annotation data and a set of sentences representing dangerous or emergency situations, a special token can be assigned to the annotation data. 2. Special Tokens <warning>Annotated data is used to train the Vision Encoder and LM (Language Model). 3. Further special tokens for LM (Language Model) <warning>If this is assigned, the system will perform additional learning tasks such as shortening the explanation, limiting the answer to keywords only, and adding proper nouns. 4. Using the Vision Encoder and LM (Language Model) trained through the above process, natural language is obtained as output. Due to the training in step 2, the Vision Encoder and LM (Language Model) will output special tokens in the event of danger. <warning>This is added. Furthermore, through learning as described in 3, the LM (Language Model) can change the language information it outputs when a dangerous situation occurs. 5. LM (Language Model) special tokens <warning>When output is generated, the output format of the user interface is changed (specifically, by using variations in volume, beeping to notify, etc.).

[0121] The following are examples of how to change the way you answer. (1) Summarize the answer into a short sentence (e.g., "There is bleeding in the left mesenteric artery. Use clips to stop the bleeding." "There is a perforation, so proceed to laparotomy.") (2) Limit your answers to keywords only (e.g., "Bleeding, Suction") (3) Do not give multiple instructions at once; give one clear instruction at a time (e.g., instead of saying, "Cut this blood vessel, then stop the bleeding and insert the drain," say, "Ligate the blood vessel. If OK, then insert the drain.") (4) Add proper nouns (e.g., always include specific names of anatomical parts, etc.) (5) Use variations in volume (e.g., give instructions in a calm tone during normal times, and give instructions in a loud voice during emergencies). (6) Give instructions by calling out the name (e.g., give instructions including the name of the surgeon or assistant).

[0122] Furthermore, if a classifier that recognizes anatomical structures or surgical procedures results in an answer that is absolutely irreversible, it may be automatically deleted. For example, if the classifier recognizes that a gastrectomy has already been completed, and the answer to the question "What should be done next?" is "Perform a gastrectomy," then the output should not be displayed (because a gastrectomy is usually not performed twice).

[0123] The following are examples of urgent situations and proposed solutions. • Bleeding: For massive bleeding and major vascular damage, use compression hemostasis, electrocautery, clips, sutures, use of vascular sealing devices, arrange for emergency blood transfusions (red blood cells, fresh frozen plasma, platelets), immediately block blood flow with forceps, perform vascular anastomosis or bypass surgery, and prepare for artificial blood vessel grafting, etc. • Circulation: For cardiac arrest, hypotension, hypertension, and air embolism, initiate cardiac massage (CPR), administer intravenous adrenaline, administer oxygen and mechanical ventilation, ensure sufficient circulating blood volume, identify bleeding sites and stop bleeding, and administer intravenous fluids and vasopressors (noradrenaline, phenylephrine). Adjust anesthetics. • Respiration: For airway obstruction, pneumothorax, and hypoxemia, secure the airway (bag-mask ventilation, reintubation), administer steroids (in case of laryngeal edema), immediately perform chest tube drainage, needle puncture to remove air, administer oxygen, adjust mechanical ventilation, and reconfirm tracheal intubation. • Neurological: For intraoperative awareness and cerebral infarction, additional anesthetics, confirmation of BIS monitoring (electroencephalogram measurement), rapid postoperative CT / MRI, blood pressure management, and consideration of anticoagulation therapy. • Infection: For intraoperative sepsis and intestinal perforation, wound irrigation and antibiotic administration; in cases of septic shock, norepinephrine administration, suturing and repair of the perforation site, intraperitoneal irrigation, and broad-spectrum antibiotic administration. Anesthesia: For malignant hyperthermia and anaphylaxis, dantrolene is administered, followed by cooling measures (ice-cold infusion / ice pack), intravenous adrenaline, and administration of steroids and antihistamines. • Instruments: In case of surgical instrument failure or robot malfunction, prepare backup instruments, interrupt the surgery and select an alternative technique, switch to manual operation, and proceed to open surgery.

[0124] <Example query> This includes the location of objects that appear during surgery (organs, instruments, gauze, bleeding, fluids such as bile and pancreatic juice, sutures), actions performed by the surgeon, assistants, and nurses during surgery (excision, coagulation, hemostasis, etc.), the progress of the surgery (lymph node dissection in progress, completion of gastrectomy, etc.), the reasons for events and actions (reasons why the surgeon performed an excision, etc.), suggestions and points to note regarding actions (the next incision line to be dissected, points to be careful about in the next action), evaluation of the technique, etc. In addition, the following items are also included.

[0125] (1) Questions about anatomy and physiology (What is the detailed anatomy of the area to be operated on? How are the blood vessels and nerves arranged? What is the function of this organ? What are the nearby important structures (blood vessels, nerves, lymph nodes, etc.)? What is the boundary between the area to be removed and the area to be left? How much bleeding is physiologically acceptable? Is the color and hardness of a particular tissue normal or pathological? etc.) (2) Questions regarding surgical techniques (How to hold the forceps correctly? How much force is appropriate? Where and how should the incision be made for the safest? To what depth should the incision be made? What is the appropriate interval and strength of the sutures? What is the appropriate choice of knot (buried suture, continuous suture, simple ligation, etc.)? What is the appropriate method for dissection? What is the direction and angle when excising? How can bleeding be minimized? How should the tissue be held to prevent damage? Which device (electrosurgical scalpel, ultrasonic scalpel, etc.) should be used? Which forceps are appropriate for grasping the tissue? etc.) (3) Questions regarding the use of surgical instruments and equipment (What is the official name of this instrument? What is the correct way to hold and use this instrument? When is it appropriate to use a specific instrument? What is the appropriate wattage setting for this electrosurgical unit? What are the differences between types and characteristics of sutures? Which type of needle (round needle, square needle, inverted triangular needle, etc.) should I choose? What is the correct method and type of drain to insert? What are the appropriate uses and indications for tissue adhesives and hemostatic agents?) (4) Questions regarding anesthesia and general management (What types of anesthesia are used and how do they affect the surgery? How should changes in blood pressure and pulse be managed during surgery? At what blood loss is a blood transfusion necessary? How should body temperature be managed during surgery? What measures should be taken if there are problems with airway management? How should postoperative pain be managed? How should fluid management (infusion and electrolyte adjustment) be managed during surgery?) (5) Questions regarding intraoperative complications and trouble management (How should bleeding be stopped if an artery is damaged? What is the best course of action in the case of venous damage? How should unexpected adhesions be dealt with? How should instruments be handled in case of malfunction? What should be done if a tumor is ruptured during surgery? How should the patient's blood pressure drop rapidly during surgery? What should be done if sudden pneumoperitoneum (air leakage during laparoscopic surgery) occurs? What should be done if an instrument is dropped or sterilization fails?) (6) Questions regarding postoperative management (What are the risks of postoperative complications (SSI, bowel obstruction, anastomotic leakage, etc.)? What is the appropriate timing for extubation? What is the best method for wound management? What are the criteria for resuming oral feeding after surgery? How should postoperative drains be managed? What are the discharge criteria and frequency of follow-up? Is postoperative antibiotic administration necessary?) (7) Questions regarding the patient's condition and surgical indications (What were the reasons for this patient being a candidate for surgery? Were there any other options? What surgical options were available? What points should not be overlooked in the preoperative evaluation? If the patient had complications, how should the surgical procedure be changed?) (8) Questions regarding the selection and rationale for the surgical procedure (Why was this surgical procedure (open surgery vs. laparoscopy vs. robotic surgery) chosen? What other surgical procedures could have been used for similar cases? What is the evidence for this surgical procedure (success rate and complication rate)? What are the reasons for changes in the surgical procedure (modified surgery or new approaches)? Why do surgical procedures differ among facilities and surgeons even for the same disease?) (9) Questions regarding medical safety and ethics (What should be explained during preoperative informed consent? To what extent should the family be informed if an unexpected situation occurs during surgery? How should medical malpractice be handled? How should a patient request DNR (Do Not Resuscitate)? What ethical considerations should be taken when a resident intervenes during surgery for educational purposes?) (10) Questions on teamwork and communication during surgery (How to accurately understand the surgeon's intentions? How to ask questions at the appropriate time during surgery? How to coordinate smoothly with nurses? How to coordinate with anesthesiologists? How to manage stress during surgery? What are some tips for maintaining concentration during surgery?)

[0126] <Input other than surgical images> In one embodiment of the present invention, not only surgical images but also other information may be acquired and input into the visual language model.

[0127] (1) Visual characteristics specific to the surgery Anatomical structures and surgical procedures: Changes in the location and shape of anatomical structures such as the recurrent laryngeal nerve, pancreas, stomach, and colon; progress of surgical procedures such as "dissection of lymph node 6" and the completion status of those procedures. Operating room environment: Medical staff wearing sterile blue or green surgical gowns, gloves, and masks. Surgical instruments: Types of instruments used in laparoscopic, thoracoscopic, and robot-assisted surgery, such as electrosurgical units, bipolar forceps, monopolar forceps, Potts scissors, suction tubes, and sealing devices; instrument location, operation, changes in frequency of use, timing of use, response to instrument operation (spark, output, smoke, discharge volume, quality, and color of aspirated fluid), instrument malfunctions, sutures, etc. Patient: Changes in the patient's skin or face, changes in vital signs during and before / after surgery. Blood: Quantity and quality of blood, gauze, etc. Feedback information from robotic surgery, etc.: range of motion, mechanical information, arm position (2) Context information Marks and logos: Hospital names and logos of medical institutions. Healthcare professionals: Doctors and nurses (3) Metadata Shooting data: EXIF ​​information of the image, etc. File name and related data: Image file name, etc. (4) Verification with external information Similar Image Search: Inference from surgical images on the internet (5) Patient information and surgical information Patient's gender, age, cancer stage, blood test results, imaging tests such as CT, MRI, and endoscopy, presence or absence of preoperative treatment (chemotherapy, radiation therapy, etc.), medical history, past medical history, lifestyle, family history, height, weight, BMI, blood type, allergy history, surgical procedure, surgical instruments used, operating room, scheduled surgery time, attending physician, presence or absence of blood transfusion, presence or absence of drain insertion, abnormalities or complications during surgery, type of surgical robot, console operation time, use of 3D navigation, application of real-time image guidance, differences from planned surgery, omissions and risks in preoperative preparation, etc.

[0128] <Effects> In one embodiment of the present invention, it is possible to understand what kind of situational judgments and decisions skilled doctors make during surgery, and to understand the thought process that leads to those situational judgments and decisions.

[0129] <Hardware Configuration> Figure 14 is a hardware configuration diagram of a surgical support device 10 according to one embodiment of the present invention. The surgical support device 10 may include a control unit 1001, a main memory unit 1002, an auxiliary memory unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0130] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003.

[0131] The main memory unit 1002 includes non-volatile memory (ROM (Read Only Memory)) and volatile memory (RAM (Random Access Memory)). The ROM stores various programs, data, etc., necessary for the control unit 1001 to execute various programs installed in the auxiliary memory unit 1003. The RAM provides a work area that is expanded when the various programs installed in the auxiliary memory unit 1003 are executed by the control unit 1001.

[0132] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when various programs are executed.

[0133] The input unit 1004 is an input device in which the operator of the surgical support device 10 inputs various instructions to the surgical support device 10.

[0134] The output unit 1005 is an output device that outputs information such as the internal status of the surgical support device 10.

[0135] The interface unit 1006 is a communication device for connecting to a network and communicating with other devices.

[0136] The embodiments disclosed above include, for example, the following aspects: (Note 1) The image acquisition unit acquires surgical images, An information acquisition unit that acquires queries and reference information from the user, A thought estimation unit that performs image recognition on the surgical image and estimates at least one of the situational judgment, decision-making, and thought processes leading to the situational judgment or decision-making, using the query from the user and the reference information, An output unit that changes the output format according to the results of the image recognition, the user's query and the reference information, and at least one of the situation judgment, the decision and the thought process, and outputs at least one of the situation judgment, the decision and the thought process to the user. Equipped with surgical support devices. (Note 2) The surgical support device according to Appendix 1, wherein the thought estimation unit generates feature quantities from the surgical images that serve as the basis for at least one of the situation judgment, the decision-making process, and the thought process. (Note 3) The surgical support device according to Appendix 1, wherein the thought estimation unit generates feature quantities from the surgical image that serve as the basis for at least one of the situation judgment, the decision-making process, and the thought process by using one or more Vision Encoders that have learned image and text embedding representations, and image recognition models that perform a given surgical image recognition task such as organ detection, instrument detection, surgical process recognition, action recognition, and bleeding detection. (Note 4) The surgical support device according to any one of the appendices 1 to 3, wherein the thought estimation unit inputs the features of the surgical image, the features of the image recognition results, the user's query, and the reference information to a language generation model and outputs at least one of the situation judgment, the decision-making process, and the thought process in natural language. (Note 5) The thought estimation unit is trained using categorized annotation data, as described in Appendix 4, for the surgical support device. (Note 6) The surgical support device according to any one of the appendices 1 to 3, wherein the output unit outputs at least one of the situation judgment, the decision-making process, and the image recognition result. (Note 7) The surgical support device as described in Appendix 6, wherein, if the user's query concerns a specific object or event, the output unit displays the image recognition result on the image. (Note 8) If the user's query is a query concerning the detected object obtained as a result of the image recognition, The output unit is a surgical support device according to Appendix 7, which highlights and displays the detected object in the surgical image or text. (Note 9) If the user's query is a query regarding the incision location, The output unit is a surgical support device as described in Appendix 7, which displays the incision position in the surgical image with emphasis. (Note 10) The surgical support device according to any one of the appendices 1 to 3, wherein the output unit changes the display position of at least one of the situation judgment, the decision-making process, and the thought process based on the results of the image recognition. (Note 11) The surgical support device according to Appendix 10, wherein the output unit displays at least one of the situation judgment, the decision-making process, and the thought process so as not to overlap with the detected object obtained as a result of the image recognition. (Note 12) The surgical support device according to Appendix 10, wherein the output unit displays at least one of the situation judgment, the decision-making process, and the thought process near the detected object obtained as a result of the image recognition. (Note 13) The surgical support device according to Appendix 6, further comprising an interaction processing unit that highlights and displays detected objects or text obtained from the image recognition results or text in response to user operations on the image recognition results and text output by the output unit. (Note 14) The surgical assistance device according to Appendix 4, wherein the thought estimation unit regenerates the query in accordance with the result of the image recognition, the query from the user and the reference information, and at least one of the situation judgment, the decision-making process, or modifies the style of the natural language output by using a language model that has been trained to explicitly change the output, or by changing the output parameters of the language model. (Note 15) The thought estimation unit is a surgical support device as described in Appendix 14, wherein the reference information relates to a specific object or event, and the style of natural language is changed according to the state of the object or event. (Note 16) If the aforementioned reference information indicates the user's skill level, The output unit is a surgical support device as described in Appendix 15, which simplifies or elaborates the output text according to the user's skill level. (Note 17) If the aforementioned reference information indicates whether the surgical support device is operating during surgery or outside of surgery, The output unit, according to the operating conditions, simplifies or modifies the output text, as described in Appendix 15, in the surgical support device. (Note 18) The surgical support device according to Appendix 14, wherein the thought estimation unit modifies the text style to provide at least one of a general explanation, a technical explanation, and an emotional explanation. (Note 19) A surgical assistance device according to any one of the appendices 1 to 3, further comprising a learning unit that applies the results of the image recognition described above to a template text, converts it into input for a language generation model, and trains the language generation model. (Note 20) The system includes an auxiliary information addition determination unit that determines whether or not to add auxiliary information to the output in response to the user's query and / or the output of the thought estimation unit. A surgical support device according to any one of the appendices 1 to 3, which changes the output of the thought estimation unit according to the determination result of the auxiliary information addition determination unit. (Note 21) The surgical support device according to Appendix 20, wherein the auxiliary information addition determination unit determines to add auxiliary information to the response if the user's query and / or the output of the thought estimation unit relate to an irreversible procedural operation. (Note 22) The surgical support device according to Appendix 20, wherein when adding auxiliary information, the thought estimation unit adds the likelihood of the image recognition result of the thought estimation unit and the likelihood of generating language information as auxiliary information. (Note 23) The surgical support device according to Appendix 20, wherein the thought estimation unit adds auxiliary information that instructs the response to repeatedly ask the same question in different expressions when adding auxiliary information. (Note 24) The system includes an emergency status determination unit that determines whether the surgical image used as input is in an emergency state, using the user's query and / or information obtained during the processing of the thought estimation unit. A surgical support device according to any one of the appendices 1 to 3, which changes the output of the thought estimation unit and the display format of the output unit according to the determination result of the auxiliary information addition determination unit, in accordance with the determination result of the emergency condition determination result. (Note 25) If the emergency condition determination unit determines that an emergency condition exists, the thought estimation unit outputs a proper noun or keyword and outputs a short sentence that does not contain multiple instructions in a single sentence, as described in Appendix 24. (Note 26) If the emergency condition determination unit determines that an emergency condition exists, the thought estimation unit outputs the names of the surgical participants at the time of instruction. This is the surgical support device as described in Appendix 24. (Note 27) The surgical support device described in Appendix 24, wherein, when the emergency condition determination unit determines that an emergency condition has occurred, the output unit outputs sound with varying intensity. (Note 28) Surgical support device, Obtaining surgical images, To obtain queries and reference information from users, The process involves performing image recognition on the surgical image, and using the user's query and the reference information to estimate at least one of the situational judgment, decision-making, and the thought process leading to the situational judgment or decision-making regarding the surgical image. The output format is changed according to the results of the image recognition, the user's query and the reference information, and at least one of the situation judgment, the decision and the thought process, and at least one of the situation judgment, the decision and the thought process is output to the user. A program that executes the command. (Note 29) A method performed by a surgical support device, Steps to obtain surgical images, Steps to obtain user queries and reference information, The steps include: performing image recognition on the surgical image, and using the user query and the reference information to estimate at least one of the situational judgment, decision-making, and the thought process leading to the situational judgment or decision-making regarding the surgical image; The steps include changing the output format according to the results of the image recognition, the user's query and reference information, and at least one of the situation judgment, the decision and the thought process, and outputting at least one of the situation judgment, the decision and the thought process to the user; A method that includes this.

[0137] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of Symbols]

[0138] 1. Surgical support system 10 Surgical support equipment 11 Input devices 12 Output device 20 Imaging device 101 Image acquisition unit 102 Information acquisition department 103 Knowledge Database Processing Unit 104 Knowledge Database Storage Unit 105 Thought estimation part 106 Output section 107 Interaction Processing Unit 108 Data Storage Unit 1001 Control Unit 1002 Main memory 1003 Auxiliary storage unit 1004 Input section 1005 Output section 1006 Interface section< / warning> < / warning> < / warning> < / warning> < / warning> < / information>

Claims

1. The image acquisition unit acquires surgical images, An information acquisition unit that acquires queries and reference information from the user, A thinking estimation unit that performs image recognition on the surgical image and estimates at least one of the situational judgment, decision-making, and thought processes leading to the situational judgment or decision-making, using the query from the user and the reference information. An output unit that changes the output format according to the results of the image recognition, the query from the user and the reference information, and at least one of the situation judgment, the decision-making process and the thought process, and outputs at least one of the situation judgment, the decision-making process and the thought process to the user. Equipped with surgical support devices.

2. The surgical support device according to claim 1, wherein the thought estimation unit generates feature quantities from the surgical images that serve as the basis for at least one of the situation judgment, the decision-making process, and the thought process.

3. The surgical support device according to claim 1, wherein the thought estimation unit generates feature quantities from the surgical image that serve as the basis for at least one of the situation judgment, the decision-making process, and the thought process by using one or more Vision Encoders that have learned image and text embedding representations, and image recognition models that perform a given surgical image recognition task such as organ detection, instrument detection, surgical process recognition, action recognition, and bleeding detection.

4. The surgical support device according to any one of claims 1 to 3, wherein the thought estimation unit inputs the features of the surgical image, the features of the image recognition results, the query from the user, and the reference information to a language generation model and outputs at least one of the situation judgment, the decision-making process, and the thought process in natural language.

5. The surgical support device according to claim 4, wherein the thought estimation unit is trained using categorized annotation data.

6. The surgical support device according to any one of claims 1 to 3, wherein the output unit outputs at least one of the situation judgment, the decision-making process, and the image recognition result.

7. The surgical support device according to claim 6, wherein if the user's query concerns a specific object or event, the output unit displays the image recognition result on the image.

8. If the user's query is a query concerning the detected object obtained as a result of the image recognition, The surgical support device according to claim 7, wherein the output unit highlights and displays the detected object in the surgical image or text.

9. If the user's query is a query regarding the incision location, The surgical support device according to claim 7, wherein the output unit displays the incision position in the surgical image with emphasis.

10. The surgical support device according to any one of claims 1 to 3, wherein the output unit changes the display position of at least one of the situation judgment, the decision-making process, and the thought process based on the results of the image recognition.

11. The surgical support device according to claim 10, wherein the output unit displays at least one of the situation judgment, the decision-making process, and the thought process in a manner that does not overlap with the detected object obtained as a result of the image recognition.

12. The surgical support device according to claim 10, wherein the output unit displays at least one of the situation judgment, the decision-making process, and the thought process near the detected object obtained as a result of the image recognition.

13. The surgical support device according to claim 6, further comprising an interaction processing unit that highlights and displays a detected object obtained from the image recognition results or the text in response to the user's operation on the image recognition results and text output by the output unit.

14. The surgical assistance device according to claim 4, wherein the thought estimation unit changes the style of the natural language output by regenerating the query in accordance with the result of the image recognition, the query from the user and the reference information, and at least one of the situation judgment, the decision-making process, or by using a language model that has been trained to explicitly change the output, or by changing the output parameters of the language model.

15. The surgical support device according to claim 14, wherein the thought estimation unit changes the style of natural language according to the state of the object or event, provided that the reference information relates to a specific object or event.

16. If the aforementioned reference information indicates the user's skill level, The surgical support device according to claim 15, wherein the output unit simplifies or elaborates the output text according to the user's skill level.

17. If the aforementioned reference information indicates whether the surgical support device is operating during surgery or outside of surgery, The surgical support device according to claim 15, wherein the output unit simplifies or elaborates the output text depending on the operating status.

18. The surgical support device according to claim 14, wherein the thought estimation unit changes the style of the text to provide at least one of a general explanation, a specialized explanation, and an emotional explanation.

19. The surgical support device according to any one of claims 1 to 3, further comprising a learning unit that applies the results of the image recognition to a template text, converts them into input for a language generation model, and learns the language generation model.

20. The system includes an auxiliary information addition determination unit that determines whether or not to add auxiliary information to the output in response to the user's query and / or the output of the thought estimation unit. The surgical support device according to any one of claims 1 to 3, wherein the output of the thought estimation unit is changed according to the determination result of the auxiliary information addition determination unit.

21. The surgical support device according to claim 20, wherein the auxiliary information addition determination unit determines to add auxiliary information to the answer if the user's query and / or the output of the thought estimation unit relate to an irreversible procedural operation.

22. The surgical support device according to claim 20, wherein when the thought estimation unit adds auxiliary information, it adds the likelihood of the image recognition result of the thought estimation unit and the likelihood of generating language information as auxiliary information.

23. The surgical support device according to claim 20, wherein the thought estimation unit adds auxiliary information that instructs the response to repeatedly ask the same question in different expressions when adding auxiliary information.

24. The system includes an emergency status determination unit that determines whether the surgical image used as input is in an emergency state, using the user's query and / or information obtained during the processing of the thought estimation unit. The surgical support device according to any one of claims 1 to 3, wherein the output of the thought estimation unit and the display format of the output unit are changed according to the result of determining an emergency state.

25. The surgical support device according to claim 24, wherein, when the emergency state determination unit determines that an emergency state exists, the thought estimation unit outputs a proper noun or keyword and outputs a short sentence that does not contain multiple instructions in a single sentence.

26. The surgical support device according to claim 24, wherein, if the emergency condition determination unit determines that an emergency condition exists, the thought estimation unit outputs the names of the surgical participants at the time of instruction.

27. The surgical support device according to claim 24, wherein, when the emergency condition determination unit determines that an emergency condition has occurred, the output unit outputs sound with varying intensity.

28. Surgical support device, Obtaining surgical images, To obtain queries and reference information from users, The process involves performing image recognition on the surgical image, and using the user's query and the reference information to estimate at least one of the situational judgment, decision-making, and the thought process leading to the situational judgment or decision-making regarding the surgical image. The output format is changed according to the results of the image recognition, the user's query and the reference information, and at least one of the situation judgment, the decision and the thought process, and at least one of the situation judgment, the decision and the thought process is output to the user. A program that executes the command.

29. A method performed by a surgical support device, Steps to obtain surgical images, Steps to obtain user queries and reference information, The steps include: performing image recognition on the surgical image, and using the user query and the reference information to estimate at least one of the situational judgment, decision-making, and the thought process leading to the situational judgment or decision-making regarding the surgical image; The steps include changing the output format according to the results of the image recognition, the user's query and reference information, and at least one of the situation judgment, the decision and the thought process, and outputting at least one of the situation judgment, the decision and the thought process to the user; A method that includes this.

Images