An open surgery scene graph automatic generation method, system, device and medium

Abstract

The present application relates to the technical field of artificial intelligence, in particular to an open operation scene graph automatic generation method, system, device and medium, the method comprises the following steps: adopting an automatic shot switching algorithm based on entity object feature recognition and object correlation detection design to extract key frames from video data; constructing a video object detection model and training the model, inputting the key frames into the video object detection model to detect the objects in the key frames; constructing a video object relationship detection model and training the model, inputting all the objects detected by the video object detection model into the trained video object relationship detection model to obtain the relationships between all the objects; constructing a directed graph structure network through the detected objects and the directed relationships between the objects, and generating scene graph content through the directed graph structure network; the present application realizes the acquisition of the field of view of the multi-angle open operation scene in real time through a multi-channel, multi-angle camera layout, and comprehensively perceives the operation scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, device and medium for automatically generating open surgical scene diagrams. Background Technology

[0002] Scene graphs are structured information about a specific visual environment, containing abstract representations of visual objects and their relationships. Directed relationships between objects are typically represented by triples <object1, predicate, object2>. Scene graphs provide a structured, abstract way to represent information in a visual environment, aiding in understanding the semantics of a scene. They can resolve an image into a simple yet meaningful structure, thus serving as a link between visual scenes and textual descriptions. Many tasks combining vision and language can be handled using scene graphs, including image description, visual question answering, content-based image retrieval, and image generation.

[0003] Visual scene graphs in open surgery serve as a crucial form of structured information, describing visual information during the surgical process, including various visual objects such as surgical instruments, medical equipment, and wounds, as well as the relationships between them. They provide a deeper semantic understanding of open surgical scene images. Scene graph generation has a very broad application prospect in this field. It not only assists medical professionals in better understanding and controlling the perioperative process but also has enormous potential in smart healthcare and automated medical care, contributing to improved surgical efficiency and safety.

[0004] Current research in the field of scene graphs is relatively limited. Scene graph generation involves the intersection of multiple fields, and the establishment of scene graphs for a specific domain requires a large amount of prior knowledge and involves an accurate understanding and abstraction of multiple objects and their relationships in an image. In addition, the establishment of large-scale, high-quality datasets for specific domains is a very complex and time-consuming fundamental task, and further research is needed on issues such as image occlusion and the recognition of complex actions in images. Summary of the Invention

[0005] To generate accurate and structured scene diagrams of important objects and their relationships in open surgical settings, this invention proposes an automatic scene diagram generation method for open surgical procedures. This method involves acquiring video data of the open surgical process using a camera mounted on a surgical light, and automatically generating the scene diagram based on the video data. The process includes the following steps:

[0006] An automatic shot switching algorithm based on entity object feature recognition and object correlation detection is used to extract keyframes from video data;

[0007] Construct a video object detection model and train the model. Input keyframes into the video object detection model to detect objects in the keyframes.

[0008] Construct a video object relationship detection model and train the model. Input all the objects detected by the video object detection model into the trained video object relationship detection model to obtain the relationships between all objects.

[0009] A directed graph network is constructed by detecting objects and the directed relationships between them, and scene graph content is generated through this directed graph network.

[0010] Furthermore, an automatic shot switching algorithm based on entity object feature recognition and object correlation detection is used to extract keyframes from the video data. This involves calculating the score of the video frame for each camera channel and selecting the highest-scoring keyframe. The calculation of the video frame score includes:

[0011]

[0012] Among them, w i x is the weight of object i appearing in a video frame. i x is a variable used to determine whether object i exists; if object i exists, then x... i =1, otherwise x i =0, where I represents the number of objects appearing in a video frame; λ n y represents the weight of event n occurring in a video frame. n y is a variable used to determine whether event n has occurred; if object n has occurred, then y... n =1, otherwise y n =0, where N represents the number of events that occur in a video frame.

[0013] Furthermore, the video object detection model is a YOLOv5 model, and the training of the video object detection model includes the following steps:

[0014] Video data was collected using a shadowless lamp, and frames were extracted from the collected open surgical scene video data.

[0015] The extracted scene image data is augmented and manually annotated using LabelImg, including the true location and category information of each object in the image.

[0016] When performing manual annotation, the annotators are divided into two groups, with each group annotating the data separately. After the annotation is completed, the two groups of annotators check and correct each other, and the annotation quality is checked by systematic sampling and double sampling. The data that passes the quality check is used for YOLOv5 model training.

[0017] When making predictions, the YOLOv5 model outputs the position vector of the detected object based on the input data, and generates the predicted bounding box position and the category based on the position vector.

[0018] Non-maximum suppression is applied to each object, and bounding boxes with IoU less than the threshold and confidence scores less than the preset value for the same predicted class are deleted.

[0019] Weighted nonmaximum suppression is used, which involves weighting the predictions based on their confidence levels to obtain the final prediction boxes.

[0020] The loss function is calculated based on the predicted bounding boxes, classification type, confidence score, and manual annotations output by the model. The YOLOv5 model parameters are then optimized based on the loss function.

[0021] Furthermore, the loss function used in the training process of the video object detection model includes:

[0022] L total =L obj +L location +L cls

[0023]

[0024] Among them, L total L represents the total loss of the video object detection model. obj L is the target confidence loss function for the video object detection model. location L is the localization loss function for the video object detection model. cls is the classification loss function for the video object detection model; IoU is the standard intersection-union ratio of the predicted bounding box and the ground truth bounding box; Dis_2 is the Euclidean distance between the two center points of the predicted bounding box and the ground truth bounding box; Dis_C is the distance between the diagonals of the minimum bounding rectangles of the predicted bounding box and the ground truth bounding box; v is a parameter that measures the consistency of aspect ratio.

[0025] Furthermore, video object relationship detection models include:

[0026] Semantic features of objects are generated by a semantic encoder based on category information obtained from a video object detection model;

[0027] Obtain surgical prior knowledge based on statistical methods and surgical prior knowledge based on language, and generate first prior semantic features and second prior semantic features respectively according to the semantic encoder;

[0028] The confidence information of the objects output by the video object detection model is converted into vector representations as object representation features, and the position information of the predicted bounding boxes is converted into vector representations as object spatial features.

[0029] Construct a reasoning model by concatenating object semantic features, object spatial features, object representation features, and first and second prior semantic features together as input to the reasoning model composed of fully connected layers;

[0030] The inference model pairs any two objects together and predicts the most likely event between the two objects.

[0031] Furthermore, the lamp holder, which is fixedly installed at the center of the shadowless lamp, is surrounded by a circular array of small lights to provide illumination for the surgical procedure. In the outermost layer of small lights, 4 to 6 cameras are evenly installed along the circular array. The image with the least obstruction is selected from the cameras as the video frame data to be collected at the current moment.

[0032] This invention also provides an open surgical scene diagram automatic generation system. This system implements an open surgical scene diagram automatic generation method. The system includes a data acquisition unit, a video object detection unit, a video object relationship detection unit, and a scene diagram output unit, wherein:

[0033] The data acquisition unit is used to acquire video information during open surgery and extract keyframe data from it, using the keyframes as input to the video object detection unit.

[0034] The video object detection unit is used to detect objects from keyframe data, including all types of surgical instruments, the head and hands of medical personnel, and the wounds and human tissues of surgical recipients.

[0035] The video object relationship detection unit is used to predict the relationships between objects output by the video object detection unit.

[0036] The scene graph output unit constructs directed relationships between objects from the objects predicted by the video object detection unit and the relationships between objects predicted by the video object relationship detection unit, and outputs these relationships.

[0037] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of an open surgical scene diagram automatic generation method.

[0038] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an open surgical scene diagram automatic generation method.

[0039] The present invention constructs a scene graph, extracts structured information from the open surgical visual environment, describes various visual objects during the surgical process, such as surgical instruments, medical equipment, and wounds, as well as the relationships between objects; collects and utilizes datasets through a specific device and trains object detection models and object relationship prediction models using deep learning methods; and achieves real-time synchronous acquisition of the field of view of the open surgical scene through a multi-channel, multi-angle camera layout, thereby realizing the construction of a scene graph for comprehensive perception of the surgical scene. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the shadowless lamp panel used in this invention;

[0041] Figure 2 This is a schematic diagram of the shadowless lamp used in this invention;

[0042] Figure 3 This is a flowchart of an open surgical scene diagram automatic generation method according to the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] This invention proposes a method for automatically generating open surgical scene diagrams. The method involves acquiring video data of an open surgical procedure using a camera mounted on a surgical light, and automatically generating a scene diagram based on the video data. The process includes the following steps:

[0045] An automatic shot switching algorithm based on entity object feature recognition and object correlation detection is used to extract keyframes from video data;

[0046] Construct a video object detection model and train the model. Input keyframes into the video object detection model to detect objects in the keyframes.

[0047] Construct a video object relationship detection model and train the model. Input all the objects detected by the video object detection model into the trained video object relationship detection model to obtain the relationships between all objects.

[0048] A directed graph network is constructed by detecting objects and the directed relationships between them, and scene graph content is generated through this directed graph network.

[0049] To address the issue of camera obstruction caused by doctors moving around while recording video, this invention involves mounting a camera on the operating light panel of a shadowless lamp. Figure 1 This is a schematic diagram of the operating light panel. A fixed lamp handle 103 is mounted in the center of the lamp panel, while a circular array of small lights 102 is arranged around the handle 103 to provide illumination. In the outermost layer of small lights, three cameras 101 are evenly distributed at equal angles along the circular array. Small lights 102 are also arranged at the other camera positions. This design allows the operating light to simultaneously perform illumination and imaging functions, ensuring that at least one camera is unobstructed at any time without affecting the normal operation of the operating room. The number of cameras can be flexibly set, ranging from four to six, all arranged at equal angles. The overall structure of the operating light is as follows. Figure 2 As shown.

[0050] In this embodiment, the open surgical scene graph automatic generation method of the present invention is described from the aspects of collecting video materials, training video object detection model, training video object relationship detection model, acquiring video content and keyframes, object recognition, relationship recognition, and scene graph generation.

[0051] (a) Collecting video materials

[0052] By modifying the operating room's shadowless lamp design, a data acquisition device was created by installing six cameras at varying angles on the lamp panel of different sizes. This ensured comprehensive coverage of the field of vision, preventing obstruction of important objects and relationships within the surgical field. The device was used to capture information from open surgical scenarios performed by senior surgeons, acquiring image data at a sampling frequency of three seconds per frame to cover various interactive operations during the procedure. The collected videos underwent anonymization and basic data processing for further analysis. Post-processing techniques were employed to adjust parameters such as exposure, contrast, and brightness to minimize their impact on subsequent experiments, addressing potential overexposure. Following these steps, the LabelImg image annotation tool was used to label key objects appearing in the video frames, including medical personnel's heads and hands, wounds, human tissue, needle clips, surgical sutures, hemostats, forceps, scalpels, suction devices, anterior and posterior traction devices, tissue scissors, electrocautery, gauze, detectors, and syringes. A quality control mechanism ensured the quality of the dataset.

[0053] In this embodiment, an automatic camera switching algorithm based on entity object feature recognition and object correlation detection is used to extract keyframes from video data. This involves calculating the score of each camera channel's video frame and selecting the highest-scoring keyframe. The calculation of the video frame score includes:

[0054]

[0055] Among them, w i x is the weight of object i appearing in a video frame. i x is a variable used to determine whether object i exists; if object i exists, then x... i =1, otherwise x i =0, where I represents the number of objects appearing in a video frame; λ n y represents the weight of event n occurring in a video frame. n y is a variable used to determine whether event n has occurred; if object n has occurred, then y... n =1, otherwise y n =0, N represents the number of events that occur in the video frame. In this embodiment, the event refers to the connection relationship between connected objects, that is, the directed relationship between objects, and the predicate in the triple.

[0056] In this embodiment, in addition to manually annotating the training data, the standards also need to be quality checked. That is, when manually annotating, the annotators are divided into two groups, each group annotating the data separately. After the annotation is completed, the two groups of annotators check and correct each other, and the annotation quality is checked by systematic sampling and double sampling.

[0057] (II) Training the Video Object Detection Model

[0058] In the stage of training the target detection model, this embodiment chooses to divide the collected video dataset into a training set and a validation set for model training to ensure the generalization performance of the model.

[0059] This embodiment requires data annotation processing before model training to obtain the true location and category information of each object in the operation, which serves as the label for model training. In each iteration, the predicted bounding box generated by the model is associated with the labeled true bounding box, and the loss function is calculated by calculating the location information, target confidence, and category information.

[0060] The model is evaluated and optimized using a validation set to improve its generalization ability and ensure good recognition capabilities on untrained surgical scene images. When optimizing model parameters, a deep learning architecture is used, continuously updating the model through a backpropagation function and optimizer to reduce the false detection rate and improve its performance in surgical scene image recognition. This embodiment fully utilizes a rich surgical scene dataset to ensure that the trained object detection model possesses efficient and accurate object recognition capabilities in practical applications.

[0061] In this embodiment, the video object detection model is the YOLOv5 model. The model prediction results are correlated with the real information, and the deviation between the location information, target confidence and category information is calculated to form a loss function. Through backpropagation and gradient update, the network parameters are continuously updated to gradually reduce the loss function and improve the accuracy of the model.

[0062] In testing or practical use, the model's processing of input data mainly involves two steps: First, using the location vector output by the network, the location and category of the predicted bounding box are calculated. Then, non-maximum suppression (NMS) is performed on each category, deleting boxes with low confidence scores that have an IoU less than a threshold and belong to the same predicted category. In this embodiment, weighted NMS is used, weighting the predicted bounding boxes based on their confidence scores to obtain the final bounding boxes. Boxes with scores below a threshold are filtered out. This process yields the final bounding box information and score, completing the target classification and localization for the surgical scene.

[0063] In this embodiment, a constructed real surgical scene dataset is used to optimize the parameters of the YOLO v5 model during training. The training process iterates 300 times, calculating the loss between the predicted bounding boxes output by the video object detection model and the manually annotated ground truth bounding boxes, considering location information, object confidence, and category information. During training, the loss function is derived from the localization loss function L. location Target confidence loss function L obj and classification loss function L cls The video object detection model consists of three parts, and the total loss is expressed as follows:

[0064] L total =L obj +L location +L cls

[0065]

[0066] Among them, L obj L is the target confidence loss function for the video object detection model. location L is the localization loss function for the video object detection model. cls is the classification loss function for the video object detection model; IoU is the standard intersection-union ratio of the predicted bounding box and the ground truth bounding box; Dis_2 is the Euclidean distance between the two center points of the predicted bounding box and the ground truth bounding box; Dis_C is the distance between the diagonals of the minimum bounding rectangles of the predicted bounding box and the ground truth bounding box; v is a parameter that measures the consistency of aspect ratio.

[0067] (III) Training the Video Object Relationship Detection Model

[0068] This embodiment integrates multimodal features, statistical prior information, linguistic prior information, and common sense information, and improves the performance of the object relation generation model in scene graph generation tasks through global message passing.

[0069] In the process of training the object relationship prediction model, this embodiment is based on the multimodal features of surgical scene images, such as spatial features, object representation features and semantic features in the surgical scene. It introduces prior information such as relevant statistical priors and linguistic priors in specific surgical scenes and considers common sense knowledge information based on medical surgical scenes to construct image features. It uses directed graphs to represent object relationships and adopts a global message passing strategy to consider the influence of semantic relevance in visual relationships, solves the problem of isolated generated triples, and improves the model training performance.

[0070] When training the model, a supervised learning algorithm can be used to train the object relationship prediction model. This can be chosen to use... The space (where n is the number of object categories and m is the maximum number of relationship categories between objects) describes all possible relationship scenarios. The prediction model will then use the scene object prediction results and prior information to... The space is searched to find the most suitable relationships between objects.

[0071] The object relationship detection model obtains the representational features, spatial features, and semantic features of each object from pre-labeled image information; it also obtains surgical prior knowledge based on statistical methods and surgical prior knowledge based on language, that is, it uses prior knowledge to construct a prior model to describe the probability of the object's occurrence and the frequency of the event;

[0072] Based on the extracted surgical text information, a knowledge graph of the surgical scene is constructed. The graph can include surgical objects, instruments, procedures, and the relationships between them, forming structured linguistic prior knowledge. The embedding vectors of the surgical knowledge graph are introduced into the relation mapping network, taking the features of the objects and the prior knowledge as input.

[0073] The object relationship prediction model is trained using a supervised learning algorithm. The data relationship is represented as a directed graph structure, and the loss function is defined as the cross-entropy function to measure the difference between the model output and the true label. The model parameters are updated using backpropagation and optimized by gradient descent. To ensure that triples are not isolated and to introduce contextual information of related objects, this embodiment adopts a fully connected graph result and performs global message passing on it.

[0074] This embodiment presents an implementation method for a video object relationship detection model, which specifically includes the following steps:

[0075] Semantic features of objects are generated by a semantic encoder based on category information obtained from a video object detection model;

[0076] Obtain surgical prior knowledge based on statistical methods and surgical prior knowledge based on language, and generate first prior semantic features and second prior semantic features respectively according to the semantic encoder;

[0077] The confidence information of the objects output by the video object detection model is converted into vector representations as object representation features, and the position information of the predicted bounding boxes is converted into vector representations as object spatial features.

[0078] Construct a reasoning model by concatenating object semantic features, object spatial features, object representation features, and first and second prior semantic features together as input to the reasoning model composed of fully connected layers;

[0079] The inference model pairs any two objects together and predicts the most likely event between the two objects.

[0080] (iv) Acquiring video content and keyframes

[0081] When surgical instruments, incisions, or a surgeon's hand appear in an image, they are marked as keyframes. This embodiment simultaneously acquires image data through six camera channels, solving the problem of target occlusion in complex surgical scenes. By employing an automatic lens switching algorithm based on entity object feature recognition and object correlation detection, an optimal image selection strategy is implemented. Among the six surgical video channels, the channel with the least occlusion and the best viewing angle is selected to help acquire more visual information. Through this strategy, real-time surgical images are selectively acquired during the prediction stage, thereby improving the system's ability to perceive the surgical scene.

[0082] (V) Object Recognition

[0083] In the prediction phase, this embodiment acquires real-time surgical images from multi-channel, multi-angle surgical lights and transmits these images to the server. The server uses a target detection model to predict various objects appearing in the real-time scene and outputs the results as real-time information. Finally, the corresponding objects, such as hands, heads, wounds, and medical instruments, can be displayed and selected on the image, showing their positions in the picture.

[0084] (vi) Relationship Identification

[0085] This embodiment obtains the position and category information of objects appearing in each real-time scene through the object detection model. That is, it uses the established object relationship prediction model to combine the real-time collected object information to predict the relationship between objects in the scene. For example, the relationship between the hand and the tissue scissors is generated as: hand -> holding -> tissue scissors.

[0086] (vii) Scene graph generation

[0087] By combining the two steps of automatic object identification and automatic acquisition of relationships between objects, a directed graph structure network containing objects and their relationships is established to represent the scene atlas in the corresponding open surgical scene image. The results can be used to assist medical teams and further achieve a comprehensive perception and understanding of the surgical scene.

[0088] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatically generating open surgical scene diagrams, characterized in that, The process of automatically generating scene diagrams based on video data from open surgical procedures by installing multi-angle cameras on a surgical lamp includes the following steps: An automatic shot switching algorithm based on entity object feature recognition and object relationship detection is used to extract keyframes from video data; Construct a video object detection model and train the model. Input the collected keyframes into the video object detection model to detect objects in the keyframes. A video object relationship detection model is constructed and trained. The video object detection model is a YOLOv5 model. The training of the video object detection model includes the following steps: Video data was collected using a shadowless lamp, and frames were extracted from the collected open surgical scene video data. The extracted scene image data is augmented and manually annotated using LabelImg, including the true location and category information of each object in the image. When performing manual annotation, the annotators are divided into two groups, with each group annotating the data separately. After the annotation is completed, the two groups of annotators check and correct each other, and the annotation quality is checked by systematic sampling and double sampling. The data that passes the quality check is used for YOLOv5 model training. When making predictions, the YOLOv5 model outputs the position vector of the detected object based on the input data, and generates the predicted bounding box position and the category based on the position vector. Non-maximum suppression is applied to each object, and bounding boxes with IoU less than the threshold and confidence scores less than the preset value for the same predicted class are deleted. Weighted nonmaximum suppression is used, which involves weighting the predictions based on their confidence levels to obtain the final prediction boxes. The loss function is calculated based on the predicted bounding boxes, classification type, confidence score, and manual annotations output by the model. This loss function is then used to optimize the YOLOv5 model parameters. The loss functions used in the training process of the video object detection model include: in, This represents the total loss of the video object detection model. Let be the target confidence loss function for the video object detection model. This is the localization loss function for the video object detection model. The classification loss function for the video object detection model; The standard intersection-union ratio (CUI) of the predicted bounding box and the ground truth bounding box. The Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box is... The distance between the diagonals of the minimum bounding rectangles of the predicted bounding box and the ground truth bounding box. It is a parameter that measures the consistency of aspect ratio; All objects detected by the video object detection model are input into a pre-trained video object relationship detection model to obtain the relationships between entity objects; the video object relationship detection model includes: Semantic features of objects are generated by a semantic encoder based on category information obtained from a video object detection model; Obtain surgical prior knowledge based on statistical methods and surgical prior knowledge based on language, and generate first prior semantic features and second prior semantic features respectively according to the semantic encoder; The confidence information of the objects output by the video object detection model is converted into vector representations as object representation features, and the position information of the predicted bounding boxes is converted into vector representations as object spatial features. Construct a reasoning model by concatenating object semantic features, object spatial features, object representation features, and first and second prior semantic features together as input to the reasoning model composed of fully connected layers; The inference model pairs any two objects together and predicts the most likely event between the two objects. A directed graph network is constructed by detecting objects and the directed relationships between them, and scene graph content is generated through this directed graph network.

2. The method for automatically generating open surgical scene diagrams according to claim 1, characterized in that, An automatic shot switching algorithm based on entity object feature recognition and object correlation detection is used to extract keyframes from video data. This involves calculating the score of video frames for each camera channel and selecting the frame with the highest score as the keyframe. The calculation of the video frame score includes: in, The weight of object i appearing in the video frame. This is a variable used to determine whether object i exists; if object i exists, then... ,otherwise I represents the number of objects appearing in a video frame; The weights of events n appearing in a video frame. This is a variable used to determine whether event n has occurred; if object n has occurred, then... ,otherwise N represents the number of events that occur in a video frame.

3. The method for automatically generating open surgical scene diagrams according to claim 1, characterized in that, The center of the shadowless lamp is fixedly installed with a lamp handle surrounded by a circular array of small lights to provide illumination for the surgical procedure. In the outermost layer of small lights, 4 to 6 cameras are evenly installed along the circular array. The image with the least obstruction is selected from the cameras as the video frame data to be collected at the current moment.

4. An automatic generation system for open surgical scene diagrams, characterized in that, This system is used to implement the open surgical scene diagram automatic generation method as described in claim 1. The system includes a data acquisition unit, a video object detection unit, a video object relationship detection unit, and a scene diagram output unit, wherein: The data acquisition unit is used to acquire video information during open surgery and extract keyframe data from it, using the keyframes as input to the video object detection unit. The video object detection unit is used to detect objects from keyframe data, including all types of surgical instruments, the head and hands of medical personnel, and the wounds and human tissues of surgical recipients. The video object relationship detection unit is used to predict the relationships between objects output by the video object detection unit. The scene graph output unit constructs directed relationships between objects from the objects predicted by the video object detection unit and the relationships between objects predicted by the video object relationship detection unit, and outputs these relationships.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the open surgical scene diagram automatic generation method as described in claim 1.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the open surgical scene diagram automatic generation method as described in claim 1.

Images