An automatic labeling method, system, terminal device, and storage medium for cystic tubes.
By constructing a spatiotemporal heterogeneous graph and an HTGNN model, and mimicking the manual labeling process, the problem of automatic labeling of cystic ducts in minimally invasive surgery was solved, achieving high accuracy and rapid cystic duct labeling and reducing the surgical error rate.
Patent Information
- Application Number
- CN202311783076.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-22
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-12-22
AI Technical Summary
Existing technologies struggle to effectively and automatically label cysts in minimally invasive surgery, especially in the absence of supervised samples and effective feature extractors, leading to increased surgical error rates and a higher risk of postoperative complications.
By mimicking the manual labeling process, a spatiotemporal heterogeneous graph is constructed. The HTGNN model is used for information aggregation, interpretable features are extracted, and the location, orientation, and size information of the cystic tubes are obtained, thus eliminating the dependence on large labeled datasets and deep networks.
It achieves highly accurate, robust, and rapid automatic labeling of sac tubes in complex surgical scenarios, with strong scene transfer capabilities and autonomous controllability, reducing the impact of noise interference and background information.
Smart Images

Figure CN117912014B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence-assisted surgery, and more specifically, relates to a method, system, terminal device and storage medium for automatic marking of sac ducts. Background Technology
[0002] Minimally invasive surgery has gradually replaced open surgery and become the mainstream for many important surgical procedures due to its advantages such as lower morbidity and mortality, less pain, less wound infection, and shorter hospital stays. Over the past decade, robot-assisted minimally invasive surgery (RAMIS) has gained popularity for its greater dexterity and efficiency compared to conventional minimally invasive surgery. Most RAMIS systems use robot-driven surgical instruments, which are still remotely operated by surgeons. The exploration of how to achieve autonomous RAMIS has attracted widespread attention, requiring a deeper understanding of the patient's internal condition during the procedure—that is, surgical scene awareness.
[0003] Because laparoscopy inherently provides limited field of view, variable angles of view, and blurred vision, coupled with the highly complex internal environment of the patient, making it difficult to distinguish between different tissues, the likelihood of surgical errors increases, leading to postoperative complications. To address this issue, intelligent surgical assistance methods are needed to mark key tissues within the surgical field, thereby providing operational warnings.
[0004] The methods for labeling landmark tissues include manual labeling and automatic label generation. Taking the key landmark tissue—cystic ducts—as an example, identification is very difficult. Manual labeling methods involve using fluorescent staining in clinical practice to assist doctors in identifying cystic ducts. Automatic label generation methods are mainly divided into three categories: label transfer, active learning, and mimicry labeling. Label transfer is primarily used for simple visual tasks, where the model already performs well on these tasks. After fine-tuning based on supervised samples, the detection results can be used as labels. Above this, there is the active learning method, which involves filtering generated labels and manually labeling those with high uncertainty. Besides these two mainstream methods, there is also a method that mimics the label-making process. For example, knowing that a doctor reads keywords from a patient's medical record, an image classification of the patient can be generated. Some studies have mimicked this process, designing algorithms to generate image labels.
[0005] For challenging visual tasks like manual labeling, a large number of samples are needed to train the model, which is extremely difficult to obtain. This is due to two main reasons: firstly, manual labeling is difficult, tissue identification is challenging, and labeling is uncertain; secondly, the number of effective samples throughout the entire surgical procedure is limited. Automatic labeling methods are further hampered by a lack of supervised samples and effective feature extractors. Moreover, the highly complex scenes within a patient's body make it difficult to distinguish different tissues, leading to difficulties in target description. Therefore, the application of automatic labeling methods requires overcoming numerous challenges. Summary of the Invention
[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides an automatic labeling method for cystic ducts. By mimicking the process of manually labeling tissues, and based on cystic ducts in a fully exposed state, it obtains cystic duct information from other scenarios through frame-by-frame tracking and reasoning. This method eliminates the reliance on large labeled datasets and feature extractors based on deep networks, and solves the limitations of lacking supervised samples and effective feature extractors in the automatic label generation process.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] In a first aspect, the present invention provides an automatic marking method for cystic ducts, comprising the following steps:
[0009] S100. Surgical video analysis: The process of manually marking the cystic tube is analyzed through surgical video analysis. By reversing the video, the cystic tube is marked back from its fully exposed state to its initial shape, and the relationship between various factors in the video is analyzed in this process.
[0010] S200. Constructing a spatiotemporal heterogeneous graph: Based on the relationship between different factors in the reverse dissection process, the surgical scene is abstracted into a heterogeneous graph with multiple nodes, and the dissection process is modeled as a temporal heterogeneous graph by connecting the heterogeneous graph through the same nodes at each time step.
[0011] S300. Extract interpretable features: In complex surgical scenarios, noise interference is eliminated and interpretable features of each node are extracted as its attributes.
[0012] S400, Dynamic Information Transmission: The spatiotemporal heterogeneous graph is used as input, and information aggregation is performed using the HTGNN model to obtain the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph.
[0013] S500, Target Tissue Parameter Regression: Based on the spatiotemporal embedding of the duct attributes obtained in the S400 dynamic information transmission step, the characterization attributes of the ducts are obtained for labeling the ducts.
[0014] Furthermore, in step S100, the relationship between the various factors includes the dynamic interaction between the surgical instruments and the exposed sac tissue;
[0015] When surgical instruments interact with the adhesive tissue covering the balloon tube, the spatial topology and temporal dynamics of the balloon tube are often obscured by the instruments.
[0016] Furthermore, in step S200, the time heterogeneity diagram has nodes including Tool, dissection operation Int, exposed cystic duct tissue OA, and overall cystic duct structure GA.
[0017] Furthermore, in step S200, the temporal heterogeneous graph and nodes are mapped to a target node set {v}. ga ,v ins ,v ia ,v oa}, the edge type is mapped to the edge type set {e ia→ins ,e ia→oa ,e oa→ga ,e ins→ga} is used to convey spatial topological relationships;
[0018] Furthermore, in step S300, extracting interpretable features includes:
[0019] S301. Extract frames from video slices and convert them into image datasets;
[0020] S302. The semantic segmentation model Mask R-CNN processes video frames to segment objects;
[0021] S303. Post-process the segmented objects, including manually marking and initializing the cysts;
[0022] S304. Extract interpretable features from post-processed objects;
[0023] Furthermore, in step S304, the interpretable features include the position, orientation, and size of the GA; the position, orientation, and depth of the Tool; the position and size of the OA; and the contact state and contact points of the instrument's interaction with the anatomical structure.
[0024] Furthermore, in step S304, the method for obtaining the interpretable features includes:
[0025] The origin, eigenvalues, and eigenvectors of the GA abstract ellipse equivalent two-dimensional Gaussian distribution; the region of interest center and coverage area of OA; the region of interest center of Tool and its contour changes and contour angles after Hough Transform; the positions of Tool and OA and their region of interest centers;
[0026] Furthermore, in step S400, the information aggregation includes the following steps:
[0027] S403. Project the feature spaces of different node types into the same space;
[0028] S404, according to v ia Information update v oa and v ins Information;
[0029] S405, according to the updated v oa and v ins Information reasoning v ga information;
[0030] Furthermore, in step S404, according to v ia Information update v oa and v ins The information, namely:
[0031]
[0032] in, The set of node types at time step t This represents the nodes of the node type set at time step t; For node u at time step t, embed it with the previous layer; For the updated node v, the embedding of the last layer at time step t, and Θ IR These are trainable parameters; AGG IR For information aggregation functions, when it is the first level (l=1), AGG IR Take the output of the heterogeneous information processing step as the input;
[0033] Furthermore, in step S404, a self-attention mechanism is used to assign a weight coefficient reflecting the importance of each node. The method for obtaining the weight coefficient is as follows:
[0034]
[0035] in, σ is the weight coefficient of the node in layer l at time step t; σ(·) is the Leaky ReLU activation function; W t,l Let a be a trainable transformation matrix, which is a real matrix with d rows and d columns, where d is a hyperparameter; t,l Let be the attention vector, which is a d-dimensional real vector, where d is a hyperparameter; ‖· is the connection function and its connection vector;
[0036] Furthermore, in step S405, based on the updated v oa and v ins Information reasoning v ga Information, namely:
[0037]
[0038] in, The embedding of the obtained u-nodes is aggregated through intra-frame information; The heterogeneous graph Node set on<Tool,GA,OA> u is The node inside; v is a set Nodes within; For the updated node v at time step t, the embedding with the last layer; Θ II These are trainable parameters, which are not shareable across node types, time steps, and aggregation layers. II With Θ IR Not shareable; AGG II It is an information aggregation function that works through an attention mechanism;
[0039] Furthermore, in step S500, the characterization attribute of the duct, i.e., the target tissue regression parameter δ{δ x δ y δ a δ b δ θ}for:
[0040] δ a =(a t / a0),δ b =(b t / b0),δ θ =θ t / π;
[0041] Where, δ x Let δ be the x-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. y Let δ be the y-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. a δ is the scaling factor of the major axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values.b δ is the scaling factor of the minor axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values. θ The ellipse rotation angle is the regression parameter of the cystic duct node. Here are the x-coordinates of the center point of the ellipse at time step t. Let a be the y-coordinate of the center point of the ellipse at time step t. t Let a0 be the major axis of the ellipse at time step t, and b be the major axis of the ellipse at the initial stage. t Let b0 be the minor axis of the ellipse at time step t, and b0 be the minor axis of the ellipse in the initial stage. θ t Let be the rotation angle of the ellipse at time step t;
[0042] In a second aspect, the present invention provides an automatic cyst duct marking device, comprising:
[0043] The video analysis module (11) is used to analyze the relationships between various factors in the processed video;
[0044] A spatiotemporal heterogeneous graph module (12) is constructed to abstract the surgical scene into a heterogeneous graph with multiple nodes based on the relationship between different factors in the reverse dissection process obtained from image and video analysis, and to model the dissection process as a temporal heterogeneous graph by connecting the heterogeneous graphs through the same nodes at each time step.
[0045] The node attribute extraction module (13) is used to eliminate noise interference in complex surgical scenarios and extract the interpretable features of each node as its attributes.
[0046] The dynamic information transmission module (14) is used to perform information aggregation of the HTGNN model, thereby obtaining the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph.
[0047] The target tissue parameter regression module (15) is used to obtain the characterization attributes of the cysts based on the spatiotemporal embedding of the obtained cyst attributes for labeling the cysts.
[0048] Thirdly, the present invention provides a terminal device, comprising:
[0049] Memory, used to store computer programs;
[0050] A processor, configured to implement the steps of the above-described automatic cystic duct marking method when executing the computer program;
[0051] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described automatic cystic duct marking method.
[0052] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:
[0053] 1. The method of the present invention includes extracting interpretable features and constructing a spatiotemporal heterogeneous graph by analyzing the relationships between various elements in the surgical video. Therefore, the method of the present invention gets rid of the reliance on a large number of labeled datasets and the reliance on feature extractors based on deep networks, and solves the problem of the lack of supervised samples and effective feature extractors in the automatic label generation process.
[0054] 2. The method of the present invention introduces the interaction process between instruments and tissues to assist reasoning in the process of constructing the spatiotemporal heterogeneous graph. It filters out the background information with large differences and invalidities in the cystic duct tissue information under various complex scenarios, and extracts interpretable features to filter out noise interference in complex surgical scenarios. This makes the method of the present invention have the advantages of strong robustness, high average coverage and strong scene transferability.
[0055] 3. The method of this invention analyzes the process of manually identifying the complete cystic duct anatomical structure and manually marking the cystic duct in the Calot triangulation video and obtains the relationship between various elements in the video. Furthermore, it obtains cystic duct information by imitating the manual marking process. Therefore, the automatic marking of cystic ducts by the method of this invention has the advantage of high accuracy.
[0056] 4. The method of the present invention constructs a spatiotemporal heterogeneous graph by analyzing the relationships between various elements in the surgical video, and the spatiotemporal heterogeneous graph has the characteristic of small network parameters. Therefore, the method of the present invention has the advantage of fast frame processing speed.
[0057] 5. The method of the present invention can read the weight coefficients of each node during the intra-frame information aggregation process in the dynamic information transmission step, providing a reference for the control algorithm and realizing the independent control of the core algorithm. Attached Figure Description
[0058] Figure 1 This is a flowchart of a method for automatic marking of cystic ducts according to an embodiment of the present invention;
[0059] Figure 2 A diagram illustrating the process of manually identifying complete cystic duct anatomy in a Calot triangulation video.
[0060] Figure 3 This is a schematic diagram of the spatiotemporal heterogeneity of a modeling method for an automatic labeling method for cystic ducts according to an embodiment of the present invention;
[0061] Figure 4 This is a flowchart illustrating a method for extracting interpretable features as graph node attributes according to an embodiment of the present invention;
[0062] Figure 5 This invention provides a method for extracting interpretable features as graph node attributes, which is used to obtain laparoscopic images during surgical procedures by frame extraction.
[0063] Figure 6This is a flowchart illustrating information aggregation according to an embodiment of the present invention;
[0064] Figure 7 This is a flowchart of an intra-frame information aggregation method according to an embodiment of the present invention;
[0065] Figure 8 This is a schematic diagram of the structure of an automatic cyst tube marking device according to the present invention;
[0066] In all the accompanying figures, the same reference numerals denote the same technical features, specifically: 11-Image and video analysis module, 12-Constructing spatiotemporal heterogeneous graph module, 13-Extracting node attributes module, 14-Dynamic information transmission module, 15-Target organization parameter regression module. Detailed Implementation
[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0068] See also Figure 1 As shown in the figure, an embodiment of the present invention provides a flowchart of a method for automatic marking of cystic ducts, the specific steps of which are as follows:
[0069] S100. Surgical video analysis: First, the process of manually marking the cystic duct was analyzed using surgical video. It was found that the cystic duct was initially covered by adhesive tissue during the marking process, and was gradually exposed through dissection. Therefore, by reversing the video, the cystic duct could be marked step-by-step from its fully exposed state back to its initial shape. The relationships between various elements in the video were analyzed during this process.
[0070] S200. Constructing a spatiotemporal heterogeneous graph: Based on the relationship between different elements in the reverse dissection process obtained in the S100 surgical image video analysis step, the surgical scene is abstracted into a heterogeneous graph with multiple nodes. By connecting the heterogeneous graph through the same node at each time step, the dissection process is modeled as a temporal heterogeneous graph.
[0071] S300. Extract interpretable features as graph node attributes: In complex surgical scenarios, noise interference is eliminated and interpretable features of each node are extracted as its attributes.
[0072] S400, Dynamic Information Transmission: The spatiotemporal heterogeneous graph obtained in the S200 step of constructing the spatiotemporal heterogeneous graph is used as input, and information aggregation is performed using the HTGNN model to obtain the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph.
[0073] S500, Target Tissue Parameter Regression: Based on the spatiotemporal embedding of the duct attributes obtained in the S400 dynamic information transmission step, the position, orientation, and size of the ducts are obtained for labeling the ducts.
[0074] Specifically, in the S100 surgical image video analysis steps, the process of manually identifying the complete cystic duct anatomy and manually marking the cystic duct in the Calot triangulation video is analyzed. For example... Figure 2 As shown, before dissection, the cystic duct is covered by adhering tissue and is not visible in the view, serving as an implicit label; after dissection, the cystic duct is separated from the adhering tissue by surgical instruments and exposed in the view, serving as an explicit label.
[0075] The reverse dissection process, obtained by reversing the procedure, involves using instruments to allow the adhering tissue to cover the cystic duct, thus gradually reducing the observation area of the cystic duct. This process involves two key factors. Firstly, there is a fixed spatial relationship between the exposed cystic duct structure and the overall cystic duct structure. Therefore, the size of the exposed cystic duct structure in the anatomical surgical scene and its change over time are taken as one factor. To utilize this factor, the anatomical area needs to be connected across time to establish its temporal dynamic model. Secondly, the establishment of the temporal dynamic model during the reverse dissection process also includes the factor that the exposed anatomical area gradually disappears as the reverse dissection process proceeds.
[0076] However, the aforementioned time dynamic factors are insufficient to infer the complete global anatomical structure. In reverse dissection, there is also a dynamic interaction between the surgical instruments and the exposed cystic duct tissue. That is, when there is an interaction between the surgical instruments and the adhesive tissue covering the cystic duct, the cystic duct is often obscured by the instruments. The dynamic interaction and time dynamic factors can help infer the location of the cystic duct.
[0077] Furthermore, in the S200 step of constructing the spatiotemporal heterogeneous graph, considering that the inference of the cystic duct is mainly based on the fixed spatial relationship between the exposed cystic duct structure and the overall cystic duct structure obtained from the reverse dissection process in the S100 surgical image video analysis step, as well as the interaction relationship between the instruments and the cystic duct structure, the surgical scene can be abstracted into a heterogeneous graph HG with multiple nodes including: instruments, dissecting operations, exposed cystic duct tissue, and the overall cystic duct structure. Here, instruments are represented as Tool, dissecting operations as Int, exposed cystic duct tissue as OA, and the overall cystic duct structure as GA. The nodes have interpretable attribute characteristics.
[0078] Furthermore, based on the different relationships between the nodes, the direction of information transmission between the nodes is obtained, that is, the Int node modulates the weight of the OA node, and the instrument node performs inference on the occluded GA node under different interaction conditions.
[0079] Specifically, the OA node, i.e., the exposed cystic duct tissue, as part of the GA (overall cystic duct structure), can be used to locate the GA node; and since surgical instruments often occlude the cystic duct tissue when interacting with it, the Tool node, i.e., the position and orientation of the surgical instrument, can be used to locate the GA node when the Int node occurs, i.e., during the dissection operation. Therefore, the surgical scene is abstracted as a heterogeneous graph HG, defined as G = (V, E), where the node type mapping of the heterogeneous graph HG is V → A, and the edge type mapping is E → R, where V represents the target node set {v ga ,v ins ,v ia ,v oa}, where A represents the set of node types {OA, Tool, Int, GA}, and the target node set V is assigned to the target nodes respectively; E represents the set of edge types {e ia→ins ,e ia→oa ,e oa→ga ,e ins→ga Let R be a set of relation types, where e ia→ins and e ia→oa The aim is to convey the dynamic interaction information between surgical instruments and exposed cystic tissue obtained from the S100 surgical image video analysis step. oa→ga and e ins→ga The aim is to capture the scene topological correlation between the exposed cystic duct structures and the overall cystic duct structure obtained from the video analysis step of the S100 surgical images.
[0080] Furthermore, by concatenating heterogeneous graphs (HGs) of the same type through nodes at each time step, the inverse dissection process is modeled as a temporal heterogeneous graph (THG), such as... Figure 3 The image shows the spatiotemporal heterogeneity graph obtained from modeling. In the spatiotemporal heterogeneity graph THG, each heterogeneity graph slice G t The arrangement remains static, while the properties of its nodes change over time.
[0081] Furthermore, since the surgical scene in the process of abstracting the surgical scene into a heterogeneous graph HG during the S200 step of constructing the spatiotemporal heterogeneous graph has various noise effects, in order to overcome the noise effects, the S300 step of extracting interpretable features as graph node attributes is performed.
[0082] See also Figure 4 One embodiment of the present invention provides a method for extracting interpretable features as graph node attributes, the specific steps of which are as follows:
[0083] S301. Extract frames from video slices and convert them into image datasets;
[0084] S302. The semantic segmentation model Mask R-CNN processes video frames to segment objects;
[0085] S303. Post-process the segmented objects, including manually marking and initializing the cysts;
[0086] S304. Extract interpretable features from the post-processed object.
[0087] Specifically, the video slices were first extracted from the reversed Calot triangulation video of the S100 surgical image video analysis step at a frame extraction frequency of 0.5s / time, thus obtaining an image dataset. Then, the semantic segmentation model Mask R-CNN was used to process the video frames to segment objects in the video frames and perform post-processing, including manually labeling and initializing the cystic duct, thereby extracting interpretable features.
[0088] like Figure 5 The image shown is a laparoscopic image of the surgical procedure obtained by frame extraction. The interpretable features of the observed objects are used as attributes hv of the target node set V of the heterogeneous graph HG. The selection of attribute hv is based on anatomical knowledge. The GA is abstracted as an ellipse, which is equivalent to a two-dimensional Gaussian distribution. The ellipse is represented by the origin, eigenvalues, and eigenvectors of the two-dimensional Gaussian distribution, as well as the pixels of the GA in the image. Specifically, the ellipse is represented by the origin (c... x ,c y The position of GA is represented by the center point of the ellipse, the orientation of GA is represented by the eigenvector θ of the two-dimensional Gaussian distribution (i.e., the rotation angle of the ellipse), and the size of GA is represented by the eigenvalues (a,b) of the two-dimensional Gaussian distribution (i.e., the major axis a and the minor axis b of the ellipse).
[0089] To deduce the location information of the GA, i.e., the parameters of the aforementioned ellipse, the following remaining node attributes hv are selected:
[0090] The location and size of the OA can provide location information for the GA, unaffected by changes in the field of view, through the center (x) of the OA's ROI (region of interest). oa ,y oa The tool (OA) is represented by its position, and its size is represented by the area m it covers. During the interaction between the instrument and the vascular bundle tissue, the tool position is related to the GA position. The tool orientation is usually orthogonal to the GA direction, and is represented by the Hough change (α) of the instrument contour. Considering the principle of image projection, the tool depth can reflect the scale of the GA, and is represented by the angle β of the instrument contour. The tool position is located at the center (x) of the instrument's ROI (region of interest). ins ,y insThe values of α and β are determined by estimating the instrument profile after Hough Transform. Since erroneous detection or missing instruments or anatomical structures can complicate the structure of the temporal heterogeneity graph (THG), to accurately detect instruments and anatomical structures and capture scene differences without altering the graph structure, the interaction between the instrument and exposed cystic duct tissue structures is treated as a node. The IoU (Intersection over Union) ζ between Tool and OA represents the contact position of Tool and OA, and the positional relationship s between Tool and OA represents the contact point. The attributes of each node and their acquisition methods are shown in Table 1.
[0091]
[0092] Table 1: Attributes of each node and methods for obtaining them
[0093] The attribute hv characterization includes the type of surgical instrument, the scaling of the surgical instrument in the reverse anatomical video, the tip of the surgical instrument, whether the instrument and the covering sac come into contact with each other during the anatomical operation, the interaction position of the instrument and the covering sac during the anatomical operation, the interaction trend of the instrument and the covering sac during the anatomical operation, the initial morphology of the sac, and the position of the sac.
[0094] Furthermore, in the S400 dynamic information transmission step, the attributes of each node obtained in the S300 step of extracting interpretable features as graph node attributes are aggregated using the HTGNN (heterogeneous temporal graph neural network) model to obtain the spatiotemporal embedding of the cystic duct node attributes at each time step. During the training of this model, HTGNN uses k as the time window size to slice the graph G. t Chinese v ga The attribute mask is taken as input and output v. ga The properties of the data. This process is similar to learning to reconstruct missing data from a spatiotemporal graph. In the inference process, we provide hand-labeled GAs in the early stages of the reverse dissection process, and then use a sliding time window for prediction; by stacking L heterogeneous spatiotemporal aggregation layers, we can obtain the embedding of the node GA at each time step.
[0095] like Figure 6 As shown, one embodiment of the present invention provides an information aggregation method, the specific steps of which include intra-frame information aggregation (S401) and inter-frame information aggregation (S402).
[0096] The intra-frame information of S401 is aggregated in the image tile G. tThis is performed on top of the target, with the aim of capturing nodes.<OA,Tool,IA,GA> The static topological relationship, due to G t Each node type in a graph has its own interpretable feature space, and the feature types and dimensions are different. The heterogeneity of the graph poses a challenge to direct message passing between nodes. Therefore, it is crucial to solve the graph heterogeneity problem before starting message passing.
[0097] Since different topological relationships exist depending on whether the tool and anatomy interact or not, it is necessary to consider the v... ia State-adaptive reasoning of topological relationships.
[0098] Therefore, as Figure 7 As shown, one embodiment of the present invention provides an intra-frame information aggregation method, the specific steps of which include:
[0099] S403. Project the feature spaces of different node types into the same space;
[0100] S404, according to v ia Information update v oa and v ins Information;
[0101] S405, according to the updated v oa and v ins Information reasoning v ga information.
[0102] Specifically, in step S403, which projects the feature spaces of different node types into the same space, the projection for a particular type involves using multiple heterogeneous temporal aggregation layers on each node before passing its features through. This projection maps the different feature vectors of the nodes to a common feature space. Mathematically, for a node of type φ(v) at timett, we map its feature vectors to the common feature space as follows:
[0103]
[0104] Furthermore, in S404 according to v ia Information update v oa and v ins In the information step, the step involves information aggregation on a heterogeneous graph, where the heterogeneous graph is:
[0105]
[0106] in, The heterogeneous graph at time step t; The set of node types at time step t for The interaction relationship represents the interaction state between surgical instruments and anatomical structures. Furthermore, at time step t, nodes of the same type... for:
[0107]
[0108] in, For nodes of the same type at time step t; * represents any node of type ins, oa, or ia; Merge any node attributes of ins, oa, or ia at time step t-2, time step t-1, and time step t.
[0109] According to v in S404 ia Information update v oa and v ins In the information steps, As the input to the final layer, it will represent the interaction state between surgical instruments and anatomical structures. Output, therefore, S404 according to v ia Information update v oa and v ins The information steps are as follows:
[0110]
[0111] in, This represents the nodes of the node type set at time step t; For node u at time step t, embed it with the previous layer; For the updated node v, the embedding of the last layer at time step t, and Θ IR These are trainable parameters, which are not shareable across node types, time steps, and aggregation layers; AGG IR For information aggregation functions, when it is the first level (l=1), AGG IR The output of the heterogeneous information processing step is taken as the input.
[0112] Since different node types play different roles in the training process, a self-attention mechanism is used to assign a weight coefficient reflecting the importance of each node. The weight coefficient is obtained as follows:
[0113]
[0114] in, σ is the weight coefficient of the node in layer l at time step t; σ(·) is the Leaky ReLU activation function; W t,l Let a be a trainable transformation matrix, which is a real matrix with d rows and d columns, where d is a hyperparameter; t,lLet be the attention vector, which is a d-dimensional real vector, where d is a hyperparameter; ‖· is the connection function, and its connection vector; further, the weight coefficients are normalized over all nodes using softmax:
[0115]
[0116] Furthermore, a weighted combination is performed to calculate mapping Embedding:
[0117]
[0118] Furthermore, in S405 according to the updated v oa and v ins Information reasoning v ga In the information step, based on the updated v oa and v ins Information can be obtained from heterogeneous graphs Get v ga The information, wherein the heterogeneous graph Represents a node<Tool,GA,OA> Spatial information on each map tile; The heterogeneous graph Node set on<Tool,GA,OA> ; for The interaction between them.
[0119] In S405, according to the updated v oa and v ins Information reasoning v ga Information steps will As the input to the last layer, and will represent The interaction between them As output, therefore, S405 is based on the updated v oa and v ins Information reasoning v ga The information steps are as follows:
[0120]
[0121] in, The embedding of the obtained u-nodes is aggregated through intra-frame information; The heterogeneous graph Node set on<Tool,GA,OA> u is The node inside; v is a set Nodes within; For the updated node v at time step t, the embedding with the last layer; Θ IIThese are trainable parameters, which are not shareable across node types, time steps, and aggregation layers. II With Θ IR Not shareable; AGG II It is an information aggregation function that provides its function through an attention mechanism.
[0122] Furthermore, the S402 inter-frame information aggregation will v ga Combining temporal neighborhood spatial embeddings to generate v ga The spatiotemporal embedding is achieved by using the attention mechanism of a transformer for aggregation.
[0123] Furthermore, the cystic duct node attributes obtained in the S400 dynamic information transmission step are embedded in the spatiotemporal data at each time step. The latent properties of the cystic tube nodes can be regressed using a multilayer perceptron model (MLP) to regress the parameters of the cystic tube node, namely the parameters of the abstract ellipse of the cystic tube and the representational properties of the cystic tube, including the position, orientation, and size of the cystic tube.
[0124] In the initial stage of the reversed manually recognized Calot triangulation video, the parameter E0 of the cyst tube node includes the center point of the ellipse in the initial stage. The rotation angle θ0 of the ellipse in the initial stage, the major axis a0 of the ellipse in the initial stage, and the minor axis b0 of the ellipse in the initial stage; the parameter E of the cyst tube node at time step t. t Including the center point of the ellipse at time step t The rotation angle θ of the ellipse at the initial stage at time step t. t The major axis a of the ellipse at the initial stage of time step t. t The minor axis b of the ellipse at the initial stage of time step t t Thus, the regression parameters δ{δ} of the cystic duct node are obtained. x δ y δ a δ b δ θ}for:
[0125] δ a =(a t / a0),δ b =(b t / b0),δ θ =θ t / π;
[0126] Where, δ x Let δ be the x-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. y Let δ be the y-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. aδ is the scaling factor of the major axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values. b δ is the scaling factor of the minor axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values. θ The ellipse rotation angle is the regression parameter of the cystic duct node. Here are the x-coordinates of the center point of the ellipse at time step t. Let a be the y-coordinate of the center point of the ellipse at time step t. t Let a0 be the major axis of the ellipse at time step t, and b be the major axis of the ellipse at the initial stage. t Let b0 be the minor axis of the ellipse at time step t, and b0 be the minor axis of the ellipse in the initial stage. θ t Let δ be the rotation angle of the ellipse at time step t, where δ∈E t .
[0127] The regression parameter loss L of the abstract ellipse of the cystic tube δ This is the sum of the regression parameter losses for each cyst node, i.e.:
[0128] L δ =L δx +L δY +L δa +L δb +λ δθ L δθ ;
[0129] Among them, L δ For the regression parameter loss of the ellipse abstracted from the cystic tube, L δx For the regression parameter loss of the x-axis coordinate of the center point of the ellipse abstracted from the cystic duct, L δy For the regression parameter loss of the y-axis coordinate of the center point of the ellipse abstracted from the cystic tube, L δa For the loss of the major axis regression parameters of the ellipse abstracted from the cystic tube, L δb For the elliptic minor axis regression parameter loss abstracted from the cystic tube, L δθ The loss is the regression parameter λ of the elliptical rotation angle abstracted from the cystic tube. δθ The weight parameter for the regression parameter of the ellipse rotation angle of the cystic tube is used to adjust the weight of the ellipse rotation angle in the training of the HTGNN model, and it is actually set to 0.1.
[0130] The method of this invention for automatic labeling of cystic ducts is advantageous because it involves extracting interpretable features and constructing a spatiotemporal heterogeneous graph network with small parameters. This eliminates the reliance on large labeled datasets and deep network-based feature extractors. Furthermore, the method can read the weight parameters of each node to provide a reference for algorithm adjustment, achieving independent control of the core algorithm. In the process of constructing the spatiotemporal heterogeneous graph, the interaction between instruments and tissues is introduced to assist in reasoning. This filters out highly variable and invalid background information in various complex scenarios, and the extracted interpretable features filter out noise interference in complex surgical scenarios. Therefore, the method of this invention has the advantages of strong robustness, high average coverage, and strong scene transfer capability. Moreover, due to the small network parameters of the constructed spatiotemporal heterogeneous graph, the method of this invention has the advantage of fast frame processing speed.
[0131] On the other hand, one embodiment of the present invention provides an automatic cystic duct marking system, such as... Figure 8 As shown, it includes:
[0132] The video analysis module 11 is used to analyze the relationships between various elements in the processed video; the spatiotemporal heterogeneous graph construction module 12 is used to abstract the surgical scene into a heterogeneous graph with multiple nodes based on the relationships between different elements in the reverse dissection process obtained in the S100 surgical video analysis step, and to model the dissection process as a temporal heterogeneous graph by connecting the heterogeneous graph through the same nodes at each time step; the node attribute extraction module 13 is used to remove noise interference in complex surgical scenes and extract the interpretable features of each node as its attributes; the dynamic information transmission module 14 is used to perform information aggregation of the HTGNN model to obtain the spatiotemporal embedding of the features of the spatiotemporal heterogeneous graph and the attributes of the cystus tube; and the target tissue parameter regression module 15 is used to obtain the position, orientation and size of the cystus tube and other attributes based on the spatiotemporal embedding of the cystus tube attributes obtained in the S400 dynamic information transmission step.
[0133] First, the image and video analysis module 11 of this device processes and extracts the relationship between the inferable target tissue and other elements in the video. Then, the spatiotemporal heterogeneous graph construction module 12 abstracts it into a heterogeneous graph with multiple nodes and combines it with spatiotemporal dynamic modeling to form a temporal heterogeneous graph (THG). Further, the node attribute extraction module 13 extracts the interpretable features of each node and eliminates noise interference from complex surgical scenes. Then, the dynamic information transmission module 14 aggregates the spatial topological relationships between each node and aggregates the temporal information to obtain the spatiotemporal embedding of the cystic duct attributes. Finally, the target tissue parameter regression module 15 regresses the cystic duct representation attribute parameters based on the obtained spatiotemporal embedding of the cystic duct attributes. This eliminates the need for a large number of labeled datasets and reliance on deep network-based feature extractors, and solves the problem of the lack of supervised samples and effective feature extractors in the automatic label generation process.
[0134] On the other hand, one embodiment of the present invention provides a terminal device including:
[0135] Memory, used to store computer programs;
[0136] A processor is used to implement the steps of the above-described automatic cystic duct marking method when executing the computer program.
[0137] On the other hand, one embodiment of the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs an automatic cystic duct marking method, comprising the following steps: S100, surgical image video analysis: First, the process of manually marking cystic ducts is analyzed through surgical image video. It is found that during the manual marking process, the cystic ducts are initially covered by adhesive tissue and then gradually exposed through dissection. Therefore, by reversing the video, the cystic ducts can be marked gradually from the fully exposed state back to the initial shape. The relationship between various factors in the video is analyzed during this process.
[0138] S200. Constructing a spatiotemporal heterogeneous graph: Based on the relationship between different factors in the reverse dissection process obtained from the surgical image video analysis step in S100, the surgical scene is abstracted into a heterogeneous graph with multiple nodes. By connecting the heterogeneous graph through the same node at each time step, the dissection process is modeled as a temporal heterogeneous graph.
[0139] S300. Extract interpretable features as graph node attributes: In complex surgical scenarios, noise interference is eliminated and interpretable features of each node are extracted as its attributes.
[0140] S400, Dynamic Information Transmission: The spatiotemporal heterogeneous graph obtained in the S200 step of constructing the spatiotemporal heterogeneous graph is used as input, and information aggregation is performed using the HTGNN model to obtain the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph.
[0141] S500, Target Tissue Parameter Regression: Based on the spatiotemporal embedding of the duct attributes obtained in the S400 dynamic information transmission step, the position, orientation, and size of the ducts are obtained for labeling the ducts.
[0142] Based on the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium. Common computer-readable storage media include: floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with perforated patterns, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), flash erasable programmable read-only memory (FLASH-EPROM), any other memory chip or cartridge, or any other computer-readable medium. Instructions can be further transmitted or received by a transmission medium. The term "transmission medium" can include any tangible or intangible medium used to store, encode, or carry instructions for machine execution, and includes digital or analog communication signals or other intangible media that facilitate the communication of such instructions. Transmission media include coaxial cables, copper wires, and optical fibers, which contain conductors for a bus used to transmit computer data signals. Computer-readable storage media include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods described in the various embodiments of the present invention.
[0143] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for automatic marking of cystic ducts, characterized in that, Includes the following steps: S100. Surgical video analysis: The process of manually marking the cystic tube is analyzed through surgical video analysis. By reversing the video, the cystic tube is marked back from its fully exposed state to its initial shape, and the relationship between various factors in the video is analyzed in this process. S200. Constructing a spatiotemporal heterogeneous graph: Based on the relationship between different factors in the reverse dissection process, the surgical scene is abstracted into a heterogeneous graph with multiple nodes. By connecting the heterogeneous graph through the same nodes at each time step, the dissection process is modeled as a spatiotemporal heterogeneous graph. The nodes of the spatiotemporal heterogeneous graph include instrument Tool, dissection operation Int, exposed cystic duct tissue OA, and overall cystic duct structure GA. S300. Extract interpretable features: In complex surgical scenarios, noise interference is eliminated to extract interpretable features of each node as its attributes; the extracted interpretable features include: S301. Extract frames from video slices and convert them into image datasets; S302. The semantic segmentation model Mask R-CNN processes video frames to segment objects; S303. Post-process the segmented objects, including manually marking and initializing the cysts; S304. Extract interpretable features from post-processed objects; S400, Dynamic Information Transmission: The spatiotemporal heterogeneous graph is used as input, and information aggregation is performed using the HTGNN model to obtain the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph. S500, Target Tissue Parameter Regression: Based on the spatiotemporal embedding of the duct attributes obtained in the S400 dynamic information transmission step, the characterization attributes of the ducts are obtained for labeling the ducts.
2. The automatic marking method for cystic ducts according to claim 1, characterized in that, In step S100, the relationship between the various factors includes the dynamic interaction between the surgical instruments and the exposed sac tissue; When surgical instruments interact with the adhesive tissue covering the sac, the spatial topology and temporal dynamics of the sac are often obscured by the instruments.
3. The automatic marking method for cystic ducts according to claim 1, characterized in that, In step S200, the spatiotemporal heterogeneous graph and node mapping are represented as the target node set {v}. ga ,v ins ,v ia ,v oa }, the edge type is mapped to the edge type set {e ia→ins ,e ia→oa ,e oa→ga ,e ins→ga } is used to convey spatial topological relationships.
4. The automatic marking method for cystic ducts according to claim 1, characterized in that, In step S304, the interpretable features include the position, orientation, and size of the GA; the position, orientation, and depth of the Tool; the position and size of the OA; and the contact state and contact points of the instrument's interaction with the anatomical structure.
5. The automatic marking method for cystic ducts according to claim 4, characterized in that, In step S304, the method for obtaining the interpretable features includes: The origin, eigenvalues, and eigenvectors of the GA abstract ellipse equivalent two-dimensional Gaussian distribution; the region of interest center and coverage area of OA; the region of interest center of Tool and the contour changes and contour angles after Hough Transform; the positions of Tool and OA and their region of interest centers.
6. The automatic marking method for cystic ducts according to claim 1, characterized in that, In step S400, the information aggregation includes the following steps: S403. Project the feature spaces of different node types into the same space; S404, according to v ia Information update v oa and v ins Information; S405, according to the updated v oa and v ins Information reasoning v ga information.
7. The automatic marking method for cystic ducts according to claim 6, characterized in that, In step S404, according to v ia Information update v oa and v ins The information is: ; in, The node type set at time step t { }; For node u at time step t, embed it with the previous layer; Let v be the last layer embedding of the updated node at time step t, and v { }; These are trainable parameters; For information aggregation functions, when it is the first level (l=1), The output of the heterogeneous information processing step is taken as the input.
8. The automatic marking method for cystic ducts according to claim 7, characterized in that, In step S404, a self-attention mechanism is used to assign a weight coefficient to each node that reflects its importance. The weight coefficient is obtained as follows: ; in, The weight coefficients of the nodes in layer l at time step t; ( () is the LeakyReLU activation function; Let be a trainable transformation matrix, which is a real matrix with d rows and d columns, where d is a hyperparameter; Let be the attention vector, which is a d-dimensional real vector, where d is a hyperparameter; Let be the connection function and its connection vector.
9. The automatic marking method for cystic ducts according to claim 6, characterized in that, In step S405, based on the updated v oa and v ins Information reasoning v ga Information, namely: ; in, The embedding of the obtained u-nodes is aggregated through intra-frame information; The heterogeneous graph Node set on<Tool, GA, OA> u is The nodes within; v is the set { Nodes within}; The updated node v is embedded with the last layer at time step t; These are trainable parameters, which are not shareable across node types, time steps, and aggregation layers. and Not shareable; It is an information aggregation function that provides its function through an attention mechanism.
10. The automatic marking method for cystic ducts according to claim 1, characterized in that, In step S500, the characterization attribute of the cyst is the target tissue regression parameter δ{δ x δ y δ a δ b δ θ }for: d x = ,d y = ,d a =(a) t / a0),δ b =(b) t / b0),δ θ =θ t / p? Where, δ x Let δ be the x-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. y Let δ be the y-axis coordinate of the center point of the ellipse for the regression parameters of the cystic duct node. a δ is the scaling factor of the major axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values. b δ is the scaling factor of the minor axis of the ellipse representing the regression parameters of the cystic duct nodes relative to the initial values. θ The ellipse rotation angle is the regression parameter of the cystic duct node. Here are the x-coordinates of the center point of the ellipse at time step t. Let a be the y-coordinate of the center point of the ellipse at time step t. t Let a0 be the major axis of the ellipse at time step t, and b be the major axis of the ellipse at the initial stage. t Let b0 be the minor axis of the ellipse at time step t, and b0 be the minor axis of the ellipse in the initial stage. θ t Let be the rotation angle of the ellipse at time step t.
11. An automatic cystic duct marking system, implemented using an automatic cystic duct marking method as described in any one of claims 1-10, characterized in that, include: The image and video analysis module (11) is used to analyze the relationships between various factors in the processed video; A spatiotemporal heterogeneous graph module (12) is constructed to abstract the surgical scene into a heterogeneous graph with multiple nodes based on the relationship between different factors in the reverse dissection process obtained from image and video analysis. The dissection process is modeled as a spatiotemporal heterogeneous graph by connecting the heterogeneous graphs through the same nodes at each time step. The node attribute extraction module (13) is used to eliminate noise interference in complex surgical scenarios and extract the interpretable features of each node as its attributes. The dynamic information transmission module (14) is used to perform information aggregation of the HTGNN model, thereby obtaining the spatiotemporal embedding of the features and ductal attributes of the spatiotemporal heterogeneous graph. The target tissue parameter regression module (15) is used to obtain the characterization attributes of the ducts based on the spatiotemporal embedding of the obtained duct attributes for labeling the ducts.
12. A terminal device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the automatic cystic duct marking method as described in any one of claims 1 to 10 when executing the computer program.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the automatic cystic duct marking method as described in any one of claims 1-10.
Citation Information
Patent Citations
Video abstraction method based on double-self-attention capsule network
CN111984820A
Zero sample image recognition method and system and storable medium
CN116433969A
