Intelligent navigation method and system for surgical scene based on graph neural network

Abstract

The application relates to the technical field of medical image processing and artificial intelligence, and provides a surgery scene intelligent navigation method and system based on a graph neural network, which comprises the following steps: acquiring a real-time video stream of a surgery scene, and identifying a segmentation result of a target object by using a pre-trained instantiation semantic segmentation model; extracting spatial structure features, position features and semantic features for each target object; taking each target object as a graph node, and fusing the three kinds of modal features to form an initial node feature vector; constructing edges of the graph according to spatial relationships; iteratively updating node and edge features by using a self-attention graph neural network; and outputting navigation information based on the enhanced graph structure, including an anatomical plane deviation early warning, a key structure proximity prompt or a recommended cutting plane. In this way, the safety of surgery operation is improved.

Description

Technical Field

[0001] This application relates to the fields of medical image processing and artificial intelligence technology, specifically to a method and system for intelligent navigation of surgical scenes based on graph neural networks. Background Technology

[0002] Total mesorectal excision (TME) is the standard surgical procedure for mid-to-low rectal cancer. Its core principle is meticulous dissection at the avascular level between the rectal fascia proper and the visceral fascia of the pelvic wall, completely removing the rectum and surrounding mesentery. In recent years, robot-assisted total mesorectal excision (R-TME) has significantly improved the ability to perform precise anatomical procedures due to its advantages such as high-definition three-dimensional visualization, multi-degree-of-freedom robotic arms, and tremor filtering.

[0003] However, in the complex pelvic anatomical environment, the identification of key structures such as the mesorectal space, blood vessels, and nerves is highly dependent on the surgeon's experience. In particular, obese patients, patients with a history of pelvic surgery, or patients who have undergone preoperative radiotherapy often have unclear anatomical landmarks, which increases surgical risks.

[0004] It is evident that the surgical procedures involved in these technologies carry significant operational risks. Summary of the Invention

[0005] This application aims to address the problem of high operational risks in surgical procedures in related technologies by providing a method and system for intelligent navigation of surgical scenarios based on graph neural networks.

[0006] To solve the above problems, this application is implemented as follows:

[0007] Firstly, this application provides an intelligent navigation method for surgical scenes based on graph neural networks, including: A real-time video stream of the surgical scene is acquired, and the real-time video stream is input into a pre-trained instantiated semantic segmentation model. The instantiated semantic segmentation model outputs segmentation results for multiple target objects. The segmentation results for the target objects include segmentation masks, category labels, and confidence scores, and each target object corresponds to a surgical instrument or anatomical structure. Using the segmentation mask of the target object as a spatial constraint, feature maps at corresponding positions are extracted from the intermediate layer of the instantiated semantic segmentation model. The feature maps are then filled with smooth features within the mask region using bitone interpolation. Global pooling is then performed on the filled feature maps to obtain the spatial structure feature vector of the target object. The centroid coordinates of the target object are calculated based on the segmentation mask of the target object, and the centroid coordinates are encoded in high dimension to obtain the position feature vector of the target object. The corresponding text description is determined based on the pixel area and category label of the segmentation mask of the target object. The text description is then input into a pre-trained text encoder to extract the semantic embedding vector and obtain the semantic feature vector of the target object. The spatial structure feature vector, the location feature vector, and the semantic feature vector are concatenated, and feature transformation and dimensionality reduction are performed through a multilayer perceptron to output a fused feature vector. Based on the confidence score and the fused feature vector, the initial node feature vector of the target object is obtained. A graph structure including multiple nodes and edges is constructed based on multiple target objects. Each target object corresponds to one node. The edges are determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined. An initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector. The multilayer graph neural network based on the self-attention mechanism iteratively updates the initial node feature matrix to obtain the enhanced node feature matrix. Based on the enhanced node feature matrix, navigation information is generated, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

[0008] Secondly, this application provides an intelligent navigation system for surgical scenarios based on graph neural networks, comprising: The acquisition module is used to acquire a real-time video stream of the surgical scene, input the real-time video stream into a pre-trained instantiated semantic segmentation model, and the instantiated semantic segmentation model outputs segmentation results of multiple target objects. The segmentation results of the target objects include segmentation masks, category labels and confidence scores, and each target object corresponds to a surgical instrument or anatomical structure. The spatial feature extraction module is used to extract feature maps of corresponding positions from the intermediate layer of the instantiated semantic segmentation model with the segmentation mask of the target object as the spatial constraint, perform feature smoothing filling on the feature map in the mask region by bitonic interpolation, and perform global pooling on the filled feature map to obtain the spatial structure feature vector of the target object. The location feature extraction module is used to calculate the corresponding centroid coordinates based on the segmentation mask of the target object, perform high-dimensional location encoding on the centroid coordinates, and obtain the location feature vector of the target object. The semantic feature extraction module is used to determine the corresponding text description based on the pixel area and category label of the segmentation mask of the target object, input the text description into the pre-trained text encoder, extract the semantic embedding vector, and obtain the semantic feature vector of the target object. The feature fusion module is used to concatenate the spatial structure feature vector, the position feature vector, and the semantic feature vector, perform feature transformation and dimensionality reduction through a multilayer perceptron, output a fused feature vector, and obtain the initial node feature vector of the target object based on the confidence score and the fused feature vector. The graph construction module is used to construct a graph structure including multiple nodes and edges based on multiple target objects. Each target object corresponds to one node. The edges are determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined. The initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector. The update module is used to iteratively update the initial node feature matrix using a multi-layer graph neural network based on a self-attention mechanism to obtain an enhanced node feature matrix. The navigation module is used to generate navigation information based on the enhanced node feature matrix, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

[0009] Thirdly, this application provides a terminal device including a processor and a memory, wherein the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0010] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0011] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the steps of the method described in the first aspect.

[0012] In a sixth aspect, this application provides a computer program product stored in a storage medium, which is executed by at least one processor to perform the steps of the method described in the first aspect.

[0013] Compared with existing technologies, this application has the following advantages: By modeling surgical instruments and anatomical structures as graph nodes and the spatial proximity relationships between nodes as edges, a structured graph representation is formed. This graph structure explicitly expresses the spatial topological relationships between entities in the surgical scene, enabling the surgeon to intuitively understand the overall layout of the current operating area. Through iterative updates of the graph neural network, the features of each node aggregate information from its neighboring nodes, further enhancing the ability to express spatial relationships. Thus, when doctors perform surgical operations based on navigation information, the operational risks of the surgery can be effectively reduced, and the safety of the surgical procedure can be improved. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating an embodiment of the intelligent navigation method for surgical scenes based on graph neural networks provided in this application. Figure 2 This is a schematic diagram of the structure of an intelligent surgical scene navigation system based on graph neural networks provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0018] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0019] The following describes the intelligent navigation method for surgical scenes based on graph neural networks provided in this application.

[0020] See Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the intelligent navigation method for surgical scenes based on graph neural networks provided in this application. Figure 1 The intelligent navigation method for surgical scenes based on graph neural networks shown can be executed by terminal devices such as mobile phones and computers.

[0021] like Figure 1 As shown, the intelligent surgical scene navigation method based on graph neural networks provided in this application may include the following steps: Step 101: Obtain the real-time video stream of the surgical scene, input the real-time video stream into the pre-trained instantiated semantic segmentation model, the instantiated semantic segmentation model outputs the segmentation results of multiple target objects, the segmentation results of the target objects include segmentation mask, category label and confidence score, and each target object corresponds to a surgical instrument or anatomical structure.

[0022] In this embodiment, a surgical robot system or a high-definition endoscope system can be connected via a video acquisition interface to obtain a real-time video stream of the surgical scene.

[0023] Before inputting the real-time video stream into the pre-trained instantiated semantic segmentation model, the real-time video stream can be pre-processed, such as adjusting the size of the real-time video stream or normalizing the pixel values ​​of the real-time video stream, so that the pre-processed real-time video stream is compatible with the pre-trained instantiated semantic segmentation model.

[0024] In some embodiments, the pre-trained instantiated semantic segmentation model is a lightweight visual converter model trained by a knowledge distillation method. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter. During training, the model parameters of the student network are optimized by spatial distillation loss, temporal distillation loss, and main task segmentation loss to obtain an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0025] In some embodiments, the training method for the pre-trained instantiated semantic segmentation model includes: Acquire training data, which includes video images captured by the surgical robot and corresponding pixel-level annotations. The annotation categories of the pixel-level annotations include surgical instruments and anatomical structures. A teacher network and a student network are constructed. The teacher network is a pre-trained general image segmentation model, and its network parameters are frozen during training. The student network is a visual converter, which is a lightweight model to be trained. The video image is simultaneously input into the teacher network and the student network. The multi-scale feature map of the teacher network and the serialized feature vector of the student network are extracted respectively, and the pixel-level segmentation prediction map output by the student network is obtained. The serialized feature vector is mapped to the same spatial dimension as the multi-scale feature map by a learnable linear projection layer to obtain the projected features of the serialized feature vector; Calculate the spatial distillation loss between the projected features and the multi-scale feature map, calculate the temporal distillation loss between the serialized feature vectors of adjacent video frames, and calculate the main task segmentation loss between the pixel-level segmentation prediction map and the pixel-level annotation; The total loss function is obtained by weighted summing of the spatial distillation loss, the temporal distillation loss, and the main task segmentation loss. The network parameters of the student network are updated by backpropagation based on the total loss function until convergence, resulting in an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0026] Step 102: Using the segmentation mask of the target object as a spatial constraint, extract the feature map of the corresponding position from the intermediate layer of the instantiated semantic segmentation model, perform feature smoothing filling on the feature map in the mask region by bitonic interpolation, and perform global pooling on the filled feature map to obtain the spatial structure feature vector of the target object.

[0027] Step 103: Calculate the centroid coordinates corresponding to the segmentation mask of the target object, and perform high-dimensional position encoding on the centroid coordinates to obtain the position feature vector of the target object.

[0028] Step 104: Determine the corresponding text description based on the pixel area and category label of the segmentation mask of the target object, input the text description into the pre-trained text encoder, extract the semantic embedding vector, and obtain the semantic feature vector of the target object.

[0029] Step 105: Concatenate the spatial structure feature vector, the position feature vector, and the semantic feature vector, perform feature transformation and dimensionality reduction through a multilayer perceptron, and output a fused feature vector. Based on the confidence score and the fused feature vector, obtain the initial node feature vector of the target object.

[0030] In this embodiment, the spatial structure feature vector, positional feature vector, and semantic feature vector extracted in the preceding steps are fused. A multilayer perceptron is then used for feature transformation and dimensionality reduction, outputting a fused feature vector. This fused feature vector is then combined with the confidence score of the target object to generate an initial node feature vector for each target object. This initial node feature vector serves as the initial representation of nodes in the graph structure and is used in subsequent iterative updates of the graph neural network.

[0031] This setup allows high-confidence target objects to play a dominant role in the graph structure, suppresses the influence of low-confidence target objects on graph reasoning, and avoids interference from low-quality detection results on navigation decisions.

[0032] Step 106: Construct a graph structure including multiple nodes and edges based on multiple target objects. Each target object corresponds to one node. The edges are determined based on the spatial relationship between two target objects. The edge feature vector of the edge is determined. The initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector.

[0033] The edge is determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined, including: When the distance between the center points of two target objects is less than a preset threshold, an undirected edge is established between the corresponding nodes, and the edge feature vector of the undirected edge is calculated. The edge feature vector is used to characterize the distance information and size difference information between the nodes. Step 107: The initial node feature matrix is ​​iteratively updated using a multilayer graph neural network based on a self-attention mechanism to obtain an enhanced node feature matrix.

[0034] In this embodiment, the initial node feature matrix is ​​input into a multi-layer graph neural network with a self-attention mechanism. Through multi-layer iterative updates, the features of each node can aggregate information from its neighboring nodes, thereby enhancing the ability of node features to express spatial topological relationships in the surgical scene. After multi-layer updates, the enhanced node feature matrix is ​​output for subsequent navigation decisions.

[0035] In some embodiments, a graph attention network (GAT) can be selected as the basic graph neural network architecture. GAT can assign different weights to different neighbor nodes through a self-attention mechanism, and can adaptively learn the differences in importance between nodes. It is particularly suitable for the characteristics of different interaction intensities between instruments and different anatomical structures in surgical scenarios.

[0036] Step 108: Based on the enhanced node feature matrix, generate navigation information, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

[0037] In some embodiments, the feature distance between the surgical instrument node and the anatomical plane node can be determined based on the enhanced node feature matrix, and when the feature distance between the surgical instrument node and the anatomical plane node exceeds a first threshold, the operation plane is judged to be deviated and an early warning signal is generated.

[0038] In some embodiments, the feature distance between surgical instrument nodes and risk structure nodes can be determined based on the enhanced node feature matrix, and when the feature distance between surgical instrument nodes and risk structure nodes is less than a second threshold, it is determined that the instrument is close to the critical structure and a proximity prompt is generated.

[0039] In some embodiments, the boundaries between adjacent anatomical structure nodes can be identified based on the enhanced node feature matrix, and the recommended cutting direction can be predicted and displayed by combining the motion trajectory of instrument nodes in historical frames.

[0040] In some embodiments, the surgical images can be visualized, specifically including: Draw a red highlighted outline around the adjacent risk structure; Draw a red dashed line connecting the instrument and the risk structure; A risk level indicator is displayed at the edge of the screen, with different colors corresponding to different risk levels.

[0041] In some embodiments, navigation warning information can also be output via voice prompts.

[0042] In this application, surgical instruments and anatomical structures are modeled as graph nodes, and the spatial proximity relationships between nodes are modeled as edges, forming a structured graph representation. This graph structure explicitly expresses the spatial topological relationships between entities in the surgical scene, enabling the surgeon to intuitively understand the overall layout of the current operating area. Through iterative updates of the graph neural network, the features of each node aggregate information from its neighboring nodes, further enhancing the ability to express spatial relationships. Thus, when surgeons perform operations based on navigation information, the operational risks of the surgery can be effectively reduced, and the safety of the surgical procedure can be improved.

[0043] In some embodiments, the proximity of the surgical instrument node and the risk structure node to the critical structure is determined in real time by calculating the feature distance in the enhanced feature space. When the feature distance is lower than a safe threshold, the system automatically generates a proximity prompt and presents it in a visual form such as a red highlighted outline, a dashed connecting line, and a risk level indicator, effectively reducing the operational risk of the surgery.

[0044] In some embodiments, by calculating the characteristic distance between the surgical instrument node and the anatomical plane node, it is possible to monitor in real time whether the instrument deviates from the correct anatomical plane. When the characteristic distance exceeds the warning threshold, the system immediately generates a deviation warning, prompting the surgeon to adjust the operation direction, thereby reducing the operational risk of the surgery.

[0045] In some embodiments, by identifying the spatial boundary lines between adjacent anatomical structural nodes and predicting the instrument's trajectory, the predicted motion direction is weighted and fused with the boundary normal direction to generate an intelligent cutting direction recommendation. This recommended direction can be displayed as a semi-transparent blue arrow overlaid on the surgical screen, providing the surgeon with intuitive operational guidance.

[0046] The intelligent surgical scene navigation method based on graph neural networks provided in this application effectively solves the problem of high surgical operation risk in existing technologies through technical innovations such as explicit modeling of spatial topological relationships, real-time proactive risk warning, intelligent cutting direction recommendation, adaptive weighting of confidence, multimodal feature fusion, and dynamic graph update mechanism. This significantly improves the safety and accuracy of surgery. See Figure 2 , Figure 2 This is a schematic diagram of the structure of a surgical scene intelligent navigation system based on graph neural networks provided in an embodiment of this application. Figure 2As shown, system 200 includes: The acquisition module 201 is used to acquire a real-time video stream of the surgical scene, input the real-time video stream into a pre-trained instantiated semantic segmentation model, and the instantiated semantic segmentation model outputs segmentation results of multiple target objects. The segmentation results of the target objects include segmentation masks, category labels and confidence scores, and each target object corresponds to a surgical instrument or anatomical structure. The spatial feature extraction module 202 is used to extract the feature map of the corresponding position from the intermediate layer of the instantiated semantic segmentation model with the segmentation mask of the target object as the spatial constraint, perform feature smoothing filling on the feature map in the mask area by bitonic interpolation, and perform global pooling on the filled feature map to obtain the spatial structure feature vector of the target object. The position feature extraction module 203 is used to calculate the corresponding centroid coordinates based on the segmentation mask of the target object, perform high-dimensional position encoding on the centroid coordinates, and obtain the position feature vector of the target object. The semantic feature extraction module 204 is used to determine the corresponding text description based on the pixel area and category label of the segmentation mask of the target object, input the text description into a pre-trained text encoder, extract the semantic embedding vector, and obtain the semantic feature vector of the target object. The feature fusion module 205 is used to concatenate the spatial structure feature vector, the position feature vector, and the semantic feature vector, perform feature transformation and dimensionality reduction through a multilayer perceptron, output a fused feature vector, and obtain the initial node feature vector of the target object based on the confidence score and the fused feature vector. Graph construction module 206 is used to construct a graph structure including multiple nodes and edges based on multiple target objects. Each target object corresponds to a node. The edges are determined based on the spatial relationship between two target objects. The edge feature vector of the edge is determined. The initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector. Update module 207 is used to iteratively update the initial node feature matrix using a multi-layer graph neural network based on a self-attention mechanism to obtain an enhanced node feature matrix; The navigation module 208 is used to generate navigation information based on the enhanced node feature matrix, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

[0047] Optionally, the edge is determined based on the spatial relationship between the two target objects, and the edge feature vector of the edge is determined, including: When the distance between the center points of two target objects is less than a preset threshold, an undirected edge is established between the corresponding nodes, and the edge feature vector of the undirected edge is calculated. The edge feature vector is used to characterize the distance information and size difference information between the nodes.

[0048] Optionally, the pre-trained instantiated semantic segmentation model is a lightweight visual converter model trained using a knowledge distillation method. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter. During training, the model parameters of the student network are optimized using spatial distillation loss, temporal distillation loss, and main task segmentation loss to obtain an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

[0049] The surgical scene intelligent navigation system based on graph neural networks provided in this application can achieve the functions described in this application. Figure 1 The various processes in the method embodiments, and the ways to achieve the same beneficial effects, will not be repeated here to avoid repetition.

[0050] like Figure 3 As shown, this application also provides a terminal device, including a processor 301 and a memory 302. The memory 302 stores a program or instructions that can run on the processor 301. When the program or instructions are executed by the processor 301, they implement the various steps of the above-described embodiment of the intelligent navigation method for surgical scenes based on graph neural networks and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0051] It should be noted that the terminal device in this application can be a terminal or other devices besides a terminal. For example, the terminal device can be a mobile phone, tablet computer, laptop computer, etc., and this application does not make any specific limitation.

[0052] This application also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described embodiments of the intelligent navigation method for surgical scenes based on graph neural networks, and achieve the same technical effect. To avoid repetition, these will not be described again here.

[0053] The processor is the processor in the terminal device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (Read-Only Memory). Only memory (ROM), random access memory (RAM), magnetic disks or optical disks, etc.

[0054] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the intelligent navigation method for surgical scenes based on graph neural networks, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0055] It should be understood that the chip mentioned in this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0056] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described embodiment of the intelligent navigation method for surgical scenes based on graph neural networks, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0057] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0058] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as a read-only memory). The device includes a number of instructions in a ROM (random access memory), RAM (magnetic disk), or optical disk to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0059] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A surgical scene intelligent navigation method based on graph neural networks, characterized in that, include: A real-time video stream of the surgical scene is acquired, and the real-time video stream is input into a pre-trained instantiated semantic segmentation model. The instantiated semantic segmentation model outputs segmentation results for multiple target objects. The segmentation results for the target objects include segmentation masks, category labels, and confidence scores, and each target object corresponds to a surgical instrument or anatomical structure. Using the segmentation mask of the target object as a spatial constraint, feature maps at corresponding positions are extracted from the intermediate layer of the instantiated semantic segmentation model. The feature maps are then filled with smooth features within the mask region using bitone interpolation. Global pooling is then performed on the filled feature maps to obtain the spatial structure feature vector of the target object. The centroid coordinates of the target object are calculated based on the segmentation mask of the target object, and the centroid coordinates are encoded in high dimension to obtain the position feature vector of the target object. The corresponding text description is determined based on the pixel area and category label of the segmentation mask of the target object. The text description is then input into a pre-trained text encoder to extract the semantic embedding vector and obtain the semantic feature vector of the target object. The spatial structure feature vector, the location feature vector, and the semantic feature vector are concatenated, and feature transformation and dimensionality reduction are performed through a multilayer perceptron to output a fused feature vector. Based on the confidence score and the fused feature vector, the initial node feature vector of the target object is obtained. A graph structure including multiple nodes and edges is constructed based on multiple target objects. Each target object corresponds to one node. The edges are determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined. An initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector. The multilayer graph neural network based on the self-attention mechanism iteratively updates the initial node feature matrix to obtain the enhanced node feature matrix. Based on the enhanced node feature matrix, navigation information is generated, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

2. The method according to claim 1, characterized in that, The edge is determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined, including: When the distance between the center points of two target objects is less than a preset threshold, an undirected edge is established between the corresponding nodes, and the edge feature vector of the undirected edge is calculated. The edge feature vector is used to characterize the distance information and size difference information between the nodes.

3. The method according to claim 1 or 2, characterized in that, The pre-trained instantiated semantic segmentation model is a lightweight visual converter model trained using the knowledge distillation method. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter. During training, the model parameters of the student network are optimized using spatial distillation loss, temporal distillation loss, and main task segmentation loss to obtain an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

4. A surgical scene intelligent navigation system based on graph neural networks, characterized in that, include: The acquisition module is used to acquire a real-time video stream of the surgical scene, input the real-time video stream into a pre-trained instantiated semantic segmentation model, and the instantiated semantic segmentation model outputs segmentation results of multiple target objects. The segmentation results of the target objects include segmentation masks, category labels and confidence scores, and each target object corresponds to a surgical instrument or anatomical structure. The spatial feature extraction module is used to extract feature maps of corresponding positions from the intermediate layer of the instantiated semantic segmentation model with the segmentation mask of the target object as the spatial constraint, perform feature smoothing filling on the feature map in the mask region by bitonic interpolation, and perform global pooling on the filled feature map to obtain the spatial structure feature vector of the target object. The location feature extraction module is used to calculate the corresponding centroid coordinates based on the segmentation mask of the target object, perform high-dimensional location encoding on the centroid coordinates, and obtain the location feature vector of the target object. The semantic feature extraction module is used to determine the corresponding text description based on the pixel area and category label of the segmentation mask of the target object, input the text description into the pre-trained text encoder, extract the semantic embedding vector, and obtain the semantic feature vector of the target object. The feature fusion module is used to concatenate the spatial structure feature vector, the position feature vector, and the semantic feature vector, perform feature transformation and dimensionality reduction through a multilayer perceptron, output a fused feature vector, and obtain the initial node feature vector of the target object based on the confidence score and the fused feature vector. The graph construction module is used to construct a graph structure including multiple nodes and edges based on multiple target objects. Each target object corresponds to one node. The edges are determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined. The initial node feature matrix of the graph structure is constructed based on the edge feature vector and the initial node feature vector. The update module is used to iteratively update the initial node feature matrix using a multi-layer graph neural network based on a self-attention mechanism to obtain an enhanced node feature matrix. The navigation module is used to generate navigation information based on the enhanced node feature matrix, wherein the navigation information includes at least one of the following: anatomical plane deviation warning, key structure approach prompt, and recommended cutting direction.

5. The system according to claim 4, characterized in that, The edge is determined based on the spatial relationship between two target objects, and the edge feature vector of the edge is determined, including: When the distance between the center points of two target objects is less than a preset threshold, an undirected edge is established between the corresponding nodes, and the edge feature vector of the undirected edge is calculated. The edge feature vector is used to characterize the distance information and size difference information between the nodes.

6. The system according to claim 4 or 5, characterized in that, The pre-trained instantiated semantic segmentation model is a lightweight visual converter model trained using the knowledge distillation method. The teacher network is a pre-trained general image segmentation model, and the network parameters of the teacher network are frozen during training. The student network is a visual converter. During training, the model parameters of the student network are optimized using spatial distillation loss, temporal distillation loss, and main task segmentation loss to obtain an instantiated semantic segmentation model that can be used in real-time surgical scenarios.

Images