A structure-prior-guided endoscopic surgical area scene understanding and safety early warning system
The system for understanding and providing safety warnings for endoscopic surgical areas guided by prior knowledge of the structure solves the problem of lack of semantic information in 3D reconstruction during endoscopic surgery. It enables real-time risk assessment and 3D semantic understanding of surgical instruments and tissue structures, thereby improving the safety and navigation efficiency of surgical procedures.
Patent Information
- Application Number
- CN202611473207.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies make it difficult to include semantic information in three-dimensional reconstruction results during endoscopic surgery, and two-dimensional semantic segmentation results cannot reflect the true spatial relationship of tissue categories, leading to increased surgical difficulty and potential risks.
An endoscopic surgical area scene understanding and safety early warning system guided by structural priors is adopted. By fusing multiple prompt words in the medical field and coarse and fine granular structural priors, combined with semantic segmentation module, semantic indexing module, semantic-guided 3D reconstruction module and visualization interaction module, a 3D Gaussian semantic point cloud is constructed to realize real-time calculation of spatial distance between surgical instruments and tissue structures and risk assessment.
It achieves high-precision 3D reconstruction of surgical scenes, maintains semantic continuity, and can maintain stable 3D semantic representation in the face of tissue deformation, instrument movement and changes in endoscopic perspective. It provides surgical area structure query, target localization and visualization warning, and improves surgical safety and navigation efficiency.
Smart Images

Figure CN123118983A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a structure-prior-guided endoscopic surgical area scene understanding and safety early warning system. Background Technology
[0002] With the development of robot-assisted surgery and intelligent surgery, intraoperative scene perception and 3D environment understanding have become crucial foundational technologies for surgical navigation, intraoperative decision support, and autonomous surgical systems. Compared to traditional open surgery, laparoscopic surgery offers advantages such as less trauma, faster recovery, and fewer complications. However, endoscopic surgery is typically performed in narrow, closed surgical areas. Intraoperative images suffer from issues such as tissue deformation, instrument occlusion, uneven lighting, mirror reflections, smoke, and blurred tissue boundaries. Surgeons cannot continuously understand different tissues, instruments, and their spatial relationships solely based on two-dimensional endoscopic videos.
[0003] CAT-Seg utilizes a visual encoder and a text encoder to construct an image-text cost volume, achieving open-vocabulary semantic segmentation through cost aggregation. However, general natural image cues are insufficient to fully describe endoscopic surgical scenarios; relying solely on image-text semantic similarity has limited ability to distinguish fine-grained instrument components, slender tissues, and tissues with similar appearances.
[0004] SurgTPGS combines semantic features with time-varying 3D Gaussian splashing to achieve 3D semantic reconstruction and text querying of surgical scenes. However, SurgTPGS preprocessing relies on SAM to first generate a class-independent mask, and then aggregate CAT-Seg features within the mask. Since the SAM mask and semantic categories are provided by different networks, problems such as a semantic object being fragmented, adjacent objects being incorrectly merged, and inconsistencies between mask boundaries and semantic boundaries can easily arise. In situations involving complex anatomical structures, tissue occlusion, and frequent instrument movement, surgeons need to rely on experience to judge the spatial relationships between tissues, which can easily increase the difficulty and potential risks of surgical operations. Therefore, constructing a 3D surgical scene model that can simultaneously express tissue spatial structure and semantic information has become an important research direction for intelligent surgical navigation, intraoperative decision support, and autonomous surgical systems.
[0005] Existing technology 1 discloses a surgical interaction method combining binocular vision, medical image semantic segmentation, 3D reconstruction, and augmented reality. This method acquires intraoperative images of medical tissue using a calibrated binocular camera, performs semantic segmentation on the left and right eye images using PyramidVision Transformer, uses the semantic segmentation prediction map as 3D modeling data, and calculates the 3D coordinates of the medical tissue using the least squares method in conjunction with the binocular camera's intrinsic and extrinsic parameters. Subsequently, the reconstructed 3D model is imported into Unity, and the positional relationship between the virtual model and the real medical tissue is determined through manual labeling and rigid registration. It is then displayed on a HoloLens 2 augmented reality device, enabling intraoperative observation and knowledge guidance of medical tissue. However, this approach primarily reconstructs the geometric model of the target tissue based on the segmentation prediction map, failing to incorporate category semantic features as optimizable attributes of the 3D model. This results in a lack of a unified 3D representation that simultaneously includes color, space, morphology, and high-dimensional semantic features. Furthermore, its 3D reconstruction employs binocular matching and the least squares method, relying on manual labeling for rigid spatial registration. For dynamic surgical scenarios with continuous deformation, instrument occlusion, and changing viewpoints, maintaining geometric continuity and semantic consistency across multiple frames is challenging.
[0006] Existing technology two discloses a thoracoscopic projection method that combines interactive voice, thoracoscopic image segmentation, lung 3D reconstruction, and keypoint matching. This method first determines semantic keypoints based on the doctor's interactive voice content; it then uses MaskR-CNN to coarsely segment the region of interest in the thoracoscopic image and U-Net for fine segmentation; simultaneously, it preprocesses and 3D reconstructs the lung CT image. Subsequently, it extracts 2D keypoints from the thoracoscopic segmentation results and matches these keypoints with the lung 3D reconstruction results, finally outputting the corresponding thoracoscopic projection result based on the semantic keypoints. However, this method primarily uses semantic information to determine and project discrete keypoints, and the 3D reconstruction object mainly comes from preoperative lung CT images, failing to directly incorporate the dense semantic features from intraoperative endoscopic images into the 3D reconstruction process. Therefore, this scheme cannot form a dense 3D semantic representation covering the entire surgical area based on intraoperative video, nor can it continuously express the spatial distribution of different tissues, instruments, and their boundaries in a dynamic surgical scenario.
[0007] The existing CAT-Seg method uses general text cues for natural images to construct the image-text cost volume, which is insufficient to fully describe medical features such as endoscopic imaging methods, surgical types, tissue visibility, and endoscopic illumination. Furthermore, existing methods primarily classify based on the semantic similarity between image and text features, lacking specific constraints on tissue edges, textures, and instrument structures. This leads to problems such as category confusion and inaccurate boundary segmentation in cases of reflection, occlusion, blurred boundaries, and similar appearances of different instrument components.
[0008] It is evident that existing technologies can achieve 3D reconstruction of surgical scenes and 2D medical semantic segmentation separately, but these are mostly processed as independent tasks. 3D reconstruction results typically only contain geometric information and lack semantic expressive power; while 2D semantic segmentation results can identify tissue categories, they cannot reflect their true spatial relationships. Therefore, how to directly introduce medical semantic information into the 3D reconstruction process, restoring the scene's geometric structure while simultaneously expressing tissue categories and spatial semantic modeling, and constructing a 3D surgical scene model with semantic information, has become a crucial problem urgently needing to be solved in the field of surgical scene understanding. Summary of the Invention
[0009] To address the aforementioned issues, this invention provides a structure-prior-guided endoscopic surgical area scene understanding and safety warning system. By fusing multiple prompt words from the medical field and using coarse- and fine-grained structural priors, it improves the semantic segmentation accuracy of tissues, instruments, and fine-grained instrument components in endoscopic images.
[0010] A structure-prior-guided endoscopic surgical area scene understanding and safety early warning system includes a semantic segmentation module, a semantic indexing module, a semantic-guided 3D reconstruction module, a visualization interaction module, and an intraoperative early warning module; The semantic segmentation module is used to obtain the two-dimensional semantic segmentation result of the endoscopic surgical area image, and at the same time obtain the text semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image. The semantic indexing module is used to determine the predicted fine-grained semantic category corresponding to each pixel based on the semantic index map generated by the two-dimensional semantic segmentation result, and then determine the text semantic feature vector corresponding to each pixel based on the predicted fine-grained semantic category corresponding to each pixel. The semantic-guided 3D reconstruction module is used to reconstruct the 3D Gaussian semantic point cloud of the endoscopic surgical area based on the already matched 2D semantic segmentation results and text semantic feature vectors. The visualization interaction module is used to display endoscopic surgical area images, two-dimensional semantic segmentation results, and three-dimensional Gaussian semantic point clouds in a unified visualization interaction interface; The intraoperative early warning module is used to calculate the spatial distance between surgical instruments and tissue structures in the endoscopic surgical area in real time based on the three-dimensional Gaussian semantic point cloud, and to conduct risk assessment in combination with a preset safety threshold.
[0011] Furthermore, the method by which the semantic segmentation module obtains the two-dimensional semantic segmentation result of the endoscopic surgical area image is as follows: The endoscopic surgical area image is input into the CAT-Seg semantic segmentation network to obtain each pixel. Corresponding semantic segmentation results ; Raw image features of endoscopic surgical area images extracted by the backbone network Channel averaging is performed to obtain the single-channel grayscale structure response. ; Extracting single-channel grayscale structural response using the Sobel operator Edge features ; Extracting single-channel grayscale structural response using the Laplacian operator Texture features ; Features of the original image Edge features Texture features and single-channel grayscale structure response The data is concatenated and structural features are obtained through convolutional mapping. ; A structural encoder consisting of convolutional layers is used to analyze structural features. Deep feature extraction is performed to obtain each pixel of the endoscopic surgical area image. Corresponding structural feature vectors ; Suppose that the total number of fine-grained semantic categories in the endoscopic surgical area image is K, and the total number of coarse-grained structural categories in the endoscopic surgical area image is J, where, ; The endoscopic surgical area image is input into the prototype acquisition module to obtain each pixel of the endoscopic surgical area image. The corresponding global structure prototype; Get each pixel separately Corresponding structural feature vectors The cosine similarity between the pixel and the global structural prototype is calculated, and after converting the cosine similarity to the range [0,1], each pixel is obtained. Prior probabilities of structures belonging to various coarse-grained structure categories ; Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, from pixels... Prior probabilities of structures belonging to various coarse-grained structure categories Determine the pixels Structure-guided priors belonging to each fine-grained semantic category ; Learnable gated multiplication is used to guide the structure prior. Semantic segmentation results compared to the output of the CAT-Seg semantic segmentation network The data is then fused to obtain the final two-dimensional semantic segmentation result. .
[0012] Furthermore, during the training phase, the prototype acquisition module acquires each pixel... The corresponding global structure prototype method is: Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, each pixel... The corresponding fine-grained semantic category determines the pixel set corresponding to each coarse-grained structural category. ; Get pixel set Pixels With the Structural feature centers of each coarse-grained structural category Cosine similarity between ; Get pixel set All pixels relative to the structural feature center Average cosine similarity ; Cosine similarity Not lower than the mean cosine similarity High confidence pixel composition structural feature center The corresponding set of high-confidence pixels; Utilizing each high-confidence pixel in the high-confidence pixel set Corresponding structural feature vectors Obtain the structural feature center Corresponding global structure prototype .
[0013] Furthermore, pixel set Pixels With the Structural feature centers of each coarse-grained structural category Cosine similarity between The method for obtaining it is as follows:
[0014]
[0015]
[0016] in, This indicates a feature normalization operation. This indicates that the value belonging to the first element is determined based on the mapping relationship. A set of pixels for each coarse-grained structural category. Represents a set of pixels The number of pixels in This indicates the pixel position in the endoscopic surgical area image. This represents the predicted fine-grained semantic category of pixel x. This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories.
[0017] Furthermore, during the training phase, structural feature centers Corresponding global structure prototype The method for obtaining it is as follows:
[0018]
[0019] Where t represents the number of training iterations. Indicates the first A set of high-confidence pixels corresponding to each coarse-grained structure category. Represents the set of high-confidence pixels The number of pixels in; Indicates high confidence pixels The structural feature vector corresponding to the t-th iteration, Indicates the first The initial structural prototypes of each coarse-grained structure category in the t-th iteration. Indicates the first The global structural prototype of each coarse-grained structural category in the (t-1)th iteration. This represents the prototype's updated momentum.
[0020] Further, determine the pixels Belongs to the Structure-guided priors for fine-grained semantic categories The method is as follows:
[0021] in, For the first Global structural prototype reliability for each coarse-grained structural category.
[0022] Furthermore, the final two-dimensional semantic segmentation result The method for obtaining it is as follows:
[0023] Where γ is the structural prior gating weight. These are learnable gating parameters.
[0024] Furthermore, the loss function used when training the semantic segmentation module as follows:
[0025] in, L seg For fine-grained semantic segmentation loss; L structure For coarse-grained structure classification loss; L protoThis represents the prototype consistency loss. l 1 and l 2 represents the loss weight; where prototype consistency loss is... The calculation method is as follows:
[0026] in, This represents the set of effectively labeled pixels in the endoscopic surgical area image during the t-th iteration. Represents pixels True fine-grained semantic categories, This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories. Determined pixels The global structural prototype of the corresponding coarse-grained structural category.
[0027] Furthermore, the method by which the semantic segmentation module obtains the text semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image is as follows: The medical-specific prompt template is input into the text encoder Etext to obtain the corresponding text semantic feature vector. The medical-specific prompt template consists of endoscopic surgical area scene prompts of different semantic categories, and the text semantic feature vector corresponding to any one semantic category of endoscopic surgical area scene prompt is:
[0028] in, For the m-th endoscopic surgical area scene prompt word corresponding to the k-th semantic category, Let be the fusion weight of the m-th endoscopic surgical area scene cue word, and Norm represent feature normalization. This is a text encoding operation.
[0029] Furthermore, the method by which the semantic indexing module determines the predicted fine-grained semantic category corresponding to each pixel in the two-dimensional semantic segmentation result based on the semantic index map generated from the two-dimensional semantic segmentation result is as follows:
[0030] in, Represents the number of pixels in the t-th frame of the endoscopic surgical area image. x Predicting fine-grained semantic categories, two-dimensional semantic segmentation results This represents the result of two-dimensional semantic segmentation.
[0031] Beneficial effects: 1. This invention provides a structure-prior-guided endoscopic surgical area scene understanding and safety early warning system. It constructs a complete technical link from binocular images, medical semantic segmentation, semantic prior generation, semantic-guided 3D Gaussian reconstruction to 3D semantic point cloud generation. It realizes the extension of robotic surgical scene from 2D semantic understanding to 3D semantic understanding, providing basic visual perception capabilities for intelligent surgical navigation, digital twin operating rooms, and autonomous surgical robots. It can achieve high-precision 3D reconstruction of surgical scene while introducing semantic features into the 3D reconstruction process, so that the reconstruction result simultaneously expresses the geometry, appearance, and medical semantic information of the surgical area, and maintains semantic continuity during tissue deformation, instrument movement, and changes in endoscopic perspective. Finally, it forms a 3D semantic scene model that can be used for surgical area semantic understanding and visualization interaction.
[0032] 2. This invention provides a structure-prior-guided endoscopic surgical area scene understanding and safety early warning system. Through outlier pixel filtering, dynamic updating of structural prototypes, prototype reliability calculation, and prototype consistency loss, a stable category structure prototype is obtained, and the guidance strength of the structure prior is adaptively adjusted according to the prototype reliability. A semantic region index and region semantic features are generated based on the two-dimensional semantic segmentation results, and the region semantic features are written into three-dimensional Gaussian points to construct a three-dimensional semantic point cloud that simultaneously contains geometric, color, and medical semantic information. Through semantic perception deformation tracking and semantic region consistency constraints, the system maintains a continuous and stable three-dimensional semantic representation of tissues when deformation, occlusion, and viewpoint changes occur, enabling surgical area structure query, target localization, spatial measurement, interactive surgical guidance, and visual early warning. Attached Figure Description
[0033] Figure 1 This invention provides a principle block diagram of a structure-prior-guided endoscopic surgical area scene understanding and safety early warning system. Detailed Implementation
[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0035] like Figure 1 As shown, a structure-prior-guided endoscopic surgical area scene understanding and safety early warning system includes a semantic segmentation module, a semantic indexing module, a semantic-guided 3D reconstruction module, a visualization interaction module, and an intraoperative early warning module. The semantic segmentation module is used to obtain the two-dimensional semantic segmentation result of the endoscopic surgical area image, and at the same time obtain the text semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image. The semantic indexing module is used to determine the predicted fine-grained semantic category corresponding to each pixel based on the semantic index map generated by the two-dimensional semantic segmentation result, and then determine the text semantic feature vector corresponding to each pixel based on the predicted fine-grained semantic category corresponding to each pixel. The semantic-guided 3D reconstruction module is used to reconstruct the 3D Gaussian semantic point cloud of the endoscopic surgical area based on the already matched 2D semantic segmentation results and text semantic feature vectors. The visualization interaction module is used to display endoscopic surgical area images, two-dimensional semantic segmentation results, and three-dimensional Gaussian semantic point clouds in a unified visualization interaction interface; The intraoperative early warning module is used to calculate the spatial distance between surgical instruments and tissue structures in the endoscopic surgical area in real time based on the three-dimensional Gaussian semantic point cloud, and to conduct risk assessment in combination with a preset safety threshold.
[0036] Furthermore, the method by which the semantic segmentation module obtains the text semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image is as follows: The medical-specific prompt template is input into the text encoder Etext to obtain the corresponding text semantic feature vector. The medical-specific prompt template consists of endoscopic surgical area scene prompts of different semantic categories, and the text semantic feature vector corresponding to any one semantic category of endoscopic surgical area scene prompt is:
[0037] in, For the m-th endoscopic surgical area scene prompt word corresponding to the k-th semantic category, Let be the fusion weight of the m-th endoscopic surgical area scene cue word, and Norm represent feature normalization. This is a text encoding operation.
[0038] It should be noted that traditional CAT-Seg uses general text prompts to construct category text features. These prompts are primarily designed for natural scenes and are insufficient to accurately describe the specialized semantics within medical images. This invention designs a medical-specific prompt template for endoscopic surgery scenarios, constructing multiple medical scenario prompt words P for each semantic category k. m (k). Prompt words include, but are not limited to: "Photos of {category name} in the scene" "Endoscopic surgical images of partial nephrectomy in pigs (category name)" Each cue word is input into the text encoder Etext to obtain the corresponding text features. The multi-cue word fusion of this invention can describe the same medical category from multiple perspectives such as imaging method, surgical type, visibility, and lighting conditions, reducing the semantic bias caused by a single cue word.
[0039] It should be noted that in the medical field, multiple cue words can be generated using manually designed fixed cue templates, or they can be generated using learnable cue words, language models, or combinations of multiple cue words; the features of multiple cue words can be fused using averaging, weighting, or attention methods.
[0040] Furthermore, the method by which the semantic segmentation module obtains the two-dimensional semantic segmentation result of the endoscopic surgical area image is as follows: S1: Input the endoscopic surgical area image into the CAT-Seg semantic segmentation network to obtain each pixel. Corresponding semantic segmentation results ; S2: Raw image features of the endoscopic surgical area extracted by the backbone network. Channel averaging is performed to obtain the single-channel grayscale structure response. :
[0041] S3: Extracting single-channel grayscale structural response using the Sobel operator. Edge features :
[0042] Among them, S x S y These represent the horizontal and vertical Sobel convolution kernels, respectively. The Sobel operator performs first-order gradient calculations to extract image edge structure information. It should be noted that the Sobel and Laplacian operators can be replaced by methods that can extract edge and texture information, such as Scharr, Canny, LoG, Gabor, wavelet transform, or learnable edge convolution. S4: Extracting single-channel grayscale structural response using the Laplacian operator Texture features :
[0043] Where L represents the Laplacian convolution kernel; the Laplacian operator calculates the second derivative, enhancing the model's ability to express subtle structural features; S5: Extract features from the original image Edge features Texture features and single-channel grayscale structure response The data is concatenated and structural features are obtained through convolutional mapping. :
[0044] S6: Employ a structural encoder composed of convolutional layers to analyze structural features. Deep feature extraction is performed to obtain each pixel of the endoscopic surgical area image. Corresponding structural feature vectors :
[0045] in, This represents a structured encoder consisting of convolutional layers; S7: Assume the total number of fine-grained semantic categories in the endoscopic surgical area image is K, and the total number of coarse-grained structural categories in the endoscopic surgical area image is J, where, ; S8: Input the endoscopic surgical area image into the prototype acquisition module to obtain each pixel of the endoscopic surgical area image. The corresponding global structure prototype; It should be noted that each coarse-grained structure category j corresponds to a global structure prototype P. j Different coarse-grained structural categories can form representative cluster centers in the structural space to represent the typical structural features of that category. During the training phase, the structural feature centers belonging to the j-th category in the current batch are calculated based on the true semantic labels. Subsequently, the cosine similarity between each structural feature and the category center is calculated, and outlier pixels with similarities lower than the average of that category are filtered out. In other words, the prototype acquisition module is a pre-trained network model. During the training phase, the prototype acquisition module acquires each pixel... The corresponding global structure prototype method is: Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, each pixel... The corresponding fine-grained semantic category determines the pixel set corresponding to each coarse-grained structural category. ; Get pixel set Pixels With the Structural feature centers of each coarse-grained structural category Cosine similarity between :
[0046]
[0047]
[0048] in, This indicates a feature normalization operation. This indicates that the value belonging to the first element is determined based on the mapping relationship. A set of pixels for each coarse-grained structural category. Represents a set of pixels The number of pixels in This indicates the pixel position in the endoscopic surgical area image. This represents the predicted fine-grained semantic category of pixel x. This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories. It should be noted that the mapping relationship between fine-grained categories and coarse-grained structural categories can be set manually or automatically generated through structural feature clustering, text similarity, or learnable mapping matrices. Get pixel set All pixels relative to the structural feature center Average cosine similarity :
[0049] Cosine similarity Not lower than the mean cosine similarity High confidence pixel composition structural feature center The corresponding set of high-confidence pixels:
[0050] Utilizing each high-confidence pixel in the high-confidence pixel set Corresponding structural feature vectors Obtain the structural feature center Corresponding global structure prototype :
[0051]
[0052] Where t represents the number of training iterations. Indicates the first A set of high-confidence pixels corresponding to each coarse-grained structure category. Represents the set of high-confidence pixels The number of pixels in; Indicates high confidence pixels The structural feature vector corresponding to the t-th iteration, Indicates the first The initial structural prototypes of each coarse-grained structure category in the t-th iteration. Indicates the first The global structural prototype of each coarse-grained structural category in the (t-1)th iteration. Indicates the prototype update momentum; S9: Obtain each pixel individually Corresponding structural feature vectors The cosine similarity between the pixel and the global structural prototype is calculated, and after converting the cosine similarity to the range [0,1], each pixel is obtained. Prior probabilities of structures belonging to various coarse-grained structure categories :
[0053] The larger the value, the larger the pixel. The more similar the structural features are to the typical structural features of the j-th coarse-grained structural category, the better; S10: Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, from pixels... Prior probabilities of structures belonging to various coarse-grained structure categories Determine the pixels Structure-guided priors belonging to each fine-grained semantic category ; Among them, determining pixels Belongs to the Structure-guided priors for fine-grained semantic categories The method is as follows:
[0054] in, For the first Global structural prototype reliability for each coarse-grained structural category; S11: Employing learnable gated multiplication to guide prior knowledge of the structure Semantic segmentation results compared to the output of the CAT-Seg semantic segmentation network The data is then fused to obtain the final two-dimensional semantic segmentation result. :
[0055] Where γ is the structural prior gating weight. These are learnable gating parameters.
[0056] Furthermore, the loss function used when training the semantic segmentation module as follows:
[0057] in, L seg For fine-grained semantic segmentation loss; L structure For coarse-grained structure classification loss; L proto This represents the prototype consistency loss. l 1 and l2 represents the loss weight; whereby, in order to bring structural features of the same coarse-grained category closer to their structural prototypes, this invention introduces a prototype consistency loss. The calculation method is as follows:
[0058] in, This represents the set of effectively labeled pixels in the endoscopic surgical area image during the t-th iteration. Represents pixels True fine-grained semantic categories, This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories. Determined pixels The global structural prototype of the corresponding coarse-grained structural category.
[0059] Prototype consistency loss is used to enhance the intra-class compactness of structural features, enabling structural prototypes to stably represent the typical geometric features of the corresponding category.
[0060] Therefore, in order to enhance the model's ability to express structural information such as edges and textures in medical images, this invention adds a structural prior branch to the original image-text cost aggregation structure of CAT-Seg, extracts coarse-grained structural features of tissues or instruments, and uses the structural information as a prior to guide fine-grained semantic segmentation.
[0061] This invention does not directly use structural features to replace semantic classification. Instead, it establishes a mapping q(k) between fine-grained semantic categories and coarse-grained structural categories, mapping easily confused but similar geometrical categories to the same coarse-grained category. This allows structural priors to serve only as spatial constraints for semantic classification, improving the ability to distinguish between similar-looking tissues and instrument parts.
[0062] Taking the EndoVis endoscopy dataset as an example, "Instrument Shaft", "Instrument Wrist" and "Instrument Clasper" can be uniformly mapped to the coarse-grained "Surgical Instrument" category, while other tissue or instrument categories are mapped to their corresponding coarse-grained structure categories.
[0063] Furthermore, for the input image at frame t, the semantic segmentation module can output the semantic segmentation result. Determine the pixel The semantic category to which it belongs, and constitutes the semantic index graph of the image in frame t. :
[0064] in, Represents the number of pixels in the t-th frame of the endoscopic surgical area image. x Predicting fine-grained semantic categories, two-dimensional semantic segmentation results This represents the result of two-dimensional semantic segmentation.
[0065] Based on this, the text encoder extracts the semantic feature vectors corresponding to each semantic category. Then, construct the semantic feature matrix. Medical semantic information used to characterize the anatomical structures or surgical instruments corresponding to this region:
[0066] in, This represents the semantic feature vector corresponding to the k-th fine-grained semantic category.
[0067] Based on the semantic index map By querying the semantic feature matrix T, we can obtain the semantic feature vector corresponding to pixel x in the t-th frame image. .
[0068] In the initialization phase of the 3D Gaussian model, an initial 3D Gaussian representation is established based on the color information, depth information, and camera parameters of the input image, and semantic feature attributes are set for each Gaussian point. The semantic features of the Gaussian points are initialized as zero vectors, and then further refined using a semantic index map. Supervised optimization is performed using the semantic feature matrix T as the ground truth. Each Gaussian point not only has spatial location, scale, and color parameters, but also carries semantic embedding information corresponding to the tissue structure or instrument category. The i-th Gaussian point is represented as:
[0069] Where i represents the Gaussian point number; Indicates the three-dimensional center position of the Gaussian point; Represents the rotation parameters of the Gaussian point; This represents the scale parameters of the Gaussian point in each direction; Indicates the opacity of a Gaussian point; The color parameter representing the Gaussian point or spherical harmonic color; This represents the semantic features carried by Gaussian points.
[0070] During training, semantic features are used as supervision information based on the projection relationship of Gaussian points under the current camera view. The semantic feature attributes of each Gaussian point are optimized through semantic feature reconstruction loss, so that the three-dimensional Gaussian model gradually forms a three-dimensional semantic representation consistent with the two-dimensional semantic segmentation result.
[0071] Since surgical tissues deform with instrument manipulation, respiration, and endoscopic perspective, a semantically aware deformation tracking network is used to predict displacement or feature residuals based on time t and current semantic features, thus maintaining semantic continuity of the same anatomical region during deformation:
[0072] in, This represents the semantic features of the i-th Gaussian point after the update at time t; Represents the basic semantic features before deformation update; This represents the semantic adjustment vector predicted by the semantic feature deformation network; Concat represents the feature concatenation operation.
[0073] Overall color reconstruction loss Deep loss semantic feature loss Spatiotemporal smoothing loss and regional consistency loss After optimization, the joint loss function of the 3D Gaussian model is expressed as:
[0074] Furthermore, this invention integrates endoscopic images, two-dimensional semantic segmentation results, and three-dimensional semantic point clouds. The reconstruction results are uniformly displayed on a visual interactive interface. This interface includes at least an endoscopic image display area, a 2D semantic segmentation overlay area, a 3D semantic point cloud display area, and an interactive control area. The 2D semantic segmentation results are overlaid on the endoscopic image as outlines, transparent blocks, or category identifiers; the 3D semantic point cloud is displayed differently according to tissue category, instrument category, or query status, allowing physicians to simultaneously observe the 2D appearance information, 2D semantic information, and 3D spatial structure of the surgical area.
[0075] The intraoperative early warning module constructs an anatomical relationship model based on semantic point clouds. The system calculates the spatial distance between surgical instruments and key tissue structures in real time and performs risk assessments based on preset safety thresholds. When the instrument tip approaches important blood vessels, kidney boundaries, or dangerous anatomical areas, the system automatically generates visual early warning information and highlights the risk area in a 3D scene. Simultaneously, surgeons can use semantic point clouds to query surgical area structures, locate targets, plan paths, and perform spatial measurements, thereby improving intraoperative safety and navigation efficiency.
[0076] Doctors can interact with the semantic point cloud through text queries, 2D image point selection, region selection, or 3D point cloud point selection. The system determines the corresponding semantic category or region number based on the query command and simultaneously displays the corresponding tissue in the 2D segmentation results and 3D semantic point cloud. Based on the 3D semantic region obtained from the query, it can also perform target localization, tissue boundary display, spatial distance measurement, path-assisted planning, and control of the visibility of surgical area structures.
[0077] In summary, this invention ultimately constructs a complete technical chain from binocular imaging, medical semantic segmentation, semantic prior generation, semantic-guided 3D Gaussian reconstruction to 3D semantic point cloud generation, realizing the extension of robotic surgical scenarios from 2D semantic understanding to 3D semantic understanding, and providing basic visual perception capabilities for intelligent surgical navigation, digital twin operating rooms, and autonomous surgical robots. Compared with existing technologies, this invention has the following advantages: 1. This invention can preserve the semantic information of different tissues and instruments in the three-dimensional reconstruction results, realizing the extension from two-dimensional image understanding to three-dimensional surgical area semantic understanding.
[0078] 2. By introducing multiple prompts from the medical field, this invention can describe the same category from multiple perspectives, such as endoscopic imaging method, surgical type, target visibility, and lighting conditions, thereby reducing the semantic discrepancy between general text prompts and medical images.
[0079] 3. This invention introduces structural branches, which include modules such as semantic segmentation, semantic indexing, and semantic-guided 3D reconstruction, so that the structural branches learn the coarse-grained structural features of instruments and tissues, and the semantic branches complete the fine-grained category judgment. This avoids the structural branches and semantic branches competing with each other as two classifiers, thus improving the stability of the segmentation results.
[0080] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.
Claims
1. A structure-prior-guided endoscopic surgical area scene understanding and safety early warning system, characterized in that, It includes a semantic segmentation module, a semantic indexing module, a semantic-guided 3D reconstruction module, a visualization and interaction module, and an intraoperative early warning module; The semantic segmentation module is used to obtain the two-dimensional semantic segmentation result of the endoscopic surgical area image, and at the same time obtain the text semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image. The semantic indexing module is used to determine the predicted fine-grained semantic category corresponding to each pixel based on the semantic index map generated by the two-dimensional semantic segmentation result, and then determine the text semantic feature vector corresponding to each pixel based on the predicted fine-grained semantic category corresponding to each pixel. The semantic-guided 3D reconstruction module is used to reconstruct the 3D Gaussian semantic point cloud of the endoscopic surgical area based on the already matched 2D semantic segmentation results and text semantic feature vectors. The visualization interaction module is used to display endoscopic surgical area images, two-dimensional semantic segmentation results, and three-dimensional Gaussian semantic point clouds in a unified visualization interaction interface; The intraoperative early warning module is used to calculate the spatial distance between surgical instruments and tissue structures in the endoscopic surgical area in real time based on the three-dimensional Gaussian semantic point cloud, and to conduct risk assessment in combination with a preset safety threshold.
2. The structural prior guidance-based endoscopic surgical area scene understanding and safety early warning system as described in claim 1, characterized in that, The method by which the semantic segmentation module obtains the two-dimensional semantic segmentation result of the endoscopic surgical area image is as follows: The endoscopic surgical area image is input into the CAT-Seg semantic segmentation network to obtain each pixel. Corresponding semantic segmentation results ; Raw image features of endoscopic surgical area images extracted by the backbone network Channel averaging is performed to obtain the single-channel grayscale structure response. ; Extracting single-channel grayscale structural response using the Sobel operator Edge features ; Extracting single-channel grayscale structural response using the Laplacian operator Texture features ; Features of the original image Edge features Texture features and single-channel grayscale structure response The data is concatenated and structural features are obtained through convolutional mapping. ; A structural encoder consisting of convolutional layers is used to analyze structural features. Deep feature extraction is performed to obtain each pixel of the endoscopic surgical area image. Corresponding structural feature vectors ; Suppose that the total number of fine-grained semantic categories in the endoscopic surgical area image is K, and the total number of coarse-grained structural categories in the endoscopic surgical area image is J, where, ; The endoscopic surgical area image is input into the prototype acquisition module to obtain each pixel of the endoscopic surgical area image. The corresponding global structure prototype; Get each pixel separately Corresponding structural feature vectors The cosine similarity between the pixel and the global structural prototype is calculated, and after converting the cosine similarity to the range [0,1], each pixel is obtained. Prior probabilities of structures belonging to various coarse-grained structure categories ; Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, from pixels... Prior probabilities of structures belonging to various coarse-grained structure categories Determine the pixel Structure-guided priors belonging to each fine-grained semantic category ; Learnable gated multiplication is used to guide the structure prior. Semantic segmentation results compared to the output of the CAT-Seg semantic segmentation network The data is then fused to obtain the final two-dimensional semantic segmentation result. .
3. The endoscopic surgical area scene understanding and safety early warning system guided by prior knowledge as described in claim 2, characterized in that, During the training phase, the prototype acquisition module acquires each pixel. The corresponding global structure prototype method is: Based on the mapping relationship between fine-grained semantic categories and coarse-grained structural categories, each pixel... The corresponding fine-grained semantic category determines the pixel set corresponding to each coarse-grained structural category. ; Get pixel set Pixels With the Structural feature centers of each coarse-grained structural category Cosine similarity between ; Get pixel set All pixels relative to the structural feature center Average cosine similarity ; Cosine similarity Not lower than the mean cosine similarity High confidence pixel composition structural feature center The corresponding set of high-confidence pixels; Utilizing each high-confidence pixel in the high-confidence pixel set Corresponding structural feature vectors Obtain the structural feature center Corresponding global structure prototype .
4. The structure-prior-guided endoscopic surgical area scene understanding and safety early warning system as described in claim 3, characterized in that, Pixel set Pixels With the Structural feature centers of each coarse-grained structural category Cosine similarity between The method for obtaining it is as follows: in, This indicates a feature normalization operation. This indicates that the value belonging to the first element is determined based on the mapping relationship. A set of pixels for each coarse-grained structural category. Represents a set of pixels The number of pixels in This indicates the pixel position in the endoscopic surgical area image. This represents the predicted fine-grained semantic category of pixel x. This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories.
5. The structural prior guidance-based endoscopic surgical area scene understanding and safety early warning system as described in claim 3, characterized in that, During the training phase, structural feature centers Corresponding global structure prototype The method for obtaining it is as follows: Where t represents the number of training iterations. Indicates the first A set of high-confidence pixels corresponding to each coarse-grained structure category. Represents the set of high-confidence pixels The number of pixels in; Indicates high confidence pixels The structural feature vector corresponding to the t-th iteration, Indicates the first The initial structural prototypes of each coarse-grained structure category in the t-th iteration. Indicates the first The global structural prototype of each coarse-grained structural category in the (t-1)th iteration. This represents the prototype's updated momentum.
6. The structure-prior-guided endoscopic surgical area scene understanding and safety early warning system as described in claim 2, characterized in that, Determine pixels Belongs to the Structure-guided priors for fine-grained semantic categories The method is as follows: in, For the first Global structural prototype reliability for each coarse-grained structural category.
7. The structural prior guidance-based endoscopic surgical area scene understanding and safety early warning system as described in claim 2, characterized in that, Final two-dimensional semantic segmentation result The method for obtaining it is as follows: Where γ is the structural prior gating weight. These are learnable gating parameters.
8. The structural prior guidance-based endoscopic surgical area scene understanding and safety early warning system as described in claim 1, characterized in that, The loss function used when training the semantic segmentation module as follows: in, L seg For fine-grained semantic segmentation loss; L structure For coarse-grained structure classification loss; L proto This represents the prototype consistency loss. λ 1 and λ 2 represents the loss weight; where prototype consistency loss is... The calculation method is as follows: in, This represents the set of effectively labeled pixels in the endoscopic surgical area image during the t-th iteration. Represents pixels True fine-grained semantic categories, This represents the mapping relationship between fine-grained semantic categories and coarse-grained structural categories. Determined pixels The global structural prototype of the corresponding coarse-grained structural category.
9. The structural prior guidance-based endoscopic surgical area scene understanding and safety early warning system as described in claim 1, characterized in that, The method by which the semantic segmentation module obtains the textual semantic feature vector of the medical-specific prompt template used to describe the endoscopic surgical area image is as follows: The medical-specific prompt template is input into the text encoder Etext to obtain the corresponding text semantic feature vector. The medical-specific prompt template consists of endoscopic surgical area scene prompts of different semantic categories, and the text semantic feature vector corresponding to any one semantic category of endoscopic surgical area scene prompt is: in, For the m-th endoscopic surgical area scene prompt word corresponding to the k-th semantic category, Let be the fusion weight of the m-th endoscopic surgical area scene cue word, and Norm represent feature normalization. This is a text encoding operation.
10. The structure-prior-guided endoscopic surgical area scene understanding and safety early warning system as described in claim 1, characterized in that, The method by which the semantic indexing module determines the predicted fine-grained semantic category corresponding to each pixel in the two-dimensional semantic segmentation result based on the semantic index map generated by the two-dimensional semantic segmentation result is as follows: in, Represents the number of pixels in the t-th frame of the endoscopic surgical area image. x Predicting fine-grained semantic categories, two-dimensional semantic segmentation results This represents the result of two-dimensional semantic segmentation.
