Semantic segmentation method and system for laparoscopic surgery internal and external sphincter gap image

Abstract

The invention relates to the technical field of medical image processing, and discloses a laparoscopic surgery internal and external sphincter gap image semantic segmentation method and system, and the method comprises the steps: obtaining the position of an endoscope light source and the geometric information of a tissue surface in a laparoscopic real-time video stream; establishing an illumination physical model based on the acquired information, and generating an illumination influence map based on the illumination physical model; during semantic segmentation of an internal and external sphincter gap image, dynamically adjusting a segmentation strategy according to an illumination influence map, and when a potential gap boundary is located in a shadow region, reducing brightness feature dependence and enhancing texture / shape feature extraction; and when in the highlight area, color gradient and structure information are analyzed. According to the method, the physical illumination model is introduced, so that the semantic segmentation process becomes informed from blind, common illumination interference in the laparoscopic surgery image can be intelligently dealt with, and the accuracy and robustness of internal and external sphincter gap segmentation are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and more specifically, to a semantic segmentation method and system for images of the intersphincteric space in laparoscopic surgery. Background Technology

[0002] In modern minimally invasive surgery, especially delicate procedures like radical resection of low rectal cancer, surgeons need to perform precise dissections at a very narrow level between the internal and external anal sphincters to preserve anal function as much as possible. This level is called the intersphincteric space. This operation demands a high level of experience and skill from the surgeon because the anatomy of this space is extremely delicate, surrounded by important nerves and blood vessels; even slight mistakes can lead to functional impairment or tumor residue. To assist surgeons in performing more precise operations, an image semantic segmentation technology based on laparoscopic surgical videos has emerged. This technology uses a pre-prepared computational program to analyze the images captured by the laparoscopic camera in real time and highlight the critical area of ​​the intersphincteric space with a specific color on the surgical monitor, providing the surgeon with a clear and intuitive navigation guide. Under ideal conditions, such as in teaching demonstrations or when dealing with cases with very clear anatomical structures, this system performs exceptionally well. It can reliably identify the boundaries of the gap, and the highlighted area perfectly matches the actual anatomical level. With this real-time visual assistance, doctors can confidently perform separation operations, resulting in a smooth surgical process that ensures complete tumor removal while effectively protecting the function of the sphincter muscles.

[0003] However, in real-world clinical surgical settings, the situation is far more complex than ideal. As the laparoscopic viewpoint is dynamically adjusted, the illumination angle of the surgical field from the endoscope's built-in light source also changes. In deep, narrow cavities, light is often difficult to distribute evenly, inevitably resulting in various shadows and highlights in the surgical field. As the endoscope moves, the shape, size, and position of these shadows continuously and dynamically change, sometimes falling precisely on the boundary of the intersphincteric space, partially or completely obscuring it. Simultaneously, highlights formed by light reflection on moist tissue surfaces also move, and these highlights may resemble certain features of the space or obscure the subtle texture of the space itself. While image segmentation programs encounter some lighting changes during the learning process, their ability to handle such continuous, drastic, and unpredictable dynamic changes in shadows and highlights is often insufficient. The program may misidentify shadow edges as space boundaries or incorrectly identify highlight areas as part of the space, leading to significant offsets or inaccurate shapes in the segmentation results. This segmentation instability caused by dynamic changes in illumination further exacerbates doctors' concerns about the reliability of visual aid systems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application discloses a semantic segmentation method and system for images of the internal and external sphincter spaces in laparoscopic surgery. The aim is to solve the technical problem that the dynamic changes in the endoscopic light source during laparoscopic surgery cause shadow and highlight areas in the surgical field, which in turn affect the accuracy and stability of semantic segmentation of images of the internal and external sphincter spaces.

[0005] The technical solution of this application is as follows: In a first aspect, this application discloses a semantic segmentation method for images of the intersphincteric space during laparoscopic surgery, comprising the following steps: Acquire real-time position information of the endoscope light source and real-time video stream of the laparoscopy, and obtain geometric information of the tissue surface from the real-time video stream of the laparoscopy. Based on the real-time position information of the endoscopic light source and the geometric information of the tissue surface, a light physics model is established, and a light influence map is generated based on the light physics model. The light influence map includes shadow areas and highlight areas. When performing semantic segmentation of the intersphincteric space image, the segmentation strategy for the space boundary is dynamically adjusted based on the illumination effect map, wherein: When potential gap boundaries are located in shadowed areas, the weighting of the dependence on image brightness features is reduced, enhancing the extraction of tissue texture and shape features; When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the highlight region edge.

[0006] Furthermore, based on the above, this application proposes an optimized scheme for generating a lighting impact map, wherein the steps of generating the lighting impact map based on a lighting physics model, the lighting impact map including shadow areas and highlight areas, include: Predicting shadow and highlight areas in the surgical field of view based on a light physics model; For shadowed areas, virtual rays are emitted from the position of the endoscope light source toward the tissue surface to perform occlusion detection and generate a shadow probability map. For the highlight region, calculate the probability of each pixel producing specular reflection under the current light source and viewing angle, and generate a highlight probability map; By overlaying the shadow probability map and the highlight probability map, a lighting effect map is formed.

[0007] Specifically, the steps for obtaining real-time position information of the endoscope light source include: The endoscope's posture and relative position change data are acquired by the measurement unit built into the laparoscope, and the real-time position information of the endoscope light source is obtained after fusion processing.

[0008] As a further improvement, the method of this application also includes: When high-reflectivity surgical instruments are introduced during surgery, the instrument pose information of the high-reflectivity surgical instruments is obtained in real time through a visual tracking algorithm; Based on the real-time position information of the endoscope light source, the geometric information of the tissue surface, and the instrument pose information, the abnormal bright areas caused by the reflection of high reflectivity surgical instruments are predicted, and a reflection risk map is generated. The reflection risk map is displayed visually overlaid on the surgical monitor, and operation guidance is provided to avoid abnormally highlighted areas.

[0009] Building upon this, this application further proposes steps to visually overlay a reflection risk map onto a surgical monitor and provide operational guidance to avoid abnormally highlighted areas, specifically including: On the surgical monitor, a reflection risk map is overlaid as a heat map in the non-core area of ​​the real-time video stream; Based on the reflection risk value of each pixel in the reflection risk map, the visual attributes of the corresponding area in the heat map are dynamically adjusted. These visual attributes include color, transparency, or brightness.

[0010] Furthermore, the steps for dynamically adjusting the visual attributes of the corresponding areas in the heatmap based on the reflection risk values ​​of each pixel in the reflection risk map include: When the reflection risk value of each pixel exceeds the preset reflection risk threshold, it is determined to be a high-risk pixel; For the heatmap area corresponding to high-risk pixels, adjust the color to the preset warning color, and / or increase its brightness, and / or decrease its transparency to enhance the display; When high-risk pixels cluster together to form a continuous high-risk area, a directional arrow icon and a text prompt related to the high-risk area are displayed at the center of the high-risk area.

[0011] Furthermore, the steps for acquiring the instrument pose information of high-reflectivity surgical instruments in real time using visual tracking algorithms include: Acquire real-time video stream of laparoscopy, and identify the locally obscured areas of high-reflectivity surgical instruments based on the real-time video stream of laparoscopy. Based on the real-time position information of the endoscope light source and the three-dimensional models and position information of other known surgical instruments in the surgical field, a local three-dimensional scene is constructed. Based on the visual features of the visible area of ​​the unobstructed portion of a high-reflectivity surgical instrument in a local 3D scene, combined with the known geometry of the high-reflectivity surgical instrument, the complete 3D position and orientation of the high-reflectivity surgical instrument are estimated, thus obtaining the instrument pose information.

[0012] Specifically, the geometric information of the tissue surface includes relative depth information, surface normals, and curvature; The steps for acquiring geometric information of tissue surfaces from a real-time laparoscopic video stream include: The relative depth information of each pixel is estimated based on the real-time video stream of laparoscopy. A depth map is generated based on the relative depth information, and the surface normal and curvature are calculated based on the depth map.

[0013] Furthermore, the method of this application also includes: Surgical instruments with integrated mechanical sensors acquire real-time mechanical response data of the tissue surface, and compare the mechanical response data with preset mechanical characteristics of normal tissue to identify abnormal areas on the tissue surface. When abnormal areas are identified, the geometric information of the tissue surface is dynamically corrected.

[0014] Secondly, this application also discloses a semantic segmentation system for images of the internal and external sphincter spaces in laparoscopic surgery, used to perform the aforementioned semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery. The system includes: The information acquisition module is used to acquire the real-time position information of the endoscope light source and the real-time video stream of the laparoscopy, and to acquire the geometric information of the tissue surface from the real-time video stream of the laparoscopy. The illumination effect prediction module is used to establish an illumination physics model based on the real-time location information of the endoscope light source and the geometric information of the tissue surface, and generate an illumination effect map based on the illumination physics model. The illumination effect map includes shadow areas and highlight areas. The segmentation strategy adjustment module is used to dynamically adjust the gap boundary segmentation strategy based on the illumination influence map when performing semantic segmentation of the internal and external sphincter gap images. When potential gap boundaries are located in shadowed areas, the weighting of the dependence on image brightness features is reduced, enhancing the extraction of tissue texture and shape features; When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the highlight region edge.

[0015] This system enables all the functions of the above methods, providing an integrated solution. Through modular design, it ensures the stability and scalability of the system, and provides an efficient and accurate semantic segmentation tool for images of the internal and external sphincter spaces for laparoscopic surgery.

[0016] In summary, this application discloses a method and system for semantic segmentation of images of the internal and external sphincter spaces during laparoscopic surgery. The method acquires real-time position information of the endoscopic light source and tissue surface geometry information from the real-time laparoscopic video stream to establish an accurate lighting physical model, and generates an illumination influence map containing shadow and highlight regions. During semantic segmentation of the internal and external sphincter spaces, this method dynamically adjusts the segmentation strategy for the space boundaries based on the illumination influence map. Specifically, when the potential space boundary is located in a shadow region, the method effectively avoids interference from shadows on the segmentation results by reducing the dependence weight on image brightness features and enhancing the extraction of tissue texture and shape features. When the potential space boundary is located in a highlight region, the method accurately identifies and processes the highlight region by analyzing the color gradient and structural information of the highlight region's edge, avoiding missegmentation. This method effectively solves the problem in existing technologies where the dynamic changes in the endoscopic light source lead to shadow and highlight regions in the surgical field, thus affecting the accuracy and stability of semantic segmentation of images of the internal and external sphincter spaces. By intelligently sensing lighting conditions and adjusting adaptive segmentation strategies, this application significantly improves the robustness and accuracy of semantic segmentation of images of the internal and external sphincter spaces under complex lighting conditions, providing surgeons with more reliable visual assistance and thus helping to improve the precision and safety of surgery. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery, provided as an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of a semantic segmentation system for images of the internal and external sphincter spaces in laparoscopic surgery, provided in an embodiment of this application.

[0019] Labeling Explanation: 101, Information Acquisition Module; 102, Illumination Impact Prediction Module; 103, Segmentation Strategy Adjustment Module. Detailed Implementation

[0020] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] In delicate minimally invasive surgical procedures such as radical resection of low rectal cancer, surgeons need to precisely dissect the narrow space between the internal and external sphincter muscles to preserve the patient's anal function. However, in the actual surgical environment, due to the dynamic adjustment of the laparoscopic view and the change in the illumination angle of the endoscopic light source, dynamically changing shadow and highlight areas often appear in the surgical field. These illumination changes may obscure or distort the boundary of the intersphincter space, causing existing image semantic segmentation techniques to deviate or become inaccurate in identifying the space, thus affecting the precision and safety of the surgery.

[0023] Firstly, please see Figure 1 This application proposes a semantic segmentation method for images of the intersphincteric space during laparoscopic surgery, including: Acquire real-time position information of the endoscope light source and real-time video stream of the laparoscopy, and obtain geometric information of the tissue surface from the real-time video stream of the laparoscopy. Based on the real-time position information of the endoscopic light source and the geometric information of the tissue surface, a light physics model is established, and a light influence map is generated based on the light physics model. The light influence map includes shadow areas and highlight areas. When performing semantic segmentation of the intersphincteric space image, the segmentation strategy for the space boundary is dynamically adjusted based on the illumination effect map, wherein: When potential gap boundaries are located in shadowed areas, the weighting of the dependence on image brightness features is reduced, enhancing the extraction of tissue texture and shape features; When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the highlight region edge.

[0024] This application establishes a lighting physics model and generates a lighting effect map by acquiring real-time positional information of the endoscopic light source and geometric information of the tissue surface, thereby enabling the identification of shadow and highlight areas in the surgical field of view. Based on this, the application can dynamically adjust the gap boundary segmentation strategy according to the lighting effect map. For example, when the potential gap boundary is located in a shadow area, the dependence weight on image brightness features is reduced, while the extraction of tissue texture and shape features is enhanced; when the potential gap boundary is located in a highlight area, the color gradient and structural information of the highlight area edge are analyzed. Therefore, this application can effectively cope with complex and variable lighting conditions during surgery, improve the accuracy and stability of semantic segmentation of internal and external sphincter gap images, and provide surgeons with more reliable visual assistance.

[0025] The implementation environment of this application typically includes a laparoscopic surgical system equipped with an endoscope, a light source, a video processor, and a display. Additionally, an image processing unit (e.g., a high-performance computer or embedded processing device) is required to execute the image semantic segmentation method described in this application.

[0026] The semantic segmentation method for the intersphincteric space in laparoscopic surgery proposed in this application aims to address the shortcomings of traditional methods in terms of segmentation accuracy and stability under complex lighting conditions. Its working principle can be summarized as follows: by sensing real-time changes in the lighting environment of the surgical environment and intelligently adjusting the image segmentation strategy accordingly, the accuracy of identifying the intersphincteric space is improved.

[0027] Specifically, this method first acquires the real-time position information of the endoscopic light source and the real-time video stream of the laparoscopy through an information acquisition module. The real-time position information of the endoscopic light source is crucial for subsequently establishing an accurate illumination physics model, while the real-time video stream of the laparoscopy provides raw image data of the surgical field. Next, geometric information of the tissue surface is extracted from the real-time video stream of the laparoscopy, such as the relative depth, surface normal, and curvature of the tissue obtained through depth estimation techniques. This geometric information, together with the position of the light source, constitutes the basis for light propagation and reflection.

[0028] Subsequently, the illumination effect prediction module establishes an illumination physics model based on the real-time position information of the endoscopic light source and the geometric information of the tissue surface. This model simulates the propagation of light in the surgical field and its interaction with the tissue surface, thereby predicting which areas will form shadows and which areas will produce highlights. Based on this model, an illumination effect map is generated, which clearly identifies the shadowed and highlighted areas in the surgical field.

[0029] During semantic segmentation of the internal and external sphincter spaces, the segmentation strategy adjustment module dynamically adjusts the segmentation strategy for the space boundaries based on a real-time generated illumination effect map. When the illumination effect map indicates that a potential space boundary is located in a shadowed area, the system reduces the weight of brightness-based segmentation because shadows reduce image brightness, making brightness-based segmentation unreliable. Instead, the system enhances the extraction of tissue texture and shape features, as these features are relatively stable in shadowed areas and provide more reliable boundary information. This can be achieved, for example, by enhancing the sensitivity of edge detection algorithms or using illumination-insensitive texture descriptors.

[0030] Conversely, when the lighting effect map indicates that the potential gap boundary is located in a highlight area, the highlight may cause local image overexposure, obscuring true tissue details. In this case, the system adjusts its strategy, focusing on analyzing the color gradation and structural information at the edges of the highlight area. This is because the edges of highlight areas are often where the curvature of the tissue surface changes or where different tissues meet. By analyzing the color transitions and structural features of these edges, the true boundary of the gap can be determined more accurately, avoiding misidentifying the highlight itself as a gap.

[0031] By dynamically adjusting the segmentation strategy, this application can effectively cope with the complex and ever-changing lighting conditions during surgery, avoiding segmentation errors caused by shadows and highlights. The various modules work closely together, from environmental perception to strategy adjustment, forming a closed-loop intelligent segmentation system that significantly improves the accuracy and stability of semantic segmentation of images of the internal and external sphincter spaces in laparoscopic surgery.

[0032] The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery proposed in this application demonstrates significant advantages and innovation in dealing with complex lighting environments compared to existing technologies.

[0033] Traditional image semantic segmentation methods can effectively identify the intersphincteric space under ideal lighting conditions. However, in real clinical surgical environments, due to the dynamic changes in the endoscopic light source and the complexity of deep cavities, dynamically changing shadow and highlight areas inevitably appear in the surgical field of view. These lighting changes often cause traditional methods to misjudge shadow edges as gap boundaries or incorrectly identify highlight areas as part of the gap, resulting in significant deviations or inaccurate shapes in the segmentation results. This seriously affects the surgeon's trust in the reliability of visual assistance systems.

[0034] The core innovation of this application lies in the introduction of an illumination effect map and the dynamic adjustment of the gap boundary segmentation strategy based on this map. Specifically, this application establishes an illumination physical model by acquiring the real-time position information of the endoscopic light source and the geometric information of the tissue surface, and generates an illumination effect map accordingly, thereby accurately identifying shadow and highlight areas in the surgical field of view. This step is generally lacking in existing technologies, and it provides crucial contextual information for subsequent intelligent segmentation strategy adjustments.

[0035] Regarding segmentation strategy adjustments, this application employs differentiated processing strategies for shadow and highlight regions based on the illumination influence map. When potential gap boundaries are located in shadow regions, this application reduces the weighting of image brightness features and instead enhances the extraction of tissue texture and shape features. This contrasts sharply with traditional methods that tend to fail when brightness information in shadow regions is unreliable. By focusing on more stable texture and shape features, this application effectively avoids the interference of shadows on the segmentation results.

[0036] When potential gap boundaries are located in highlight areas, this application focuses on analyzing the color gradation and structural information at the edges of the highlight areas. This allows the application to identify the true boundaries between highlight areas and surrounding tissues by seeing through the masking of real tissue details by highlights, avoiding misjudging highlights themselves as gaps. In contrast, existing methods often suffer from oversegmentation or undersegmentation in highlight areas.

[0037] In summary, this application achieves accurate perception of complex lighting environments by establishing a lighting physics model and generating a lighting influence map. Based on this, the application can dynamically adjust the gap boundary segmentation strategy according to the lighting influence map, employing different feature extraction and analysis methods for shadow and highlight areas, thereby significantly improving the accuracy and stability of semantic segmentation of images of the internal and external sphincter spaces in laparoscopic surgery. This intelligent lighting-adaptive segmentation method effectively solves the challenges faced by existing technologies in real surgical environments, providing surgeons with more reliable and accurate visual assistance, and has significant clinical application value and technological advancement significance.

[0038] Specifically, the steps described above for generating a lighting impact map based on a lighting physics model, which includes shadow and highlight areas, can be further refined into the following process.

[0039] The steps involved in generating a lighting impact map based on a lighting physics model, which includes shadow and highlight areas, are as follows: Predicting shadow and highlight areas in the surgical field of view based on a light physics model; For shadowed areas, virtual rays are emitted from the position of the endoscope light source toward the tissue surface to perform occlusion detection and generate a shadow probability map. For the highlight region, calculate the probability of each pixel producing specular reflection under the current light source and viewing angle, and generate a highlight probability map; By overlaying the shadow probability map and the highlight probability map, a lighting effect map is formed.

[0040] Predicting shadow and highlight areas in the surgical field of view involves using an established lighting physics model, combined with real-time positional information of the endoscopic light source and geometric information of the tissue surface, to preliminarily determine which areas may be obscured by shadows or produce highlight reflections. Specifically, shadow areas refer to areas where light is blocked by objects and cannot be directly illuminated, while highlight areas refer to areas on the tissue surface that produce specular reflections of the light source at specific angles.

[0041] Furthermore, for the generation of shadow areas, virtual rays are emitted from the endoscope light source towards the tissue surface by simulating the light propagation path. It is then detected whether these virtual rays are blocked by other tissues or instruments. If the light is blocked, the area is identified as a shadow area, and a corresponding shadow probability map is generated based on the degree of occlusion. The values ​​in the shadow probability map represent the probability that a pixel is in shadow or the intensity of the shadow.

[0042] Furthermore, generating highlight regions requires calculating the probability of specular reflection for each pixel under the current endoscopic light source and viewing angle. This typically involves a BRDF (Bidirectional Reflectance Distribution Function) model, which quantifies the probability of each pixel generating a highlight by analyzing the optical properties of the tissue surface, the incident angle of the light source, and the viewing exit angle, thus generating a highlight probability map. The values ​​in the highlight probability map can represent the probability or intensity of the highlight generated by that pixel.

[0043] Finally, the generated shadow probability map and highlight probability map are overlaid. The overlay operation can be a simple addition of pixel values, a weighted average, or other fusion algorithms. The goal is to combine the effects of shadows and highlights to form a comprehensive lighting effect map. This map clearly indicates which areas in the surgical field of view are shadows, which areas are highlights, and the intensity of these lighting effects.

[0044] This application's solution refines the generation process of the illumination impact map into four specific steps: prediction, shadow probability map generation, highlight probability map generation, and overlay fusion. This enables precise modeling of illumination conditions in complex laparoscopic surgical environments. First, based on an illumination physics model, preliminary predictions of shadow and highlight regions are made, providing direction for subsequent refined processing. Second, shadow probability maps are generated by detecting occlusion through virtual ray projection, accurately capturing shadows formed by tissue or instrument occlusion and avoiding errors from judging shadows solely based on brightness. Simultaneously, highlight probability maps are generated by calculating the probability of specular reflection, accurately identifying highlighted areas caused by tissue surface reflection, which is crucial for distinguishing real tissue features from illumination artifacts. Finally, the shadow and highlight probability maps are overlaid to form a comprehensive illumination impact map. This map comprehensively reflects the illumination distribution within the surgical field of view, providing a reliable basis for subsequent dynamic adjustment of gap boundary segmentation strategies.

[0045] Specifically, the steps for obtaining the real-time position information of the endoscope light source include: acquiring the endoscope's posture and relative position change data through the measurement unit built into the laparoscope, and obtaining the real-time position information of the endoscope light source after fusion processing.

[0046] The measurement unit built into the laparoscope can be understood as a sensor assembly integrated inside the laparoscope body or its proximal portion. Specifically, this measurement unit may include, but is not limited to, an inertial measurement unit (IMU), a miniature optical tracker, an electromagnetic sensor, or a combination thereof. Its purpose is to capture the laparoscope's motion in the surgical space in real time and with precision. The endoscope's attitude refers to its orientation and tilt in three-dimensional space, typically represented by pitch, roll, and yaw angles. Relative position change data refers to the displacement information of the laparoscope relative to a reference point or initial position. In practical applications, these data can be directly measured by the built-in measurement unit or calculated using internal algorithms.

[0047] This application's solution utilizes the measurement unit built into the laparoscopy to directly and in real-time acquire data on the endoscope's attitude and relative position changes. This raw data is then input into a fusion processing module, where it is processed using advanced sensor fusion algorithms such as Kalman filtering, extended Kalman filtering, or particle filtering. Through fusion processing, sensor noise, drift errors, and other interference factors can be effectively corrected, thereby generating high-precision, robust real-time position information of the endoscope's light source. This direct and accurate acquisition method provides reliable foundational data for subsequently establishing an accurate lighting physics model, ensuring the accuracy of the generated lighting influence map.

[0048] In some embodiments, this application proposes a semantic segmentation method for images of the intersphincteric space during laparoscopic surgery. This method acquires real-time positional information of the endoscopic light source and geometric information of the tissue surface to establish a lighting physical model and generate a lighting influence map. It then dynamically adjusts the segmentation strategy for the gap boundary to address the impact of shadow and highlight areas on segmentation accuracy. However, in actual laparoscopic surgery, in addition to shadows and highlights caused by direct illumination from the endoscopic light source, various highly reflective surgical instruments are often introduced into the surgical field of view. Under the illumination of the endoscopic light source, these instruments produce strong specular reflections, forming abnormally bright areas. These areas may severely obscure critical tissue structures, especially the intersphincteric space, thus significantly interfering with the accuracy of image semantic segmentation. If the above problems are not addressed, even if the lighting influence caused by the endoscopic light source is compensated for, the abnormally bright areas caused by instrument reflections may still lead to segmentation algorithm failure or erroneous results, thereby affecting the accuracy and safety of the surgery.

[0049] In response, this application further proposes a semantic segmentation method for images of the intersphincteric space during laparoscopic surgery, which also includes: When high-reflectivity surgical instruments are introduced during surgery, the instrument pose information of the high-reflectivity surgical instruments is obtained in real time through a visual tracking algorithm; Based on the real-time position information of the endoscope light source, the geometric information of the tissue surface, and the instrument pose information, the abnormal bright areas caused by the reflection of high reflectivity surgical instruments are predicted, and a reflection risk map is generated. The reflection risk map is displayed visually overlaid on the surgical monitor, and operation guidance is provided to avoid abnormally highlighted areas.

[0050] Specifically, high-reflectivity surgical instruments refer to instruments whose surfaces produce significant specular reflections under endoscopic light source illumination. The introduction of these instruments significantly alters the light distribution within the surgical field of view. Visual tracking algorithms can be understood as a computer vision technology aimed at identifying and tracking the position and orientation of surgical instruments in three-dimensional space in real time. This algorithm continuously monitors instrument movement and outputs its real-time pose information in the endoscopic coordinate system, including position (X, Y, Z coordinates) and orientation (rotation angle). Instrument pose information refers to the three-dimensional position and orientation of the high-reflectivity surgical instrument within the surgical field of view. Abnormally bright areas refer to image areas formed by the reflection of the endoscopic light source from high-reflectivity surgical instruments, exhibiting brightness far exceeding that of surrounding tissue areas. These areas typically manifest as strong glare or specular reflection points, potentially completely obscuring underlying tissue details. A reflection risk map can be understood as a two-dimensional image or data matrix, where the value of each pixel represents the probability or intensity of the point being affected by instrument reflection. This map quantifies the risk of abnormally bright areas within the surgical field of view due to instrument reflection. Visual overlay displays refer to the real-time presentation of a reflection risk map, such as a heatmap, outline, or warning markers, on the live laparoscopic video stream on the surgical monitor. Operational guidance involves using visual or auditory cues to instruct the surgeon to adjust the position or angle of surgical instruments to avoid or mitigate abnormally highlighted areas.

[0051] The solution presented in this application addresses the issue of abnormal brightness caused by highly reflective surgical instruments because it further considers instrument reflection as a key factor, building upon existing illumination physics models. First, a visual tracking algorithm acquires the instrument pose information of highly reflective surgical instruments in real time, enabling the system to accurately determine the location and direction of the reflection source. Second, based on the real-time position information of the endoscope light source, the geometric information of the tissue surface, and the instrument pose information, the system can establish a more comprehensive illumination physics model. This model not only considers direct illumination and tissue reflection but also predicts abnormally bright areas caused by instrument reflection. It is precisely because of the accurate prediction of these abnormally bright areas that the system can generate a reflection risk map, thereby quantifying the reflection risk of each region. Finally, by visually overlaying the reflection risk map onto the surgical monitor and providing operational guidance, surgeons can intuitively understand potential reflection risk areas and adjust instrument operations in a timely manner, thus proactively avoiding the generation of abnormally bright areas or mitigating their impact. This proactive avoidance mechanism effectively prevents abnormally bright areas from interfering with the semantic segmentation of the internal and external sphincter spaces, ensuring the robustness and accuracy of the segmentation algorithm under complex illumination conditions.

[0052] In some embodiments described above, this application proposes that when high-reflectivity surgical instruments are introduced during surgery, abnormally bright areas are predicted and a reflection risk map is generated. This reflection risk map is then overlaid on the surgical monitor in a visual manner, and operational guidance is provided to avoid abnormally bright areas. However, in practical applications, if the display method of the reflection risk map is not intuitive enough or may obscure the core surgical area, it may affect the surgeon's acquisition of critical information and the smoothness of operation. To address this, this application further proposes a step of overlaying the reflection risk map on the surgical monitor in a visual manner and providing operational guidance to avoid abnormally bright areas. This step includes: on the surgical monitor, dynamically adjusting the visual attributes of the corresponding area in the heatmap based on the reflection risk value of each pixel in the reflection risk map. These visual attributes include color, transparency, or brightness.

[0053] Specifically, the steps described above, which involve visually overlaying the reflection risk map onto the surgical monitor and providing operational guidance to avoid abnormally highlighted areas, include: On the surgical monitor, a reflection risk map is overlaid as a heat map in the non-core area of ​​the real-time video stream; Based on the reflection risk value of each pixel in the reflection risk map, the visual attributes of the corresponding area in the heat map are dynamically adjusted. The visual attributes include color, transparency, or brightness.

[0054] The overlay display of the reflection risk map as a heatmap involves using color gradients or intensity changes to visually represent the reflection risk level of different areas. For example, a higher risk value may result in a brighter or darker color, and vice versa. This display method allows surgeons to quickly identify potentially abnormally bright areas and their severity. "Non-core areas" can be understood as the peripheral portions of the surgical field that do not contain critical anatomical structures or the main surgical area, such as the top, bottom, or side areas of the monitor screen. Their purpose is to provide risk indication while minimizing interference with the main surgical field.

[0055] This application's solution effectively solves the visual interference problem that may exist in traditional display methods by overlaying a reflection risk map as a heatmap onto non-core areas of a real-time video stream. The gradient colors or intensity changes of the heatmap can intuitively reflect the level of reflection risk, allowing surgeons to quickly perceive the risk distribution without complex numerical judgments. Simultaneously, placing it in a non-core area ensures the clarity of the core surgical field of view and avoids obstruction of critical operating areas, thereby ensuring the smoothness and safety of the surgery. Furthermore, by dynamically adjusting the visual attributes of the heatmap area based on the reflection risk value of each pixel, this application can provide graded and refined risk warnings. When the risk is low, the warning information is not too abrupt, avoiding unnecessary interference; while when the risk is high, by enhancing color, brightness, or reducing transparency, it can quickly attract the surgeon's attention, prompting timely adjustments to the operation and effectively avoiding abnormally bright areas, thus significantly improving the safety and efficiency of the surgery.

[0056] In some embodiments described above, this application proposes displaying a reflection risk map overlaid on a surgical monitor and dynamically adjusting the visual attributes of the heatmap region based on the reflection risk value. However, in actual surgical environments, simply adjusting color, transparency, or brightness may not be sufficient to quickly and effectively alert surgeons to potentially high-risk reflection areas in emergency situations, especially when the surgical field is complex or the surgeon's attention is distracted. This could lead to untimely or unclear identification and avoidance of high-risk areas.

[0057] In this regard, this application further proposes the following steps for dynamically adjusting the visual attributes of the corresponding area in the heat map based on the reflection risk value of each pixel in the reflection risk map: When the reflection risk value of each pixel exceeds the preset reflection risk threshold, it is determined to be a high-risk pixel; For the heatmap area corresponding to high-risk pixels, adjust the color to the preset warning color, and / or increase its brightness, and / or decrease its transparency to enhance the display; When high-risk pixels cluster together to form a continuous high-risk area, a directional arrow icon and text prompts related to the high-risk area are displayed at the center of the high-risk area.

[0058] Specifically, the preset reflection risk threshold is a value set by the system based on clinical experience or preset safety standards to distinguish between normal reflection risk and high-risk reflections that require special attention. When the reflection risk value of a pixel exceeds this threshold, the pixel is identified by the system as a high-risk pixel, indicating that there is a high risk of abnormal bright reflection in that area.

[0059] Specifically, for the heatmap areas corresponding to pixels identified as high-risk, their visual attributes will be adjusted to enhance the warning effect. For example, the color can be adjusted to a preset warning color, such as red or orange, to visually indicate danger. Simultaneously, the brightness of the area can be increased or its transparency decreased to make it more conspicuous on the surgical monitor, thereby ensuring that surgeons can quickly notice these potentially dangerous areas.

[0060] This application's solution introduces a reflection risk threshold, transforming reflection risk from a continuous numerical value into discrete risk levels. This allows the system to clearly identify high-risk areas requiring special attention. Enhanced visual attributes such as warning colors, increased brightness, or reduced transparency for high-risk pixels enable surgeons to quickly detect these critical areas within the complex surgical field. Furthermore, when high-risk pixels cluster to form a continuous area, the solution provides directional arrow icons and text prompts. This not only warns of the risk but also offers surgeons intuitive operational guidance, helping them quickly assess the situation and take evasive action. This effectively solves the problem that simply adjusting general visual attributes is insufficient to provide clear and timely operational guidance.

[0061] In some of the embodiments described above in this application, a method is proposed to obtain the instrument pose information of high reflectivity surgical instruments in real time using a visual tracking algorithm. However, in actual surgical environments, high reflectivity surgical instruments may be partially obscured due to tissue occlusion, occlusion by other instruments, or limitations in the surgical field of view, which poses a challenge to accurately obtaining their complete three-dimensional position and pose.

[0062] In this regard, this application further proposes that the steps for obtaining the instrument pose information of high-reflectivity surgical instruments in real time through the aforementioned visual tracking algorithm include: Acquire real-time video stream of laparoscopy, and identify the locally obscured areas of high-reflectivity surgical instruments based on the real-time video stream of laparoscopy. Based on the real-time position information of the endoscope light source and the three-dimensional models and position information of other known surgical instruments in the surgical field, a local three-dimensional scene is constructed. Based on the visual features of the visible area of ​​the unobstructed portion of a high-reflectivity surgical instrument in a local 3D scene, combined with the known geometry of the high-reflectivity surgical instrument, the complete 3D position and orientation of the high-reflectivity surgical instrument are estimated, thus obtaining the instrument pose information.

[0063] Specifically, acquiring a real-time laparoscopic video stream refers to continuously capturing a sequence of real-time images of the surgical area using a laparoscopic system. Based on this real-time video stream, identifying locally occluded areas of highly reflective surgical instruments can be understood as using image processing and computer vision techniques to analyze pixel information in video frames, detecting and locating the visible and invisible portions of the highly reflective surgical instruments in the image. For example, the contours of the instruments can be identified through background subtraction, object detection algorithms (such as deep learning-based models), or motion analysis, and it can be determined whether there are discontinuous or missing parts, thereby identifying the occluded areas. The purpose is to provide crucial information about instrument visibility for subsequent 3D scene construction and pose estimation.

[0064] The construction of a local 3D scene based on the real-time position information of the endoscopic light source and the known 3D models and position information of other surgical instruments in the surgical field involves fusing information from different sensors and preset data to establish a 3D model reflecting the local spatial structure of the current surgical area. The real-time position information of the endoscopic light source can be acquired and fused by the measurement unit built into the laparoscope. The known 3D models and position information of other surgical instruments can be obtained preoperatively through scanning or modeling and updated in real-time during the operation by other tracking systems. This information is used to locate the light source, other instruments, and their relative relationships to the tissue surface in 3D space, thus providing an accurate reference system for estimating the pose of high-reflectivity surgical instruments.

[0065] This application's solution first identifies the partially occluded areas of high-reflectivity surgical instruments, thus clarifying the effective information range available for pose estimation. Subsequently, by integrating real-time position information of the endoscope light source and the 3D models and position information of other known surgical instruments, a precise local 3D scene is constructed, providing the necessary spatial context for instrument pose estimation. Finally, based on matching the visual features of the unoccluded portion of the instrument with its known geometry, even under partial occlusion, the complete 3D position and orientation of the instrument can be accurately estimated using a model-driven approach. This method effectively utilizes multi-source information and prior knowledge, overcoming the limitations of relying solely on visual information for instrument tracking in complex surgical environments.

[0066] Specifically, in the semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery described above, the geometric information of the tissue surface can be specifically defined and obtained.

[0067] The geometric information of the aforementioned tissue surface includes relative depth information, surface normals, and curvature; The steps described above for obtaining the geometric information of the tissue surface from the real-time video stream of a laparoscopy include: The relative depth information of each pixel is estimated based on the real-time video stream of the laparoscopy, a depth map is generated based on the relative depth information, and the surface normal and curvature are calculated based on the depth map.

[0068] Among these, the geometric information of the tissue surface is one of the key inputs for constructing an accurate lighting physics model. Specifically, relative depth information refers to the distance information of each pixel in the surgical field of view relative to a certain reference plane, reflecting the three-dimensional structure of the tissue surface. Surface normals are the normal vectors of each point on the tissue surface, indicating the orientation of the surface at that point, and are crucial for understanding how light interacts with the surface. Curvature describes the degree of bending of the tissue surface and can reflect the local morphological characteristics of the tissue.

[0069] The reason why the proposed solution can perform more accurate semantic segmentation of images of the internal and external sphincter spaces is because it defines and acquires the geometric information of the tissue surface in detail. By clarifying that the geometric information of the tissue surface includes relative depth information, surface normals, and curvature, and by detailing the specific steps for acquiring this information from real-time laparoscopic video streams, the establishment of the lighting physical model has a more solid and accurate three-dimensional geometric foundation. Relative depth information provides the spatial positional relationship of the tissue surface, surface normals reveal the directionality of light incidence and reflection, and curvature reflects the local changes in surface morphology. It is precisely because of these detailed geometric features that the lighting physical model can more accurately simulate the propagation, reflection, and occlusion effects of light on complex tissue surfaces, thereby generating a more refined and accurate lighting effect map.

[0070] In some embodiments described above in this application, the geometric information of the tissue surface is primarily acquired through real-time video streaming from a laparoscopy. However, relying solely on visual information to acquire the geometric information of the tissue surface may have limitations. For example, some pathological tissues (such as tumors or inflammatory areas) may appear visually indistinguishable from normal tissues, but their internal structure and mechanical properties have changed, making it impossible for purely visual methods to accurately capture their true geometric shape. This inaccurate geometric information may affect the subsequent establishment of lighting physics models and the generation of lighting effect maps, thereby reducing the effectiveness of gap boundary segmentation strategies and ultimately affecting the accuracy of semantic segmentation.

[0071] In response, this application further proposes an optimization scheme, the method of which also includes: Surgical instruments with integrated mechanical sensors acquire real-time mechanical response data of the tissue surface, and compare the mechanical response data with preset mechanical characteristics of normal tissue to identify abnormal areas on the tissue surface. When an abnormal area is identified, the geometric information of the tissue surface is dynamically corrected.

[0072] Specifically, surgical instruments integrating mechanical sensors refer to surgical tools equipped with miniature pressure sensors, tactile sensors, or force feedback sensors. These instruments can measure and transmit the tissue's mechanical properties, such as hardness, elasticity, and viscosity, in real time upon contact with the tissue surface. Mechanical response data can be understood as the physical quantities measured by these sensors, such as pressure values, degree of deformation, or resistance. The purpose is to obtain the true mechanical properties of the tissue under physical contact, which is crucial for identifying tissue abnormalities that are difficult to detect with the naked eye. Comparing the mechanical response data with preset normal tissue mechanical characteristics involves comparing the real-time acquired mechanical data with a pre-established mechanical parameter model of healthy tissue (e.g., normal intestinal wall, normal sphincter tissue). These preset normal tissue mechanical characteristics can be obtained through extensive clinical data, in vitro experiments, or finite element analysis. By comparing, areas with significantly different mechanical properties from normal tissue can be identified, thus determining abnormal areas on the tissue surface. When an abnormal area is identified, the geometric information of the tissue surface is dynamically corrected, which means correcting the geometric information previously obtained from the video stream based on the mechanical properties of the identified abnormal area. For example, if a region is identified as having abnormal hardness by a mechanical sensor, even if its visual geometry appears normal, the system will adjust its depth, curvature, or normal based on the mechanical data to more accurately reflect its true physical state.

[0073] The solution proposed in this application overcomes the aforementioned limitations by incorporating tissue biomechanical response data as the basis for geometric information correction. Traditionally, the geometric information of tissue surfaces is primarily acquired from laparoscopic video streams using visual methods. However, this method may not accurately reflect the true physical geometry of tissues when faced with internal structural changes or pathological abnormalities. For example, a tissue with internal nodules may appear visually identical to normal tissue, but its mechanical properties will be significantly altered. Because surgical instruments with integrated mechanical sensors can perceive the tissue's biomechanical response under physical contact in real time and compare it with preset normal tissue biomechanical characteristics, the system can identify abnormal areas that are difficult to detect visually alone. Once these abnormal areas are identified, the system can dynamically correct the previously acquired visually-based geometric information of the tissue surface, thereby compensating for the shortcomings of purely visual methods in acquiring geometric information and ensuring the accuracy of subsequent adjustments to the illumination physics model and segmentation strategy.

[0074] In some embodiments described above in this application, a semantic segmentation method for images of the internal and external sphincter spaces during laparoscopic surgery is proposed. This method dynamically adjusts the segmentation strategy of the space boundaries to cope with complex lighting conditions. However, in order to effectively apply the method to actual surgical scenarios, a hardware or software system capable of efficiently and accurately executing the method is needed to support the coordinated operation of its various functional modules, ensuring stable and reliable image processing in surgical environments with high real-time requirements.

[0075] Secondly, see Figure 2 This application proposes a semantic segmentation system for images of the internal and external sphincter spaces in laparoscopic surgery, used to perform the aforementioned semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery. The system includes: The information acquisition module 101 is used to acquire the real-time position information of the endoscope light source and the real-time video stream of the laparoscopy, and to acquire the geometric information of the tissue surface from the real-time video stream of the laparoscopy. The illumination effect prediction module 102 is used to establish an illumination physical model based on the real-time position information of the endoscope light source and the geometric information of the tissue surface, and generate an illumination effect map based on the illumination physical model. The illumination effect map includes shadow areas and highlight areas. The segmentation strategy adjustment module 103 is used to dynamically adjust the gap boundary segmentation strategy based on the illumination influence map when performing semantic segmentation of the internal and external sphincter gap image, wherein: When potential gap boundaries are located in shadowed areas, the weighting of the dependence on image brightness features is reduced, enhancing the extraction of tissue texture and shape features; When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the highlight region edge.

[0076] The proposed solution decouples the semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery into three core modules: information acquisition, illumination effect prediction, and segmentation strategy adjustment, forming a collaborative system. The information acquisition module 101, acting as the system's perception layer, captures dynamic data of the surgical environment in real time, providing accurate and comprehensive input for subsequent processing. The illumination effect prediction module 102, acting as the system's intelligent analysis layer, uses a physical model to accurately model complex illumination, generating a guiding illumination effect map, thus overcoming the limitation of traditional methods where segmentation performance degrades under varying illumination. The segmentation strategy adjustment module 103, acting as the system's decision-making execution layer, dynamically optimizes the segmentation algorithm based on the illumination effect map, ensuring high-precision segmentation in both shadow and highlight areas. It is precisely this modular design and collaborative working mechanism that enables the system to robustly and in real-time cope with the complex and changing illumination conditions in laparoscopic surgery, significantly improving the accuracy and reliability of semantic segmentation of images of the internal and external sphincter spaces.

[0077] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A semantic segmentation method for images of the intersphincteric space in laparoscopic surgery, characterized in that, include: Acquire real-time position information of the endoscope light source and real-time video stream of the laparoscopy, and obtain geometric information of the tissue surface from the real-time video stream of the laparoscopy. Based on the real-time position information of the endoscope light source and the geometric information of the tissue surface, an illumination physical model is established, and an illumination influence map is generated based on the illumination physical model. The illumination influence map includes shadow areas and highlight areas. When performing semantic segmentation of the internal and external sphincter gap image, the gap boundary segmentation strategy is dynamically adjusted based on the illumination influence map, wherein: When the potential gap boundary is located in the shadow region, the dependence weight on image brightness features is reduced, and the extraction of tissue texture and shape features is enhanced. When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the edge of the highlight region.

2. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 1, characterized in that, The step of generating a lighting effect map based on the lighting physics model, wherein the lighting effect map includes shadow areas and highlight areas, includes: Based on the aforementioned light physics model, predict the shadow and highlight areas in the surgical field of view; For the shadowed area, virtual light rays are emitted from the position of the endoscope light source toward the tissue surface to perform light projection detection occlusion and generate a shadow probability map; For the highlighted region, the probability of each pixel producing specular reflection under the current light source and viewing angle is calculated to generate a highlight probability map; The shadow probability map and the highlight probability map are superimposed to form the illumination impact map.

3. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 1, characterized in that, The steps for obtaining real-time position information of the endoscope light source include: The endoscope's posture and relative position change data are acquired by the measurement unit built into the laparoscope, and the real-time position information of the endoscope light source is obtained after fusion processing.

4. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 1, characterized in that, The method further includes: When high-reflectivity surgical instruments are introduced during surgery, the instrument pose information of the high-reflectivity surgical instruments is obtained in real time through a visual tracking algorithm; Based on the real-time position information of the endoscope light source, the geometric information of the tissue surface, and the instrument pose information, the abnormal bright areas caused by the reflection of the high reflectivity surgical instruments are predicted, and a reflection risk map is generated. The reflection risk map is displayed visually overlaid on the surgical monitor, and operation guidance is provided to avoid the abnormally highlighted areas.

5. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 4, characterized in that, The step of visually overlaying the reflection risk map onto the surgical monitor and providing operational guidance to avoid the abnormally highlighted areas includes: On the surgical monitor, the reflection risk map is overlaid as a heat map in the non-core area of ​​the real-time video stream; Based on the reflection risk value of each pixel in the reflection risk map, the visual attributes of the corresponding area in the heat map are dynamically adjusted, including color, transparency, or brightness.

6. A semantic segmentation method for images of the intersphincteric space in laparoscopic surgery according to claim 5, characterized in that, The step of dynamically adjusting the visual attributes of the corresponding area in the heat map based on the reflection risk value of each pixel in the reflection risk map includes: When the reflection risk value of each pixel exceeds the preset reflection risk threshold, it is determined to be a high-risk pixel; For the heatmap area corresponding to the high-risk pixel, adjust the color to a preset warning color, and / or increase its brightness, and / or decrease its transparency to enhance the display; When high-risk pixels cluster together to form a continuous high-risk area, a directional arrow icon and a text prompt related to the high-risk area are displayed at the center of the high-risk area.

7. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 4, characterized in that, The step of acquiring the instrument pose information of the high-reflectivity surgical instrument in real time through a visual tracking algorithm includes: Acquire a real-time video stream of a laparoscopy, and based on the real-time video stream of the laparoscopy, identify the locally obscured areas of the high-reflectivity surgical instruments; Based on the real-time position information of the endoscope light source and the three-dimensional models and position information of other known surgical instruments in the surgical field of view, a local three-dimensional scene is constructed; Based on the visual features of the visible area of ​​the unobstructed portion of the high-reflectivity surgical instrument in the local 3D scene, and combined with the known geometry of the high-reflectivity surgical instrument, the complete 3D position and orientation of the high-reflectivity surgical instrument are estimated to obtain the instrument pose information.

8. The semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery according to claim 1, characterized in that, The geometric information of the tissue surface includes relative depth information, surface normals, and curvature; The step of acquiring the geometric information of the tissue surface from the laparoscopic real-time video stream includes: The relative depth information of each pixel is estimated based on the real-time video stream of the laparoscopy, a depth map is generated based on the relative depth information, and the surface normal and curvature are calculated based on the depth map.

9. A semantic segmentation method for images of the intersphincteric space in laparoscopic surgery according to claim 8, characterized in that, The method further includes: Surgical instruments with integrated mechanical sensors acquire mechanical response data of the tissue surface in real time, and compare the mechanical response data with preset mechanical characteristics of normal tissue to identify abnormal areas of the tissue surface. When the abnormal area is identified, the geometric information of the tissue surface is dynamically corrected.

10. A semantic segmentation system for images of the internal and external sphincter spaces in laparoscopic surgery, used to perform the semantic segmentation method for images of the internal and external sphincter spaces in laparoscopic surgery as described in any one of claims 1 to 9, characterized in that, The system includes: The information acquisition module is used to acquire the real-time position information of the endoscope light source and the real-time video stream of the laparoscopy, and to acquire the geometric information of the tissue surface from the real-time video stream of the laparoscopy. The illumination effect prediction module is used to establish an illumination physical model based on the real-time position information of the endoscope light source and the geometric information of the tissue surface, and to generate an illumination effect map based on the illumination physical model. The illumination effect map includes shadow areas and highlight areas. The segmentation strategy adjustment module is used to dynamically adjust the gap boundary segmentation strategy based on the illumination influence map when performing semantic segmentation of the internal and external sphincter gap image, wherein: When the potential gap boundary is located in the shadow region, the dependence weight on image brightness features is reduced, and the extraction of tissue texture and shape features is enhanced. When the potential gap boundary is located in the highlight region, analyze the color gradient and structural information of the edge of the highlight region.

Images