A method, apparatus and system for processing endoscope video stream

Abstract

The present disclosure relates to the technical field of medical image processing, in particular to a method, device and system for processing endoscope video stream, the method comprising: inputting video frames collected in an endoscope imaging process into a segmentation network of an encoding-decoding structure, performing adaptive weight distribution on intermediate feature maps along channel dimension and spatial dimension through a double attention feature reweighting mechanism embedded in an encoding-decoding path of the segmentation network to generate a semantic segmentation result; obtaining a tubular structure semantic mask carrying a type identifier in parallel from the semantic segmentation result, the type identifier being used to distinguish a first type of tubular structure and a second type of tubular structure; and generating a stereoscopic display signal with parallax based on the semantic mask. The above scheme realizes synchronous segmentation and accurate distinction of arteries and veins in endoscope images, solves the core pain point of lack of spatial depth information of traditional 2D laparoscopes, and significantly reduces the dependence on the surgical experience of doctors.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, specifically to a method, apparatus, and system for processing endoscopic video streams. Background Technology

[0002] Laparoscopic pancreaticoduodenectomy (LPD) is one of the most challenging surgeries in the field of hepatopancreatobiliary (HPB) surgery. In this procedure, surgeons must precisely dissect and protect complex vascular structures such as the gastroduodenal artery (GDA) and the gastrocolic trunk (GCT, also known as Henle's trunk) within a limited laparoscopic field of vision to avoid accidental damage that could lead to serious complications.

[0003] Existing research indicates that over 50% of postoperative complications after laparoscopic surgery stem from human error, with nearly 19% related to misidentification of anatomical structures. In current clinical practice, surgeons primarily rely on static three-dimensional models constructed from preoperative imaging data (such as CT and MRI) or intraoperative ultrasound for vascular localization. However, these methods have the following technical limitations: First, preoperative three-dimensional reconstruction and intraoperative ultrasound examination increase the economic burden on patients, and preoperative images cannot reflect the displacement of blood vessel positions caused by tissue traction and changes in body position during the operation. Second, laparoscopic surgery lacks tactile feedback, and traditional monitors can only provide two-dimensional planar images. It is difficult for doctors to obtain accurate spatial depth information in real time during surgery, and the judgment of anatomical structures is highly dependent on surgical experience. Third, existing deep learning technologies have been attempted to be applied to the recognition of anatomical structures during laparoscopy. However, the above studies have mostly focused on structures such as the colorectum, ureter, gallbladder, and surgical instruments, while research on semantic segmentation of abdominal vessels is still in its early stages. Summary of the Invention

[0004] To address the problems in the related technologies, this disclosure provides a method, apparatus, and system for processing endoscopic video streams.

[0005] In a first aspect, embodiments of this disclosure provide a method for processing an endoscopic video stream, including: Video frames acquired during endoscopic imaging are input into a segmentation network with an encoder-decoder structure. Through a dual attention feature reweighting mechanism embedded in the encoder-decoder path of the segmentation network, adaptive weight allocation is performed on the intermediate feature maps along the channel dimension and spatial dimension to generate semantic segmentation results. A tubular structure semantic mask carrying a type identifier is obtained in parallel from the semantic segmentation result. The type identifier is used to distinguish between the first type of tubular structure and the second type of tubular structure. A stereoscopic display signal with parallax is generated based on the semantic mask.

[0006] Optionally, the dual attention feature reweighting mechanism includes cascaded channel attention feature reweighting and spatial attention feature reweighting. The channel attention feature reweighting performs adaptive weight allocation on the intermediate feature map along the channel dimension, and the spatial attention feature reweighting performs adaptive weight allocation on the feature map along the spatial dimension after the channel attention feature reweighting.

[0007] Optionally, the channel attention feature reweighting includes: Adaptive average pooling and adaptive max pooling operations are performed on the input feature map to obtain the first channel descriptor and the second channel descriptor. The first channel descriptor and the second channel descriptor are respectively input into a shared multilayer perceptron. The shared multilayer perceptron includes a dimensionality reduction layer and a dimensionality increase layer. The dimensionality reduction layer compresses the channel dimension to a preset ratio of the original dimension, and the dimensionality increase layer restores the channel dimension to the original dimension. The two sets of results output by the shared multilayer perceptron are added element by element and then processed by an activation function to generate channel attention weights; The channel attention weights are multiplied channel by channel by the input feature map to obtain a channel-reweighted feature map; The spatial attention feature reweighting includes: Perform channel-level average pooling and channel-level max pooling operations along the channel dimension on the channel reweighted feature map to obtain a first spatial descriptor and a second spatial descriptor; The first spatial descriptor and the second spatial descriptor are concatenated along the channel dimension to generate a spatial feature descriptor; The spatial feature descriptor is subjected to a convolution operation, and after being processed by an activation function, a spatial attention weight map is generated. The spatial attention weight map is multiplied by the input feature map at each spatial location to obtain a double attention reweighted feature map.

[0008] Optionally, the first type of tubular structure is an artery, and the second type of tubular structure is a vein. The segmentation network outputs probability maps corresponding to each type in parallel through a multi-channel output layer. Each probability map indicates the probability of the existence of the corresponding type of tubular structure at each pixel position in the video frame.

[0009] Optionally, the segmentation network is trained on a labeled training dataset containing multiple frames of endoscopic sample images, each sample image labeled with segmentation classification information of tubular structures, the segmentation classification information including: Tubular structures containing only arterial vessels are designated as Category I; Tubular structures containing only veins are classified as Category II. Tubular structures that contain both arteries and veins are labeled as Category III. The segmentation network performs multi-task learning during training based on sample images labeled with the first, second, and third categories, so as to simultaneously have the ability to distinguish between arterial and venous blood vessels.

[0010] Optionally, generating a stereoscopic display signal with parallax based on the semantic mask includes: The semantic mask is overlaid onto the video frame to generate an enhanced video frame; A left-eye view and a right-eye view are generated based on the enhanced video frame, and there is a parallax between the left-eye view and the right-eye view.

[0011] Optionally, the left eye view and the right eye view are encoded into a single frame output image in a side-by-side format, wherein the left half of the single frame output image carries the left eye view and the right half carries the right eye view; The single-frame output image is transmitted to a polarized stereoscopic display terminal, which converts the left half and the right half into left-hand circularly polarized light signals and right-hand circularly polarized light signals respectively through a surface polarization layer. The operator wears polarized glasses with corresponding polarizers for the left and right eyes, so that the left eye receives the left-hand circularly polarized light signal and the right eye receives the right-hand circularly polarized light signal. Based on the principle of binocular parallax, a three-dimensional surgical field with a sense of depth is synthesized in the visual cortex.

[0012] Optionally, before inputting the video frame into the segmentation network, the data augmentation process is further performed on the video frame, the data augmentation process including at least one of photometric distortion, color space enhancement, channel flipping, mosaic enhancement, and hybrid enhancement.

[0013] Secondly, embodiments of this disclosure provide an apparatus for processing endoscopic video streams, comprising: The encoding-decoding processing unit is configured to perform semantic segmentation on video frames acquired during endoscopic imaging. A dual attention feature reweighting module is embedded in the encoding-decoding path of the encoding-decoding processing unit. The dual attention feature reweighting module performs adaptive weight allocation on the intermediate feature map along the channel dimension and spatial dimension. A multi-type segmentation output unit is configured to output a tubular structure semantic mask carrying a type identifier in parallel, wherein the type identifier distinguishes at least a first type of tubular structure and a second type of tubular structure. A stereoscopic display signal generation unit is configured to generate a stereoscopic display signal with parallax based on the semantic mask.

[0014] Thirdly, embodiments of this disclosure provide a real-time navigation system for endoscopic surgery, comprising: The endoscopic image acquisition unit is configured to acquire video frames of the surgical field of view; The apparatus for processing endoscopic video streams as described in the second aspect; The stereoscopic display terminal is configured to receive the stereoscopic display signal output by the processing device and present the operator with a stereoscopic surgical field of view superimposed with the results of tubular structure segmentation.

[0015] The technical solutions provided in this disclosure may have the following beneficial effects: This disclosure achieves simultaneous segmentation and accurate differentiation of arteries and veins in endoscopic images by using a segmentation network with an encoder-decoder structure to output semantic masks for multiple types of tubular structures in parallel. Experimental results show that in LPD surgical vessel segmentation tasks, the scheme of this disclosure achieves a Dice coefficient of 0.917 and an average crossover ratio (mIoU) of 0.848 for joint segmentation of arteries and veins on a test set containing 798 test images.

[0016] This embodiment of the present disclosure embeds a dual attention feature reweighting mechanism (including channel attention and spatial attention) in each stage of the decoder, enabling the network to adaptively filter background noise such as connective tissue and fat, accurately capture blood vessel edges in low-contrast scenes, and simulate various interference environments in the abdominal cavity through diverse data augmentation strategies, which greatly enhances the stability and recognition accuracy of the model under complex conditions such as occlusion, lighting changes, and anatomical variations.

[0017] The segmentation network model in this embodiment runs at a speed of 30 frames per second, with an end-to-end processing latency of less than 50 milliseconds, which is far below the seamless latency threshold of human visual perception (100 milliseconds), truly meeting the clinical needs of real-time intraoperative navigation.

[0018] This embodiment combines AI semantic segmentation results with polarized light imaging technology. Through side-by-side (SBS) format encoding and polarized stereoscopic display terminal output, surgeons can obtain a stereoscopic surgical field with a real sense of depth by wearing polarized glasses. This solves the core pain point of traditional 2D laparoscopy lacking spatial depth information and significantly reduces the reliance on the surgeon's experience.

[0019] This embodiment directly extracts vascular information from the endoscopic video stream in real time and generates a three-dimensional navigation signal, eliminating the need for expensive auxiliary examinations such as preoperative three-dimensional reconstruction or intraoperative ultrasound, thus effectively reducing the perioperative economic burden on patients.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments, taken in conjunction with the accompanying drawings.

[0022] Figure 1 This diagram illustrates the principle of a method for processing an endoscopic video stream according to an embodiment of the present disclosure.

[0023] Figure 2 A flowchart illustrating a method for processing an endoscopic video stream according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A schematic diagram of the overall architecture of a segmentation network according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A structural block diagram of a dual attention feature reweighting mechanism according to an embodiment of the present disclosure is shown.

[0026] Figure 5 A flowchart illustrating a method for generating stereoscopic display signals according to an embodiment of the present disclosure is shown.

[0027] Figure 6 A structural block diagram of an apparatus for processing an endoscopic video stream according to an embodiment of the present disclosure is shown.

[0028] Figure 7 A structural block diagram of a real-time navigation system for endoscopic surgery according to an embodiment of the present disclosure is shown.

[0029] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0030] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown. Detailed Implementation

[0031] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement them. Furthermore, for clarity, portions unrelated to the description of exemplary embodiments have been omitted from the drawings.

[0032] In this disclosure, it should be understood that terms such as “comprising” or “having” are intended to indicate the presence of features, figures, steps, behaviors, components, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility of the presence or addition of one or more other features, figures, steps, behaviors, components, parts or combinations thereof.

[0033] It should also be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] The method for processing endoscopic video streams provided in this disclosure can be applied to the real-time identification and three-dimensional navigation of key vascular structures in laparoscopic pancreaticoduodenectomy, and can also be extended to other surgical scenarios that require distinguishing different types of tubular structures (such as arteries, veins, ureters, bile ducts, etc.) in the endoscopic field of vision. Figure 1 This diagram illustrates the principle of a method for processing an endoscopic video stream according to an embodiment of the present disclosure. Figure 1 As shown, the endoscopic imaging system acquires endoscopic video streams of the surgical field of view, which are then transmitted via the methods provided in this embodiment of the present disclosure. Figure 6 After the processing device performs real-time semantic segmentation and generates stereo display signals, it presents the surgeon with a stereo surgical field of view superimposed with the blood vessel segmentation results through a polarized stereo display terminal.

[0035] Figure 2 A flowchart illustrating a method for processing an endoscopic video stream according to an embodiment of the present disclosure is shown. Figure 2 As shown, the method includes the following steps S101 to S103.

[0036] In step S101, the video frames acquired during the endoscopic imaging process are input into the encoding-decoding structure of the segmentation network. Through the dual attention feature reweighting mechanism embedded in the encoding-decoding path of the segmentation network, adaptive weight allocation is performed on the intermediate feature maps along the channel dimension and spatial dimension to generate semantic segmentation results.

[0037] According to embodiments of this disclosure, the segmentation network is built on an improved PlainConvUNet architecture, which employs a classic encoder-decoder structure. The encoder progressively downsamples the input video frames, extracting multi-scale feature maps ranging from low-level texture details to high-level semantic information. The decoder progressively upsamples the multi-scale feature maps, gradually restoring spatial resolution. At each level of the decoder, skip connections are used to concatenate the feature maps of the corresponding encoder level and the current decoder level along the channel dimension, thereby fusing global contextual information with local spatial details.

[0038] According to embodiments of this disclosure, the dual attention feature reweighting mechanism is embedded in each stage of the decoder. Specifically, after concatenation and convolution processing at each stage of the decoder, channel attention feature reweighting and spatial attention feature reweighting are applied sequentially. The channel attention feature reweighting performs adaptive weight allocation along the channel dimension on the intermediate feature map to enhance the discriminative feature channels for classification and suppress irrelevant channels; the spatial attention feature reweighting performs adaptive weight allocation along the spatial dimension on the feature map after the channel attention feature reweighting to focus on key spatial regions in the image (such as the location of blood vessels) and filter background noise.

[0039] Figure 3 A schematic diagram of the overall architecture of a segmentation network according to an embodiment of this disclosure is shown. Figure 3 As shown, the left side represents the encoder path, the right side represents the decoder path, and the bottom represents the bottleneck layer. The dashed arrows spanning the left and right indicate skip connections. Each upsampling stage of the decoder includes a Convolutional Block Attention Module (CBAM), which is the specific implementation module of the dual attention feature reweighting mechanism.

[0040] In step S102, a tubular structure semantic mask carrying a type identifier is obtained in parallel from the semantic segmentation result. The type identifier is used to distinguish between the first type of tubular structure and the second type of tubular structure.

[0041] According to embodiments of this disclosure, the semantic segmentation result is generated through a multi-channel output layer of the segmentation network. The multi-channel output layer employs a 1×1 convolutional layer, mapping the feature map finally output by the decoder to a multi-channel probability map. For example, in a vascular recognition scenario during LPD surgery, the first type of tubular structure represents arterial vessels, and the second type represents venous vessels. Accordingly, the output layer contains two channels (corresponding to arteries and veins respectively), plus one channel corresponding to the background region. By performing Softmax normalization on the probability values ​​of each channel at each pixel location, a probability distribution for each pixel belonging to an artery, vein, or background is obtained. The multi-type segmentation output unit extracts the arterial vessel semantic mask and the venous vessel semantic mask in parallel from the probability distribution.

[0042] In step S103, a stereoscopic display signal with parallax is generated based on the semantic mask.

[0043] According to embodiments of this disclosure, the semantic segmentation results (i.e., semantic masks for arteries and veins) are superimposed onto the original video frame as a colored semi-transparent mask to generate an enhanced video frame. Specifically, a first color (e.g., red) can be assigned to arteries, and a second color (e.g., blue) can be assigned to veins to help the surgeon quickly distinguish between different types of blood vessels. Subsequently, a stereoscopic display signal is generated based on the enhanced video frame. The stereoscopic display signal includes a left-eye view and a right-eye view, with parallax between the left-eye view and the right-eye view, to present a depth-sensory stereoscopic surgical field on a polarized stereoscopic display terminal.

[0044] Figure 4 A structural block diagram of a dual attention feature reweighting mechanism according to an embodiment of this disclosure is shown. Figure 4 As shown, the dual attention feature reweighting mechanism includes cascaded channel attention feature reweighting (such as...). Figure 4 (as shown in the upper part) and spatial attention feature reweighting (such as...) Figure 4 (See the lower half).

[0045] According to embodiments of this disclosure, the channel attention feature reweighting includes the following steps: First, the input feature map (in For the number of channels, For feature map height, Adaptive average pooling and adaptive max pooling operations are performed on the feature map width to obtain two channel descriptors. and Among them, adaptive average pooling is used to extract global statistical information of features and preserve the overall distribution trend of features; adaptive max pooling is used to capture high-response significant signals in features (such as edges, corners, etc.).

[0046] Then, the first channel descriptor and the second channel descriptor The inputs are then processed by a shared multilayer perceptron. This shared multilayer perceptron comprises a dimensionality reduction layer and a dimensionality increase layer. The dimensionality reduction layer contains a fully connected layer that reduces the channel dimension from... Compress to ,in For the preset dimensionality reduction ratio (exemplary, The value is set to 16 to facilitate interaction between channels while reducing computational load; the dimensionality-upgrading layer contains another fully connected layer that increases the channel dimension from... Restore to The two fully connected layers are connected via a ReLU activation function. The shared multilayer perceptron shares parameters between the two branches.

[0047] Next, the two sets of results output by the shared multilayer perceptron (from the average pooling branch and the max pooling branch, respectively) are fused element-wise and then processed by the Sigmoid activation function to generate channel attention weights. :

[0048] in, This represents the Sigmoid activation function. This represents the mapping function for a shared multilayer perceptron.

[0049] Finally, the channel attention weights With the input feature map Multiplying each channel sequentially yields a channel-reweighted feature map. :

[0050] in, This indicates a channel-by-channel multiplication operation. Through this operation, the network can enhance the feature channels that are discriminative for the current segmentation task, while suppressing redundant and noisy channels.

[0051] According to embodiments of this disclosure, the spatial attention feature reweighting includes the following steps: First, the channel reweighted feature map Perform channel-level average pooling and channel-level max pooling operations along the channel dimension to obtain two spatial descriptors. and In this process, average pooling takes the average value at each spatial location along the channel dimension, reflecting the average response at that location across all channels; max pooling takes the maximum value at each spatial location along the channel dimension, reflecting the most significant response at that location across all channels.

[0052] Then, the first space descriptor and the second space descriptor Spatial feature descriptors are generated by splicing along the channel dimension. This stitching operation allows subsequent convolutional layers to simultaneously utilize both average and maximum response information to locate key spatial regions.

[0053] Next, the spatial feature descriptor The convolution operation is performed, and the spatial attention weight map is generated after processing with the Sigmoid activation function. :

[0054] in, Indicates the kernel size as Convolution operation, This represents the Sigmoid activation function.

[0055] Finally, the spatial attention weight map Compared with the original input feature map Multiplying by spatial location yields a refined feature map, namely the double attention reweighted feature map. :

[0056] in, This represents the spatial position-wise multiplication operation (i.e., multiplying each element of the spatial attention weight map with...). (Features from all channels at the corresponding spatial location are multiplied). Through this operation, the network can focus on key local spatial regions in the image, effectively suppressing interference from background regions such as connective tissue and fat.

[0057] By employing the aforementioned dual attention feature reweighting mechanism, the network can adaptively redistribute feature weights in both channel and spatial dimensions, significantly improving segmentation accuracy in low-contrast and complex anatomical scenarios. Experimental data show that in LPD surgical vessel segmentation, this mechanism achieves a Dice coefficient of 0.917 and a mean intersection-over-union (mIoU) ratio of 0.848, both of which are significant improvements over baseline models without an attention mechanism.

[0058] According to embodiments of this disclosure, reference is made to Figure 3 The detailed structure of the segmentation network is as follows: The encoder section comprises eight downsampling stages. Each downsampling stage contains two convolutional units. Each convolutional unit consists of sequentially executed three-dimensional operations: two-dimensional convolution (Conv2d), instance normalization (InstanceNorm2d), and a linear unit with leakage correction (LeakyReLU). In each downsampling stage, the stride of the first convolutional unit is set to 2 for spatial downsampling, and the stride of the second convolutional unit is set to 1 for feature refinement. Through eight stages of progressive downsampling, the spatial resolution of the input image is gradually reduced to its original value. Meanwhile, the number of channels is gradually increased to encode richer semantic information.

[0059] The bottleneck layer connects the encoder and the decoder, performing in-depth processing of the core abstract semantic information at the lowest spatial resolution.

[0060] The decoder section comprises eight upsampling stages, corresponding one-to-one with the eight downsampling stages of the encoder. Each upsampling stage performs the following operations sequentially: First, the previous-level feature map is upsampled by a factor of 2 using transposed convolution (also known as deconvolution) to restore one-time spatial resolution; then, the upsampled feature map is concatenated with the feature map of the corresponding stage in the encoder along the channel dimension. Specifically, let the dimension of the upsampled feature map be... The corresponding encoder feature map dimension is The dimensions after splicing are Next, convolution processing (including convolutional layers, instance normalization, and LeakyReLU activation function) is performed on the concatenated feature map; finally, channel attention and spatial attention are processed sequentially through the dual attention feature reweighting mechanism to obtain the output feature map of the current level.

[0061] Through eight stages of progressive upsampling, the decoder gradually restores the spatial resolution of the image, and the final output feature map has the same spatial resolution as the original input image. Finally, through... The convolutional layer maps the feature map output by the decoder to a multi-channel semantic segmentation result.

[0062] In one embodiment of this disclosure, for an LPD surgical scenario, the training dataset contains multiple frames of endoscopic sample images, each sample image being labeled with segmentation and classification information of tubular structures.

[0063] According to an embodiment of this disclosure, the training dataset contains 3990 endoscopic surgical images with a resolution of [missing information]. The training set contains 3192 images, and the test set contains 798 images. All images were meticulously annotated pixel by pixel by four experienced LPD surgeons using the ISAT annotation tool (https: / / github.com / yatengLG / ISAT_with_segment_anything.git).

[0064] For each sample image, the physician annotated each segment along the course of the blood vessel. The specific segmentation and classification information includes the following three categories: (1) Tubular structures containing only arterial vessels (e.g., gastroduodenal artery, GDA) are labeled as Category I (e.g., Category label “A”). (2) Tubular structures containing only veins (e.g., the gastrocolonic trunk GCT, also known as the Henle trunk) are labeled as Category II (e.g., Category label "V"). (3) Tubular structures that contain both arterial and venous vessels (i.e., areas where arteries and veins are spatially adjacent, run alongside, or cross each other) are marked as Category III (e.g., category label “A+V”).

[0065] By employing the aforementioned three-class labeling strategy—pure arterial segments, pure venous segments, and mixed arteriovenous segments—the segmentation network can simultaneously learn the independent morphological features of arteries and veins during training, as well as their co-occurrence patterns when they are spatially adjacent or intersecting. This enables accurate differentiation between arteries and veins during the inference phase. Particularly in regions where blood vessels accompany or intersect, this labeling strategy helps the network better capture the boundary differences between the two types of vessels.

[0066] According to embodiments of this disclosure, the training process of the segmentation network is as follows: Before training, preprocessing operations are performed on the sample images. The preprocessing operations include: converting the original image to np.float32 format; performing center cropping to remove background areas with all values ​​equal to 0; performing Z-Score normalization on each channel so that the pixel value distribution of each channel has zero mean and unit variance; and recording the foreground position (i.e., pixels containing artery or vein annotations) of each image. The preprocessed data is saved as .b2nd and .pkl format files for training purposes.

[0067] During training, images are sampled from the training set according to a preset patch size. The image is cropped. If the patch exceeds the image boundaries, it is padded; the image padding value is set to 0, and the padding value for the corresponding label is set to -1 (indicating that the padded area is ignored in the loss calculation). To mitigate the class imbalance problem, a foreground oversampling parameter is set. This ensures that at least 33% of the patches in each training batch contain foreground pixels of arteries or veins.

[0068] The segmentation network undergoes multi-task learning during training based on sample images labeled with the first, second, and third categories to simultaneously distinguish between arterial and venous vessels. For example, end-to-end training is performed using a joint loss function combining cross-entropy and Dice loss functions to optimize network parameters. The Adam optimizer is used during training, with an initial learning rate set to... Furthermore, a learning rate decay strategy is employed to promote convergence.

[0069] To enhance the model's generalization ability, online data augmentation is performed on the input images during training. The data augmentation includes at least one of the following methods: Optical distortion: The brightness, contrast, saturation, and hue of the image are randomly adjusted to simulate variations in intra-abdominal lighting conditions. For example, the brightness adjustment range is set to 0.75 to 1.25 times, the contrast adjustment range is set to 0.75 to 1.25 times, and the probability of each is set to 0.15.

[0070] Color space enhancement: The image is converted from the RGB color space to the HSV (hue, saturation, brightness) color space. Each channel is randomly offset in the HSV space and then converted back to the RGB space to enhance the model's ability to recognize blood vessels under different color representations.

[0071] Channel flipping: Randomly swapping the order of the three RGB channels forces the model to learn morphological features that are independent of the color channel order, rather than relying excessively on the color information of specific channels.

[0072] Mosaic enhancement: Multiple training images are stitched together into a large image at random scaling ratios, enabling the model to learn vascular features at different scales and backgrounds simultaneously, thus enhancing its adaptability to vascular anatomical variations.

[0073] Hybrid enhancement: Two training images and their corresponding labels are linearly mixed with random weights to generate new training samples, thereby improving the robustness and generalization ability of the model.

[0074] Low-resolution simulation: The image resolution is randomly reduced and then restored to the original resolution, with a probability of 0.25 and a scale range of 0.5 to 1.0, to simulate image blurring caused by lens contamination, fog, or defocus in endoscopic imaging.

[0075] Gaussian noise: Gaussian noise is randomly added to the image with a probability of 0.1 to improve the model's robustness to imaging noise.

[0076] Gaussian blur: Apply a Gaussian blur to the image randomly with a probability of 0.2 to simulate the resolution loss caused by smoke during surgery.

[0077] Gamma transformation: Randomly perform Gamma correction on the image to simulate non-uniform brightness changes in the laparoscopic field of view caused by tissue occlusion or uneven illumination.

[0078] Through the aforementioned diverse data augmentation strategies, the segmentation network can effectively cope with complex situations such as changes in intraoperative lighting, smoke obstruction, connective tissue coverage, and vascular anatomy variations, demonstrating excellent robustness and segmentation accuracy in actual clinical scenarios.

[0079] Furthermore, during the inference phase (i.e., the actual clinical application phase), the segmentation network provided in this disclosure can run at 30 frames per second (FPS) with an end-to-end processing latency of less than 50 milliseconds. Since the seamless latency threshold for human visual perception is approximately 100 milliseconds, this processing speed fully meets the clinical needs of real-time intraoperative navigation. On the test set, the network's overall segmentation performance achieves a Dice coefficient of 0.917 and a mean intersection-over-union (mIoU) of 0.848, with a significantly improved accuracy in distinguishing between arteries and veins.

[0080] Figure 5 A flowchart illustrating a method for generating stereoscopic display signals according to an embodiment of the present disclosure is shown. Figure 5 As shown, the generation of a stereoscopic display signal with parallax based on a semantic mask includes the following steps S201 to S203.

[0081] In step S201, the semantic mask is superimposed on the video frame to generate an enhanced video frame.

[0082] According to embodiments of this disclosure, arterial and venous semantic masks, obtained in parallel from the multi-channel output layer, are superimposed onto the original video frame as colored semi-transparent masks. For example, a red (RGB value represented as [255,0,0]) semi-transparent mask is assigned to arterial vessels, and a blue (RGB value represented as [0,0,255]) semi-transparent mask is assigned to venous vessels, with a transparency coefficient ranging from 0.4 to 0.6, to ensure that the original tissue texture beneath the mask remains visible. The superimposed pixel values... It can be calculated using the following formula:

[0083] in, These are the pixel values ​​of the original video frames. These are the pixel values ​​(corresponding to colors) of the semantic mask. This is the transparency factor (0.5 for example).

[0084] In step S202, a left-eye view and a right-eye view are generated based on the enhanced video frame, and the left-eye view and the right-eye view have parallax.

[0085] According to embodiments of this disclosure, the endoscopic imaging system is a dual-channel synchronous imaging system capable of simultaneously acquiring high-fidelity video signals from both the left and right perspectives. During the surgery, the dual-channel high-definition laparoscope captures surgical field video at a high frame rate (e.g., 60 frames / second). The system performs semantic segmentation processing as described in steps S101 to S102 on the left-eye and right-eye video frames respectively, obtaining semantic segmentation results for the left-eye perspective and the right-eye perspective, respectively.

[0086] For the left eye channel: the semantic segmentation result from the left eye perspective is superimposed on the original left eye video frame to generate an enhanced left eye video frame. For the right eye channel: the semantic segmentation result from the right eye perspective is superimposed on the original right eye video frame to generate an enhanced right eye video frame. Due to the positional difference between the left and right eyes in the dual-channel imaging system, the enhanced left eye video frame and the enhanced right eye video frame naturally possess a parallax that matches the binocular parallax of the human body. This parallax provides a physiological basis for subsequent stereoscopic vision synthesis.

[0087] In step S203, the left-eye view and the right-eye view are encoded into a single-frame output image in a side-by-side format, and the single-frame output image is transmitted to the polarized stereoscopic display terminal.

[0088] According to embodiments of this disclosure, the left-eye view and right-eye view are encoded into a single-frame output image in a side-by-side (SBS) format. Specifically, the total width of the single-frame output image is the sum of the width of the left-eye view and the width of the right-eye view. For example, if the original image resolution is... The resolution of a single frame output image in SBS format is . The left half The right half of the left eye view is included. The SBS format is used to encode the right eye view. The specific encoding process is implemented using image processing libraries such as OpenCV: OpenCV is used to read the dual-channel video stream frame by frame, each frame is split into left and right eye images, the split left and right eye images are sent to the segmentation network for semantic segmentation, the segmented left and right eye results are then concatenated (horizontal concatenation) into the SBS format, and finally written to the output video stream.

[0089] The single-frame output image is transmitted to the polarized stereoscopic display terminal at a frame rate of, for example, 60 frames per second. The display screen surface of the polarized stereoscopic display terminal is covered with a polarization layer (e.g., a micro-phase difference film or a wire grid polarization film), which converts the left and right halves of the single-frame output image into left-hand circularly polarized light signals and right-hand circularly polarized light signals, respectively.

[0090] The polarized stereoscopic display terminal can be polarized glasses, with polarizers for the left and right eyes respectively. Specifically, the left eye polarizer allows left-handed circularly polarized light to pass through while blocking right-handed circularly polarized light, and the right eye polarizer allows right-handed circularly polarized light to pass through while blocking left-handed circularly polarized light. When the surgeon wears the polarized glasses, the left eye can only receive the left-handed circularly polarized light signal (i.e., the left-eye view), and the right eye can only receive the right-handed circularly polarized light signal (i.e., the right-eye view). Based on the principle of binocular parallax, the surgeon's visual cortex synthesizes the two parallax-affected image signals received by the left and right eyes into a stereoscopic surgical field with a true sense of depth.

[0091] The core advantage of the aforementioned stereoscopic display signal generation scheme lies in the fact that the enhanced view, superimposed with a vascular semantic mask, is presented in stereoscopic form, allowing the surgeon to clearly perceive the spatial depth, location, and hierarchical relationships of blood vessels. In actual LPD surgery, the surgeon can accurately determine the stereoscopic anatomical location of key blood vessels such as the gastroduodenal artery and the gastrocolic trunk, as well as their spatial hierarchical relationship with surrounding tissues, thereby effectively avoiding intraoperative injury. This effect is verified by the following test data: Test results show that the system's end-to-end processing latency is 50 milliseconds, far below the human threshold of 100 milliseconds for imperceptible latency. The navigation screen is smooth and lag-free under dual-screen configuration (one screen displays the original video, and the other displays the navigation-enhanced video). In subjective evaluations by surgeons, the improvement in surgical safety by the stereoscopic view was unanimously recognized.

[0092] According to embodiments of this disclosure, the method further includes: in a dual-monitor configuration, a first monitor displays the original endoscopic video stream, and a second monitor displays a stereoscopic navigation screen overlaid with semantic segmentation results. The surgeon can switch focus as needed, flexibly switching between the original and enhanced visual fields, obtaining AI-assisted vascular recognition and stereoscopic positioning information while fully observing the original tissue texture.

[0093] Figure 6 A structural block diagram of an apparatus for processing endoscopic video streams according to embodiments of the present disclosure is shown. This apparatus can be implemented as part or all of an electronic device through software, hardware, or a combination of both.

[0094] like Figure 6 As shown, the apparatus 600 for processing endoscopic video streams includes an encoding-decoding processing unit 601, a multi-type segmentation output unit 602, and a stereoscopic display signal generation unit 603.

[0095] The encoding-decoding processing unit 601 is configured to perform semantic segmentation on video frames acquired during endoscopic imaging. A dual-attention feature reweighting module is embedded in the encoding-decoding path of the encoding-decoding processing unit. This module performs adaptive weight allocation on intermediate feature maps along the channel and spatial dimensions. The specific structure and working principle of the encoding-decoding processing unit 601 have been described in detail in the aforementioned step S101 and related embodiments, and will not be repeated here.

[0096] The multi-type segmentation output unit 602 is configured to output tubular structure semantic masks carrying type identifiers in parallel, wherein the type identifiers distinguish at least between the first type of tubular structure and the second type of tubular structure. The specific structure and working principle of the multi-type segmentation output unit 602 have been described in detail in the aforementioned step S102 and related embodiments, and will not be repeated here.

[0097] The stereoscopic display signal generation unit 603 is configured to generate a stereoscopic display signal with parallax based on the semantic mask. The specific structure and working principle of the stereoscopic display signal generation unit 603 have been described in detail in the aforementioned step S103 and related embodiments, and will not be repeated here.

[0098] Other technical details of the embodiments disclosed herein can be found in the above embodiments, and will not be repeated here.

[0099] In addition, this disclosure also provides a real-time navigation system for endoscopic surgery. Figure 7 A structural block diagram of a real-time navigation system for endoscopic surgery according to an embodiment of the present disclosure is shown.

[0100] like Figure 7 As shown, the real-time navigation system 700 for endoscopic surgery includes: an endoscopic image acquisition unit 701, a semantic segmentation and stereoscopic visualization processing device 702, and a stereoscopic display terminal 703.

[0101] The endoscopic image acquisition unit 701 is configured to acquire video frames of the surgical field of view. The endoscopic image acquisition unit 701 includes a dual-channel high-definition laparoscope capable of simultaneously acquiring two high-fidelity video signals from the left and right perspectives at a high frame rate (e.g., 60 frames / second). The front end of the dual-channel laparoscope is equipped with dual imaging sensors (e.g., CMOS or CCD image sensors), with a baseline distance between the two sensors matching the interpupillary distance of the human eyes, to acquire left-eye and right-eye images with physiological parallax.

[0102] The semantic segmentation and stereoscopic visualization processing device 702 can be an embodiment of the aforementioned device 600 for processing endoscopic video streams. In practical deployment, this device can be implemented in a standalone image processing workstation equipped with a high-performance graphics processor (such as the NVIDIA RTX series) to support real-time inference operations of deep neural networks. For example, the workstation can be configured with a dual-GPU parallel computing architecture, achieving a real-time semantic segmentation processing speed of no less than 30 frames per second.

[0103] The stereoscopic display terminal 703 is configured to receive the stereoscopic display signal output by the processing device and present the operator with a stereoscopic surgical field of view superimposed with the tubular structure segmentation results. The stereoscopic display terminal 703 includes a polarized stereoscopic display, the display surface of which is covered with a polarization layer (e.g., a micro-phase difference film) for converting the left and right halves of a single-frame output image in SBS format into left-handed and right-handed circularly polarized light signals, respectively.

[0104] The workflow of the real-time navigation system 700 for endoscopic surgery is as follows: First, the endoscopic image acquisition unit 701 acquires dual-channel video signals of the surgical field in real time and transmits them to the semantic segmentation and stereoscopic visualization processing device 702. Next, the processing device 702 performs semantic segmentation on each frame of the image, extracts semantic masks of arteries and veins from the video frames, and superimposes the segmentation results onto the original video frames in the form of color masks to generate enhanced video frames. Then, the processing device 702 encodes the left-eye and right-eye enhanced video frames into stereoscopic display signals in SBS format and transmits them to the stereoscopic display terminal 703. Finally, the stereoscopic display terminal 703 outputs the stereoscopic display signals in the form of left / right circularly polarized light through a polarization layer. The surgeon wears polarized glasses, with the left eye receiving left-handed circularly polarized light signals and the right eye receiving right-handed circularly polarized light signals. Based on the principle of binocular parallax, a stereoscopic surgical field with a realistic sense of depth is synthesized in the visual cortex, thereby clearly perceiving the three-dimensional spatial position of vascular structures and their hierarchical relationship with surrounding tissues during the surgery.

[0105] This disclosure provides a method, apparatus, and system for processing endoscopic video streams. These systems can perform semantic segmentation and generate stereoscopic display signals in real time based on the endoscopic video stream. Even in complex intraoperative environments (including changes in lighting, smoke obstruction, connective tissue coverage, and vascular anatomy variations), the system can automatically and accurately distinguish between arteries and veins and generate a stereoscopic navigation field with realistic depth perception. By combining a segmentation network with a dual attention mechanism, the network can adaptively focus on key vascular areas and filter background noise, significantly improving segmentation accuracy in low-contrast scenes. The system achieves end-to-end low-latency processing within 50 milliseconds and a real-time inference speed of 30 frames per second, fully meeting the stringent real-time requirements of intraoperative navigation. By combining polarized light imaging technology to present the semantic segmentation results in a stereoscopic manner, the system addresses the lack of spatial depth in traditional 2D laparoscopic vision, significantly improving the accuracy of anatomical structure localization for surgeons.

[0106] Experimental data show that, on the LPD-Vessel test set containing 798 test images, the scheme provided in this disclosure achieves excellent performance in joint artery and vein segmentation, with a Dice coefficient of 0.917 and a mean intersection-over-union (mIoU) of 0.848. Compared with the existing technology (High Resolution Network (HRNet) - Fully Convolutional Network (FCN) scheme, with a Dice coefficient of 0.754), the segmentation accuracy is significantly improved. Especially in the identification of key blood vessels such as the gastroduodenal artery and the gastrocolic trunk, this scheme can effectively cope with challenges such as anatomical variations and tissue occlusion, providing a reliable technical guarantee for the prevention of high-risk vascular injury during LPD surgery.

[0107] This disclosure also discloses an electronic device. Figure 8A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0108] like Figure 8 As shown, the electronic device includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to embodiments of the present disclosure.

[0109] Figure 9 A schematic diagram of the structure of a computer system suitable for implementing the method according to embodiments of the present disclosure is shown.

[0110] like Figure 9 As shown, the computer system includes a processing unit that can execute various methods described above based on a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM). The RAM also stores various programs and data required for the operation of the computer system. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0111] The following components are connected to the I / O interface: input sections including keyboards, mice, etc.; output sections including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage sections including hard disks; and communication sections including network interface cards such as LAN cards and modems. The communication section performs communication processes via a network such as the Internet. Drives are also connected to the I / O interface as needed. Removable media, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required. The processing unit can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0112] In particular, according to embodiments of this disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium.

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The units or modules described in the embodiments of this disclosure can be implemented in software or programmable hardware. The described units or modules can also be located in a processor, and the names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0115] In another aspect, this disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system described above; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs, which are used by one or more processors to perform the methods described in this disclosure.

[0116] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for processing endoscopic video streams, characterized in that, include: Video frames acquired during endoscopic imaging are input into a segmentation network with an encoder-decoder structure. Through a dual attention feature reweighting mechanism embedded in the encoder-decoder path of the segmentation network, adaptive weight allocation is performed on the intermediate feature maps along the channel dimension and spatial dimension to generate semantic segmentation results. A tubular structure semantic mask carrying a type identifier is obtained in parallel from the semantic segmentation result. The type identifier is used to distinguish between the first type of tubular structure and the second type of tubular structure. A stereoscopic display signal with parallax is generated based on the semantic mask.

2. The method according to claim 1, characterized in that, The dual attention feature reweighting mechanism includes cascaded channel attention feature reweighting and spatial attention feature reweighting. The channel attention feature reweighting performs adaptive weight allocation on the intermediate feature map along the channel dimension, and the spatial attention feature reweighting performs adaptive weight allocation on the feature map along the spatial dimension after the channel attention feature reweighting.

3. The method according to claim 2, characterized in that, The channel attention feature reweighting includes: Adaptive average pooling and adaptive max pooling operations are performed on the input feature map to obtain the first channel descriptor and the second channel descriptor. The first channel descriptor and the second channel descriptor are respectively input into a shared multilayer perceptron. The shared multilayer perceptron includes a dimensionality reduction layer and a dimensionality increase layer. The dimensionality reduction layer compresses the channel dimension to a preset ratio of the original dimension, and the dimensionality increase layer restores the channel dimension to the original dimension. The two sets of results output by the shared multilayer perceptron are added element by element and then processed by an activation function to generate channel attention weights; The channel attention weights are multiplied channel by channel by the input feature map to obtain a channel-reweighted feature map; The spatial attention feature reweighting includes: Perform channel-level average pooling and channel-level max pooling operations along the channel dimension on the channel reweighted feature map to obtain a first spatial descriptor and a second spatial descriptor; The first spatial descriptor and the second spatial descriptor are concatenated along the channel dimension to generate a spatial feature descriptor; The spatial feature descriptor is subjected to a convolution operation, and after being processed by an activation function, a spatial attention weight map is generated. The spatial attention weight map is multiplied by the input feature map at each spatial location to obtain a double attention reweighted feature map.

4. The method according to claim 1, characterized in that, The first type of tubular structure is an artery, and the second type of tubular structure is a vein. The segmentation network outputs probability maps corresponding to each type in parallel through a multi-channel output layer. Each probability map indicates the probability of the existence of the corresponding type of tubular structure at each pixel position in the video frame.

5. The method according to claim 4, characterized in that, The segmentation network is trained on an labeled training dataset containing multiple frames of endoscopic sample images. Each sample image is labeled with segmentation classification information of tubular structures, including: Tubular structures containing only arterial vessels are designated as Category I; Tubular structures containing only veins are classified as Category II. Tubular structures that contain both arteries and veins are labeled as Category III. The segmentation network performs multi-task learning during training based on sample images labeled with the first, second, and third categories, so as to simultaneously have the ability to distinguish between arterial and venous blood vessels.

6. The method according to claim 1, characterized in that, The generation of a stereoscopic display signal with parallax based on the semantic mask includes: The semantic mask is overlaid onto the video frame to generate an enhanced video frame; A left-eye view and a right-eye view are generated based on the enhanced video frame, and there is a parallax between the left-eye view and the right-eye view.

7. The method according to claim 6, characterized in that, The left eye view and the right eye view are encoded into a single frame output image in a side-by-side format. The left half of the single frame output image carries the left eye view, and the right half carries the right eye view. The single-frame output image is transmitted to a polarized stereoscopic display terminal, which converts the left half and the right half into left-hand circularly polarized light signals and right-hand circularly polarized light signals respectively through a surface polarization layer. The operator wears polarized glasses with corresponding polarizers for the left and right eyes, so that the left eye receives the left-hand circularly polarized light signal and the right eye receives the right-hand circularly polarized light signal. Based on the principle of binocular parallax, a three-dimensional surgical field with a sense of depth is synthesized in the visual cortex.

8. The method according to claim 1, characterized in that, Before inputting the video frame into the segmentation network, the data augmentation process is further performed on the video frame, which includes at least one of photometric distortion, color space enhancement, channel flipping, mosaic enhancement, and blending enhancement.

9. An apparatus for processing endoscopic video streams, characterized in that, include: The encoding-decoding processing unit is configured to perform semantic segmentation on video frames acquired during endoscopic imaging. A dual attention feature reweighting module is embedded in the encoding-decoding path of the encoding-decoding processing unit. The dual attention feature reweighting module performs adaptive weight allocation on the intermediate feature map along the channel dimension and spatial dimension. A multi-type segmentation output unit is configured to output a tubular structure semantic mask carrying a type identifier in parallel, wherein the type identifier distinguishes at least a first type of tubular structure and a second type of tubular structure. A stereoscopic display signal generation unit is configured to generate a stereoscopic display signal with parallax based on the semantic mask.

10. A real-time navigation system for endoscopic surgery, characterized in that, include: The endoscopic image acquisition unit is configured to acquire video frames of the surgical field of view. The apparatus for processing endoscopic video streams as described in claim 9; The stereoscopic display terminal is configured to receive the stereoscopic display signal output by the processing device and present the operator with a stereoscopic surgical field of view superimposed with the results of tubular structure segmentation.

Images