A blood vessel segmentation method for infrared surgical images based on improved U-Net

Abstract

The application provides a blood vessel segmentation method for an infrared operation image based on an improved U-Net, comprising the following steps: step 1, converting an input original picture into standardized data with a high signal-to-noise ratio and meeting the input requirements of a model tensor; step 2, simultaneously calling pre-trained models with different random seeds, the pre-trained models performing parallel prediction and independently generating prediction results; step 3, performing a mean aggregation operation to generate an uncertainty measurement map; step 4, scaling the segmentation result back to the resolution of the original picture through interpolation, converting the continuous probability into a conclusion of whether it is a blood vessel, and finally labeling the input infrared operation image based on the conclusion. Under the premise of not increasing the model parameter quantity, the method realizes physical superposition of weak features. It utilizes the redundancy between feature map channels, forcibly fuses the information of the encoder and the decoder through addition, and significantly improves the detection rate and continuity of small blood vessels in the infrared operation image.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral image processing and video understanding, and specifically relates to a blood vessel segmentation method for infrared surgical images based on an improved U-Net. Background Technology

[0002] Due to the rapid development of deep learning technology, the field of computer vision has made great progress in recent years and is increasingly being applied to other fields. The medical field is one such area closely integrated with computer vision. Blood vessel segmentation is a vivid example of applying computer vision technology to medicine. Segmenting blood vessels in images captured during surgery is of great significance, as it can assist surgeons in accurately identifying and locating important blood vessels, thus improving the success of the operation. However, most current blood vessel segmentation algorithms are designed for fundus images, which often suffer from complex tissue structures and low contrast. Therefore, how to implement a blood vessel segmentation algorithm using existing deep convolutional networks to accurately segment surgical images is a problem that urgently needs to be solved. Summary of the Invention

[0003] Objective: The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing a blood vessel segmentation method for infrared surgical images based on an improved U-Net. Without altering the overall architecture of U-Net, this method improves the accuracy of segmentation edges by introducing a micro-signal enhancement mechanism based on "channel folding" and a residual dual-path feature fusion architecture. Simultaneously, through ensemble learning, it provides interpretable risk assessment, reducing the unpredictability of AI-assisted clinical judgment.

[0004] This method includes the following steps:

[0005] Step 1: Convert the input raw image into standardized data with a high signal-to-noise ratio that meets the requirements of the model tensor input;

[0006] Step 2: Simultaneously call N1 pre-trained models with different random seeds. The pre-trained models are based on the improved U-Net model for blood vessel segmentation. The N1 pre-trained models make predictions in parallel and generate prediction results independently.

[0007] Step 3: Perform mean aggregation operation to average the probability results of N1 pre-trained models to generate a final smooth probability map. Then, calculate the deviation between the predicted values ​​of each pre-trained model to generate an uncertainty metric map.

[0008] Step 4: The segmentation result is scaled back to the resolution of the original image by interpolation. Then, based on a preset threshold, the probability of continuity is converted into a conclusion of whether it is a blood vessel. Finally, the input infrared surgical image is labeled based on the conclusion.

[0009] Step 1 includes:

[0010] Step 1-1: Receive the original infrared surgical image, load the image using the Image.open function in the Pillow image processing library, and call the .convert('RGB') function in the Pillow image processing library to force a unified color space, regardless of the original image's format, converting it into a standard color image; call the operating system interface to dynamically create a temporary working directory in memory or temporary storage space, and copy the input original image to the temporary working directory using the copy2 function in the shutil standard library of Python;

[0011] Step 1-2, Multi-scale normalization processing: Define the standard size of the target image, and then use the Image.resize method in the Pillow library to change the image size. At the same time, select the Image.BILINEAR option for the resample parameter in the Image.resize method to perform bilinear interpolation on the three channels of red (R), green (G), and blue (B) simultaneously. That is, by calculating the weighted average of the surrounding four pixels, the image is adjusted to the target size.

[0012] Steps 1-3: The image is subjected to non-linear transformation using a preset intensity threshold pthrehold. In the code, it is assumed by default that pixels with brightness below the intensity threshold pthrehold within the grayscale range of 0 to 255 are considered as background noise. Only pixels with brightness above the intensity threshold pthrehold are used to calculate the mean and variance of the entire image. The mean and variance calculated here will be used to normalize the entire image later.

[0013] Steps 1-4 involve using the deep learning framework PyTorch. In the code, this is manifested by importing `torch` and using the `torchvision.transforms.ToTensor()` function from the PyTorch vision library `torchvision` to transform the data. `torchvision.transforms.ToTensor()` automatically divides the pixel values ​​in the range [0, 255] by 255, linearly mapping them to the range [0, 1.0]. Simultaneously, it performs axis swapping, converting the data arrangement from (H, W, C) to the (C, H, W) required by the PyTorch framework. Here, H represents the image height (number of rows), W represents the image width (number of columns), and C represents the number of image channels (3 channels for RGB images and 1 channel for grayscale images). While generating the tensor, the original physical parameters of the image are extracted and cached.

[0014] Step 2 includes:

[0015] Step 2-1: Initialize N1 segmentation networks with the same architecture but different training seeds in parallel. The weights of each segmentation network are obtained by training under different random seeds. The segmentation network adopts an improved U-Net architecture. The improvement is that a custom Up_new composite residual module is introduced into the decoder.

[0016] Step 2-2, Preprocessed image tensors The input is a segmentation network, which is an improved symmetric encoder-decoder architecture. Through multi-scale feature extraction and dual-path feature fusion mechanism, it achieves accurate segmentation of microvessels. The segmentation network is divided into the following three stages: encoder path, decoder path and output layer.

[0017] Steps 2-3: Execute the first stage, encoder path;

[0018] Steps 2-4: Execute the second stage, decoder path;

[0019] Steps 2-5 involve performing dual-path processing on the processed low-resolution feature map and high-resolution detail feature map, with the two paths being path A and path B respectively.

[0020] Steps 2-6 execute the third stage, the output layer.

[0021] Steps 2-3 include: In the first stage, the preprocessed image tensor T first passes through the self.inc layer, which is the initial convolutional layer of the segmentation network. As the first layer of the network, it performs preliminary feature extraction on the original input image. In the self.inc layer, it passes through DoubleConv, which is a custom double convolution module. It performs two 3*3 convolution operations using the torch.nn.Conv2d method. torch.nn.Conv2d is a two-dimensional convolution operator in the torch.nn library, used to perform feature extraction. After each convolution operation, torch.nn.BatchNorm2d and ReLU activation functions are applied. torch.nn.BatchNorm2d is a two-dimensional batch normalization component in the torch.nn library, used to normalize the mean and variance of the activation values ​​of each neuron. The primary features of the image are extracted, the number of channels becomes n_filters=32, and the result is saved in x1, which is an intermediate variable used to save the first-size feature map.

[0022] The x1 layer is downsampled stepwise for feature extraction, i.e., through self.down1, self.down2, self.down3, and self.down4. These four layers are custom-defined four-layer feature compression units built upon a custom Down class. Within each Down class, torch.nn.MaxPool2d, a two-dimensional max-pooling operator from the torch.nn library, is executed first to retain the maximum activation value within a preset window, discarding secondary information and halving the width and height. Then, DoubleConv is executed to double the number of channels. The output of each layer is temporarily stored in x2, x3, x4, and x5, respectively. x2, x3, and x4 are custom-defined intermediate variables used to store intermediate-sized feature maps, and x5 is a custom-defined intermediate variable used to store low-level semantic feature maps.

[0023] Steps 2-4 include: After passing through the encoder module consisting of the self.inc layer and self.down1, self.down2, self.down3, and self.down4 layers, the segmentation network stores feature maps of different sizes in intermediate variables x1, x2, x3, x4, and x5 respectively. These feature maps of different sizes then enter the improved upsampling modules self.up1, self.up2, self.up3, and self.up4 respectively. `self.up4` is built based on a custom `Up_new` decoding module. Within each `Up_new` decoding module, low-resolution feature maps and high-resolution detail feature maps need to be processed. First, the low-resolution feature map is processed by first reducing the number of channels through a single-layer 3*3 convolution, and then upsampling is performed using the `torch.nn.Upsample` function. Here, `torch.nn.Unsample` is an upsampling operator in the `torch.nn` library. By setting the `scale_factor` option to 2, the width and height are magnified by a factor of 2, at which point the low-resolution feature map and the high-resolution detail feature map have the same size. Subsequently, feature alignment is performed by calculating the size difference between the low-resolution feature map and the high-resolution detail feature map. If the magnification results in misalignment due to an odd size, padding pixels are used to align the spatial dimensions.

[0024] Steps 2-5 include: Path A is a channel folding branch. Through the DoubleAdd component, the processed low-resolution feature map and high-resolution detail feature map are fused across channels. DoubleAdd is a custom dual-path addition module in the code. The specific operation is as follows: First, the output tensor is reshaped using the torch.reshape function in the torch library, changing the dimension of the input tensor from (1, C, H, W) to (1, C / 2, 2, H, W). Then, the second dimension of the tensor, which has now become (1, C / 2, 2, H, W), is summed, and the number of channels of the tensor changes from C to C / 2. Finally, the processed upsampled features and the skip connection features are concatenated together, and the total number of channels returns to C / 2 + C / 2 = C. Finally, the channels are mean normalized and non-linearly activated using ReLU using the torch.nn.Sequential function in the torch.nn library. The torch.nn.Sequential function is a sequential execution module that wraps the mean normalization operation and non-linear activation operation that need to be performed continuously into a whole.

[0025] Channel B is the convolutional refinement branch. The low-resolution feature map and the high-resolution detail feature map are concatenated in the channel dimension by the torch.cat function in the torch library. Then, it is passed through a DoubleConv and through a cascaded 3*3 convolution, normalization layer and activation layer to extract deep non-linear semantic information.

[0026] Finally, the `torch.add` function from the `torch` library performs a pixel-level addition operation on the results of path A and path B. The output of this pixel-level addition operation is the final output of the `Up_new` module. Based on the `Up_new` module, `self.up1`, `self.up2`, `self.up3`, and `self.up4` are constructed. `self.up1` processes `x5` and `x4`, where `x5` is used as a low-resolution feature map and `x4` is used as a high-resolution detail feature map. The result is stored in the variable `x`, which is then continuously updated by passing through `self.up2`, `self.up3`, and `self.up4`. x and x3 are input into self.up2 for processing, where x is used as a low-resolution feature map and x3 is used as a high-resolution detail feature map. The result is saved to x, and x is updated. At this point, the size of x is the same as that of x3. x and x2 are input into self.up3 for processing, where x is used as a low-resolution feature map and x2 is used as a high-resolution detail feature map. The result is saved to x, and x is updated. At this point, the size of x is the same as that of x2. x and x1 are input into self.up4 for processing, where x is used as a low-resolution feature map and x1 is used as a high-resolution detail feature map. The result is saved to x, and x is updated. Finally, the size and number of channels of x are the same as those of x1.

[0027] Steps 2-6 include: x undergoes a 1*1 convolution through self.outc. self.outc is an output layer built based on a custom OutConv module. In the OutConv module, a 1*1 convolution kernel is used to compress the number of channels of x from n_filters = 32 to n_classes. n_classes is the number of categories in the final classification, which is set to 1 here because there is only one category, blood vessels.

[0028] Step 3 includes:

[0029] Step 3-1, collect the information from step 2. The model generated Output tensor ~ The output of the first model is denoted as , No. The output of each model is denoted as... The Sigmoid activation function is used to transform the value of each point in the logistic regression algorithm. The formula is as follows:

[0030] ,

[0031] Where i is the row number and j is the column number. Let x be the value in the i-th row and j-th column. Let be the probability that the pixel in the i-th row and j-th column belongs to the blood vessel category, with a value range of [0,1]; e is the natural constant.

[0032] based on get Two-dimensional probability matrices generated by each model;

[0033] Step 3-2: Using the `torch.stack` function from the `torch` library, stack the two-dimensional probability matrices generated by the N1 pre-trained models along dimension 0 to form a shape of ( The integrated feature tensor of (1,1,H,W);

[0034] Step 3-3: Using the `torch.mean` function from the `torch` library, perform a pixel-level arithmetic mean operation on the 0th dimension of the stacked tensor.

[0035] ,

[0036] in This represents the probability that the pixel in the i-th row and j-th column of the final result belongs to a blood vessel, which is the value at position (i,j) in the final blood vessel probability distribution map. This represents the value at position (s,1,1,i,j) of the stacked tensor.

[0037] based on Generate the final blood vessel probability distribution map;

[0038] Steps 3-4: For the prediction results of N1 models, calculate the standard deviation pixel-by-pixel using the torch.std function in the torch library. The formula is: , in This represents the value at position (i,j) in the final uncertainty metric plot;

[0039] based on Generate the final uncertainty metric plot.

[0040] Step 4 includes:

[0041] Step 4-1: Retrieve the original physical parameters of the image extracted and cached in Step 1. Use the PIL.Image.fromarray() function in the Pillow library to convert the floating-point channel of [0,1] into an 8-bit unsigned integer single-channel image of [0,255]. Use nearest neighbor interpolation and bilinear interpolation to restore the final blood vessel probability distribution map and the final uncertainty metric map to the original physical size.

[0042] Step 4-2: Set the discrimination threshold. Based on the final blood vessel probability distribution map, identify the positions with a probability higher than the discrimination threshold as blood vessel regions and the positions with a probability lower than the discrimination threshold as non-blood vessel regions. Generate a blood vessel mask. Finally, convert the continuous probability distribution map into a binary blood vessel segmentation map and save it.

[0043] Step 4-3: The binarization mask is superimposed on the original image with yellow and a certain transparency to form a blood vessel overlay image. Based on the uncertainty quantification results, an uncertainty threshold is set. Areas with a standard deviation exceeding the preset uncertainty threshold are identified as high-risk areas, while areas with a standard deviation not exceeding the preset uncertainty threshold are identified as low-risk areas. Areas with a standard deviation exceeding the preset uncertainty threshold are superimposed on the original image with red and a certain transparency to form an uncertainty overlay image.

[0044] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0045] In summary, this method achieves accurate segmentation of blood vessels in surgical infrared images by processing the input infrared surgical images, including data preparation and preprocessing, ensemble model inference, feature aggregation and uncertainty quantification, post-processing and multi-dimensional visualization. This provides clinicians performing surgery with highly valuable blood vessel segmentation images.

[0046] The present invention has the following beneficial effects: (1) The channel reshaping and summation mechanism in the DoubleAdd module of the improved U-Net network proposed in this invention realizes the physical superposition of weak features without increasing the number of model parameters. It utilizes the redundancy between feature map channels and forces the fusion of encoder and decoder information through addition, which significantly improves the detection rate and coherence of small blood vessels in infrared surgical images.

[0047] (2) This invention designs a parallel dual-path decoding structure—path A is responsible for superimposing features extracted from surgical infrared images to ensure structural stability, and path B is responsible for double convolutional refinement to restore nonlinear details. This residual aggregation method alleviates the gradient vanishing problem in deep networks, thereby improving the edge localization accuracy of the model.

[0048] (3) This invention introduces ensemble learning and standard deviation quantification. An uncertainty measurement map is automatically generated through consistency testing of 10 independent seed models. The final output not only includes a blood vessel segmentation map, but also automatically identifies high-risk areas, providing an interpretable risk assessment and a clear confidence index for clinical decision-making, reducing the blindness of AI assistance. Attached Figure Description

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0050] Figure 1 This is a flowchart of the method of the present invention.

[0051] Figure 2 This is a schematic diagram of the self.inc layer.

[0052] Figure 3 This is a schematic diagram of the Down module.

[0053] Figure 4 This is a diagram of the Up_new module.

[0054] Figure 5 This is the fundus image corresponding to specific example 1.

[0055] Figure 6 yes Figure 5 The corresponding binary blood vessel segmentation image.

[0056] Figure 7 for Figure 5 The corresponding blood vessel overlay diagram.

[0057] Figure 8 yes Figure 5 The corresponding uncertainty overlay diagram.

[0058] Figure 9 This is the coronary angiography image corresponding to specific example 2.

[0059] Figure 10 yes Figure 9 The corresponding binary blood vessel segmentation image.

[0060] Figure 11 for Figure 9 The corresponding blood vessel overlay diagram.

[0061] Figure 12 yes Figure 9 The corresponding uncertainty overlay diagram.

[0062] Figure 13 The image shows a coronary artery bypass grafting procedure, as shown in Example 3.

[0063] Figure 14 yes Figure 13 The corresponding binary blood vessel segmentation image.

[0064] Figure 15 for Figure 13 The corresponding blood vessel overlay diagram.

[0065] Figure 16 yes Figure 13 The corresponding uncertainty overlay diagram. Detailed Implementation

[0066] like Figure 1 As shown, this embodiment of the invention provides a blood vessel segmentation method for infrared surgical images based on an improved U-Net, including the following steps:

[0067] Step 1: Convert the input raw image into standardized data with a high signal-to-noise ratio that meets the requirements of the model tensor input.

[0068] Step 2 involves simultaneously invoking 10 pre-trained models with different random seeds. Each of these models is based on an improved U-Net model for vessel segmentation. The 10 models perform predictions in parallel and produce prediction results independently.

[0069] Step 3: First, perform mean aggregation to average the probability results of the ten models and generate a final smooth probability plot. Then, calculate the deviation between the predicted values ​​of each model to generate an uncertainty metric plot.

[0070] Step 4: First, the segmentation result is scaled back to the original image resolution using interpolation. Then, based on a preset threshold, the probability of continuity is converted into a conclusion of whether it is a blood vessel. Finally, the input infrared surgical image is labeled based on the results.

[0071] Step 1 includes:

[0072] Step 1-1: Receive the original infrared surgical image, load the image using the Image.open function in the Pillow image processing library, and call the .convert('RGB') function in the same library to force a uniform color space, regardless of the original image's format, converting it into a standard color image; call the operating system interface to dynamically create a temporary working directory in memory or temporary storage space, and copy the input original image to the temporary working directory using the copy2 function in the shutil standard library of Python;

[0073] Steps 1-2 involve multi-scale normalization. Based on the improved U-Net neural network model for blood vessel segmentation mentioned earlier and used in subsequent step 2, a standard size for the target image is defined, which is 912*912. Then, the Image.resize function from the Pillow library is used to resize the image. Simultaneously, the resample parameter in this function is set to the Image.BILINEAR option to simultaneously perform bilinear interpolation on the red (R), green (G), and blue (B) channels. This involves calculating the weighted average of the surrounding four pixels to adjust the image to the target size.

[0074] Steps 1-3 involve performing a non-linear transformation on the image using a preset intensity threshold, pthrehold. The code assumes that pixels with brightness below pthrehold within the 0-255 grayscale range are considered background noise. Only pixels with brightness above pthrehold are used to calculate the mean and variance of the entire image. These calculated mean and variance will be used later to normalize the entire image. Here, pthrehold is set to 10.

[0075] Steps 1-4: The deep learning framework used in this invention is PyTorch, specifically embodied in the code as `import torch`. In this step, the `torchvision.transforms.ToTensor()` function from the PyTorch vision library `torchvision` is used to transform the data. This function automatically divides pixel values ​​in the range [0, 255] by 255, linearly mapping them to the range [0, 1.0]. Simultaneously, axis swapping is performed, converting the data arrangement from (H, W, C) to (C, H, W) required by the PyTorch deep learning framework. Here, H represents the image height (number of rows), W represents the image width (number of columns), and C represents the number of image channels (3 channels for RGB images and 1 channel for grayscale images). While generating the tensor, the original physical parameters of the image are extracted and cached.

[0076] Step 2 includes:

[0077] Step 2-1: Initialize 10 segmentation networks with the same architecture but different training seeds in parallel. The weights of each segmentation network are obtained by training under different random seeds. The segmentation network adopts an improved U-Net architecture. The improvement is that a custom Up_new composite residual module is introduced into the decoder.

[0078] Step 2-2, Preprocessed image tensors The input segmentation network used in this invention is an improved symmetric encoder-decoder architecture. Through multi-scale feature extraction and dual-path feature fusion mechanisms, it achieves accurate segmentation of fine blood vessels. Broadly speaking, the network can be divided into three stages: the encoder path, the decoder path, and the output layer. In the first stage—the encoder path—the main operations are initial feature extraction, generating initial feature maps and saving them to intermediate variables. Then, progressive downsampling is performed to generate feature maps of different scales and save them to other intermediate variables. These feature maps retain rich local detail information and will be fused with deep semantic features during the encoding stage to compensate for the information loss caused by downsampling. The second stage is the decoder path, where the decoder restores the feature maps to their original size through four improved upsampling modules. The third stage is the output layer. The process described above is explained in detail below: In the first stage, the preprocessed image tensor T first passes through the self.inc layer, which is the model's initialization convolutional layer, such as... Figure 2 As shown, this layer, as the first layer of the network, performs preliminary feature extraction on the original input image. In the self.inc layer, it passes through DoubleConv, a custom classic double convolution module, which performs two 3*3 convolution operations using the torch.nn.Conv2d method. torch.nn.Conv2d is a two-dimensional convolution operator in the torch.nn library used to perform feature extraction. After each convolution operation, torch.nn.BatchNorm2d and ReLU activation functions are applied. torch.nn.BatchNorm2d is a two-dimensional batch normalization component in the torch.nn library used to normalize the mean and variance of the activation values ​​of each neuron. The primary features of the image are extracted, the number of channels becomes n_filters=32, and the result is saved in x1, which is an intermediate variable used to save the first-size feature map.

[0079] Steps 2-3 involve downsampling x1 to extract features step-by-step, i.e., sequentially passing it through layers self.down1, self.down2, self.down3, and self.down4. Here, self.down1, self.down2, self.down3, and self.down4 are custom multi-layer feature compression units, all built upon a custom Down class, such as... Figure 3 As shown, in each custom Down class, torch.nn.MaxPool2d, which is the two-dimensional max pooling operator in the torch.nn library, is first executed to retain the maximum activation value within a preset window, discard secondary information, and halve the width and height. Then DoubleConv is executed, where DoubleConv is the custom double convolution module mentioned earlier, which doubles the number of channels. The output of each layer is temporarily saved in x2, x3, x4, and x5, respectively. x2, x3, and x4 are custom intermediate variables used to save intermediate-sized feature maps, and x5 is an intermediate variable used to save low-level semantic feature maps.

[0080] Steps 2-4: After passing through the encoder module consisting of the `self.inc` layer and `self.down1`, `self.down2`, `self.down3`, and `self.down4` layers, the network stores feature maps of different sizes in intermediate variables `x1`, `x2`, `x3`, `x4`, and `x5`, respectively. These feature maps of different sizes then enter the improved upsampling modules `self.up1`, `self.up2`, `self.up3`, and `self.up4`, respectively. These four upsampling modules are built based on the custom `Up_new` decoding module. Figure 4 As shown, within each Up_new decoding module, low-resolution feature maps and high-resolution detail feature maps need to be processed. First, the low-resolution feature map is processed by performing a single-layer 3x3 convolution to reduce the number of channels, and then upsampling using the `torch.nn.Upsample` function. Here, `torch.nn.Unsample` is an upsampling operator from the `torch.nn` library. By setting the `scale_factor` option to 2, the width and height are magnified by a factor of 2, at which point the low-resolution feature map and the high-resolution detail feature map have the same size. Next, feature alignment is performed by calculating the size difference between the low-resolution feature map and the high-resolution detail feature map. If the magnification results in misalignment due to an odd size, padding pixels are used to align the spatial dimensions.

[0081] Steps 2-5 involve performing dual-path processing on the processed low-resolution feature map and high-resolution detail feature map, with the two paths being path A and path B respectively.

[0082] Path A is a channel folding branch. The DoubleAdd component performs cross-channel fusion of the processed low-resolution feature map and the high-resolution detail feature map. DoubleAdd is a custom dual-path addition module in the code, and its specific operations are as follows: First, the input tensor is reshaped using the torch.reshape function from the torch library, changing its dimensions from (1, C, H, W) to (1, C / 2, 2, H, W). Then, the second dimension of the tensor, now at (1, C / 2, 2, H, W), is summed, reducing the number of channels from C to C / 2. Finally, the results of the low-resolution feature map and the high-resolution detail feature map are concatenated along the channel dimension, bringing the total number of channels back to C / 2 + C / 2 = C. Finally, the torch.nn.Sequential function from the torch.nn library performs mean normalization and ReLU non-linear activation on the channels. This function is a sequential execution module that packages the consecutive mean normalization and non-linear activation operations into a single unit.

[0083] Channel B is the convolutional refinement branch. The low-resolution feature map and the high-resolution detail feature map are concatenated in the channel dimension by the torch.cat function in the torch library. Then, it is passed through a DoubleConv and through a cascaded 3*3 convolution, normalization layer and activation layer to extract deep non-linear semantic information.

[0084] Finally, the `torch.add` function from the `torch` library performs a pixel-level addition operation on the results of path A and path B. The output of this pixel-level addition operation is the final output of the `Up_new` module. Based on the `Up_new` module, `self.up1`, `self.up2`, `self.up3`, and `self.up4` are constructed. `self.up1` processes `x5` and `x4`, where `x5` is used as a low-resolution feature map and `x4` is used as a high-resolution detail feature map. The result is stored in the variable `x`, which is then continuously updated by passing through `self.up2`, `self.up3`, and `self.up4`. x and x3 are input into self.up2 for processing, where x is used as a low-resolution feature map and x3 is used as a high-resolution detail feature map. The result is saved into x, and x is updated. At this point, the size of x is the same as that of x3. x and x2 are input into self.up3 for processing, where x is used as a low-resolution feature map and x2 is used as a high-resolution detail feature map. The result is saved into x, and x is updated. At this point, the size of x is the same as that of x2. x and x1 are input into self.up4 for processing, where x is used as a low-resolution feature map and x1 is used as a high-resolution detail feature map. The result is saved into x, and x is updated. At this point, the size of x is the same as that of x1. Finally, the size and number of channels of x are the same as those of x1.

[0085] Steps 2-6: x undergoes a 1x1 convolution with self.outc, which is the output layer built based on the custom OutConv module. In the OutConv module, a 1x1 convolution kernel is used to compress the number of channels in x from n_filters = 32 to n_classes, where n_classes is the final number of classification categories. Here, it is set to 1 because there is only one category: blood vessels.

[0086] Step 3 includes:

[0087] Step 3-1: Collect the 10 output tensors generated by the 10 models in Step 2. The output of the first model is denoted as The output of the 10th model is denoted as The Sigmoid activation function, used in logistic regression, is applied to transform the value of each point in each output. The formula is as follows:

[0088] ,

[0089] Where i is the row number and j is the column number. Let x be the value in the i-th row and j-th column. Let be the probability that the pixel in the i-th row and j-th column belongs to the blood vessel category, with a value range of [0,1]; e is the natural constant; after processing, we obtain the two-dimensional probability matrices of the 10 models respectively.

[0090] Step 3-2: Using the torch.stack function in the torch library, stack the two-dimensional probability matrices generated by the 10 pre-trained models along dimension 0 to form an integrated feature tensor of shape (10,1,1,H,W).

[0091] Step 3-3: Using the `torch.mean` function from the `torch` library, perform a pixel-level arithmetic mean operation on the 0th dimension of the stacked tensor.

[0092] ,

[0093] in This represents the probability that the pixel in the i-th row and j-th column of the final result belongs to a blood vessel, which is the value at position (i,j) in the final blood vessel probability distribution map. This represents the value at position (s,1,1,i,j) of the stacked tensor.

[0094] This yields the probability that each pixel belongs to a blood vessel, generating the final blood vessel probability distribution map.

[0095] Steps 3-4: For the prediction results of N1 models, calculate the standard deviation pixel-by-pixel using the torch.std function in the torch library. The formula is: , in This represents the value at position (i,j) in the final uncertainty metric graph.

[0096] Generate the final uncertainty metric map. Regions with large standard deviations reflect low confidence areas in the algorithm due to poor image quality, lesion occlusion, or abnormal physiological structures.

[0097] Step 4 includes:

[0098] Step 4-1: Retrieve the original image metadata from the first stage backup. Use the PIL.Image.fromarray() function in the Pillow library to convert the floating-point channel [0,1] into an 8-bit unsigned integer single-channel image [0,255]. Utilize nearest-neighbor interpolation and bilinear interpolation to restore the final blood vessel probability distribution map and the final uncertainty metric map to their original physical dimensions, ensuring that the segmented blood vessel contours can achieve pixel-level precise overlap with the original infrared surgical image.

[0099] Step 4-2: Set a discrimination threshold. Areas with a probability higher than the discrimination threshold are classified as vascular regions, and areas with a probability lower than the discrimination threshold are classified as non-vascular regions. Finally, the continuous probability distribution map is transformed into a Boolean vascular mask map.

[0100] Step 4-3: The binarization mask is superimposed on the original image with yellow and a certain transparency to form a blood vessel overlay image; combined with the uncertainty quantification results, the areas with standard deviation exceeding the preset uncertainty threshold are superimposed on the original image with red and a certain transparency to form a risk warning layer, and finally a complete clinical reference image is formed. The transparency is generally set to 0.6.

[0101] Here is a specific example 1: Figure 5 This is a fundus image. Figure 5 As input, it is processed through steps 1-4. In the processing, the preset intensity threshold pthrehold is set to 40, the discrimination threshold is set to 0.05, the preset uncertainty threshold is set to 0.5, and the superimposed transparency is set to 0.6. Figure 6 for Figure 5 The processed binary blood vessel segmentation image is shown, where the white area represents the blood vessels segmented by the model. Figure 7 for Figure 5 The processed and output blood vessel overlay image Figure 7 for Figure 5 The uncertainty graph is output after processing. It can be seen that, compared to... Figure 5 compared to, Figure 6 The topological structure of blood vessels in the fundus image is clearly and comprehensively presented. Figure 7 This is the result of overlaying the vascular image with the original image; the yellow area represents the blood vessels. For Figure 8 The areas marked in red are regions of high uncertainty, meaning that the probabilities output by different models differ significantly, resulting in large standard deviations. These areas require extra attention.

[0102] The following is a specific example 2: Figure 9 This is a coronary angiography image. Figure 9 As input, it is processed through steps 1-4. In the processing, the preset intensity threshold pthrehold is set to 40, the discrimination threshold is set to 0.05, the preset uncertainty threshold is set to 0.5, and the superimposed transparency is set to 0.6. Figure 10 for Figure 9 The processed binary blood vessel segmentation image is shown, where the white area represents the blood vessels segmented by the model. Figure 11 for Figure 9 The processed and output blood vessel overlay image Figure 12 for Figure 5 The uncertainty graph is output after processing. It can be seen that, compared to... Figure 9 compared to, Figure 10 The vascular topology in the coronary angiography images is clearly and comprehensively presented. Figure 11 This is the result of overlaying the vascular image with the original image; the yellow area represents the blood vessels. For Figure 12 The areas marked in red represent regions of high uncertainty, meaning areas where different models output significantly different probabilities, resulting in larger standard deviations.

[0103] The following is a specific example 3: Figure 13 These are surgical images of a coronary artery bypass graft (CABG) procedure, captured using an infrared camera. The images show the heart undergoing surgery. Figure 13 As input, it is processed through steps 1-4. In the processing, the preset intensity threshold pthrehold is set to 10, the discrimination threshold is set to 0.05, the preset uncertainty threshold is set to 0.5, and the superimposed transparency is set to 0.6. Figure 14 for Figure 13 The processed binary blood vessel segmentation image is shown, where the white area represents the blood vessels segmented by the model. Figure 14 for Figure 13 The processed and output blood vessel overlay image Figure 16 for Figure 13 The uncertainty graph is output after processing. It can be seen that, compared to... Figure 13 compared to, Figure 14 The white area represents the blood vessels segmented by the model. Figure 15 This is the result of overlaying the vascular image with the original image, and... Figure 13 compared to, Figure 15 The blood vessels in the middle and lower parts are marked in yellow, as are some of the upper blood vessels. For Figure 16 The areas marked in red are regions of high uncertainty, meaning that the probabilities output by different models differ significantly, resulting in large standard deviations. These areas require extra attention.

[0104] This invention provides a method for blood vessel segmentation in infrared surgical images based on an improved U-Net. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for blood vessel segmentation in infrared surgical images based on an improved U-Net, characterized in that, Includes the following steps: Step 1: Convert the input raw image into standardized data with a high signal-to-noise ratio that meets the requirements of the model tensor input; Step 2: Simultaneously call N1 pre-trained models with different random seeds. The pre-trained models are based on the improved U-Net model for blood vessel segmentation. The N1 pre-trained models make predictions in parallel and generate prediction results independently. Step 3: Perform mean aggregation operation to average the probability results of N1 pre-trained models to generate a final smooth probability map. Then, calculate the deviation between the predicted values ​​of each pre-trained model to generate an uncertainty metric map. Step 4: The segmentation result is scaled back to the resolution of the original image by interpolation. Then, based on a preset threshold, the probability of continuity is converted into a conclusion of whether it is a blood vessel. Finally, the input infrared surgical image is labeled based on the conclusion.

2. The method according to claim 1, characterized in that, Step 1 includes: Step 1-1: Receive the original infrared surgical image, load the image using the Image.open function in the Pillow image processing library, and call the .convert('RGB') function in the Pillow image processing library to force a unified color space, turning the original image into a standard color image; call the operating system interface to dynamically create a temporary working directory in memory or temporary storage space, and copy the input original image to the temporary working directory using the copy2 function in the shutil standard library of Python; Step 1-2, Multi-scale normalization processing: Define the standard size of the target image, and then use the Image.resize method in the Pillow library to change the image size. At the same time, select the Image.BILINEAR option for the resample parameter in the Image.resize method to perform bilinear interpolation on the three channels of red (R), green (G), and blue (B) simultaneously. That is, by calculating the weighted average of the surrounding four pixels, the image is adjusted to the target size. Steps 1-3: The image is subjected to non-linear transformation using a preset intensity threshold pthrehold. In the code, it is assumed by default that pixels with brightness below the intensity threshold pthrehold within the grayscale range of 0 to 255 are considered as background noise. Only pixels with brightness above the intensity threshold pthrehold are used to calculate the mean and variance of the entire image. Steps 1-4 involve using the deep learning framework PyTorch. In the code, this is manifested by importing `torch` and using the `torchvision.transforms.ToTensor()` function from the PyTorch vision library `torchvision` to transform the data. `torchvision.transforms.ToTensor()` automatically divides the pixel values ​​in the range [0, 255] by 255, linearly mapping them to the range [0, 1.0]. Simultaneously, it performs axis swapping, converting the data arrangement from (H, W, C) to the (C, H, W) required by the PyTorch framework. Here, H represents the image height (number of rows), W represents the image width (number of columns), and C represents the number of image channels (3 channels for RGB images and 1 channel for grayscale images). While generating the tensor, the original physical parameters of the image are extracted and cached.

3. The method according to claim 2, characterized in that, Step 2 includes: Step 2-1: Initialize N1 segmentation networks with the same architecture but different training seeds in parallel. The weights of each segmentation network are obtained by training under different random seeds. The segmentation network adopts an improved U-Net architecture. The improvement is that a custom Up_new composite residual module is introduced into the decoder. Step 2-2, Preprocessed image tensors The input is a segmentation network, which is an improved symmetric encoder-decoder architecture. Through multi-scale feature extraction and dual-path feature fusion mechanism, it achieves accurate segmentation of microvessels. The segmentation network is divided into the following three stages: encoder path, decoder path and output layer. Steps 2-3: Execute the first stage, encoder path; Steps 2-4: Execute the second stage, decoder path; Steps 2-5 involve performing dual-path processing on the processed low-resolution feature map and high-resolution detail feature map, with the two paths being path A and path B respectively. Steps 2-6 execute the third stage, the output layer.

4. The method according to claim 3, characterized in that, Steps 2-3 include: In the first stage, the preprocessed image tensor T first passes through the self.inc layer, which is the initial convolutional layer of the segmentation network. As the first layer of the network, it performs preliminary feature extraction on the original input image. In the self.inc layer, it passes through DoubleConv, which is a custom double convolution module. It performs two 3*3 convolution operations using the torch.nn.Conv2d method. torch.nn.Conv2d is a two-dimensional convolution operator in the torch.nn library, used to perform feature extraction. After each convolution operation, torch.nn.BatchNorm2d and ReLU activation functions are applied. torch.nn.BatchNorm2d is a two-dimensional batch normalization component in the torch.nn library, used to normalize the mean and variance of the activation values ​​of each neuron. The primary features of the image are extracted, the number of channels becomes n_filters=32, and the result is saved in x1, which is an intermediate variable used to save the first-size feature map. The x1 layer is downsampled stepwise for feature extraction, i.e., through self.down1, self.down2, self.down3, and self.down4. These four layers are custom-defined four-layer feature compression units built upon a custom Down class. Within each Down class, torch.nn.MaxPool2d, a two-dimensional max-pooling operator from the torch.nn library, is executed first to retain the maximum activation value within a preset window, discarding secondary information and halving the width and height. Then, DoubleConv is executed to double the number of channels. The output of each layer is temporarily stored in x2, x3, x4, and x5, respectively. x2, x3, and x4 are custom-defined intermediate variables used to store intermediate-sized feature maps, and x5 is a custom-defined intermediate variable used to store low-level semantic feature maps.

5. The method according to claim 4, characterized in that, Steps 2-4 include: After passing through the encoder module consisting of the self.inc layer and self.down1, self.down2, self.down3, and self.down4 layers, the segmentation network stores feature maps of different sizes in intermediate variables x1, x2, x3, x4, and x5 respectively. These feature maps of different sizes then enter the improved upsampling modules self.up1, self.up2, self.up3, and self.up4 respectively. `self.up4` is built based on a custom `Up_new` decoding module. Within each `Up_new` decoding module, low-resolution feature maps and high-resolution detail feature maps need to be processed. First, the low-resolution feature map is processed by first reducing the number of channels through a single-layer 3*3 convolution, and then upsampling is performed using the `torch.nn.Upsample` function. Here, `torch.nn.Unsample` is an upsampling operator in the `torch.nn` library. By setting the `scale_factor` option to 2, the width and height are magnified by a factor of 2, at which point the low-resolution feature map and the high-resolution detail feature map have the same size. Subsequently, feature alignment is performed by calculating the size difference between the low-resolution feature map and the high-resolution detail feature map. If the magnification results in misalignment due to an odd size, padding pixels are used to align the spatial dimensions.

6. The method according to claim 5, characterized in that, Steps 2-5 include: Path A is a channel folding branch. Through the DoubleAdd component, the processed low-resolution feature map and high-resolution detail feature map are fused across channels. DoubleAdd is a custom dual-path addition module in the code. The specific operation is as follows: First, the output tensor is reshaped using the torch.reshape function in the torch library, changing the dimension of the input tensor from (1, C, H, W) to (1, C / 2, 2, H, W). Then, the second dimension of the tensor, which has now become (1, C / 2, 2, H, W), is summed, and the number of channels of the tensor changes from C to C / 2. Finally, the processed upsampled features and the skip connection features are concatenated together, and the total number of channels returns to C / 2 + C / 2 = C. Finally, the channels are mean normalized and non-linearly activated using ReLU using the torch.nn.Sequential function in the torch.nn library. The torch.nn.Sequential function is a sequential execution module that packages the mean normalization operation and non-linear activation operation that need to be performed continuously into a whole. Channel B is the convolutional refinement branch. The low-resolution feature map and the high-resolution detail feature map are concatenated in the channel dimension by the torch.cat function in the torch library. Then, it is passed through a DoubleConv and through a cascaded 3*3 convolution, normalization layer and activation layer to extract deep non-linear semantic information. Finally, the `torch.add` function from the `torch` library is used to directly perform pixel-level addition on the results of path A and path B. The output of this pixel-level addition operation is the final output of the `Up_new` module. Based on the `Up_new` module, `self.up1`, `self.up2`, and `self.up3` are constructed. `self.up4` and `self.up1` process `x5` and `x4`, where `x5` is used as a low-resolution feature map and `x4` as a high-resolution detail feature map. The result is saved to the variable `x`. `x` is then processed sequentially through `self.up2`, `self.up3`, and `self.up4` and continuously updated. `x` and `x3` are then processed by `self.up2`, where `x` is used as a low-resolution feature map and `x3` as a high-resolution detail feature map. The result is saved to `x`, and `x` is updated. At this point, the size of `x` is the same as that of `x3`. `x` and `x2` are then processed by `self.up3`, where `x` is used as a low-resolution feature map and `x2` as a high-resolution detail feature map. The result is saved to `x`, and `x` is updated. At this point, the size of `x` is the same as that of `x2`. `x` and `x1` are then processed by `self.up4`, where `x` is used as a low-resolution feature map and `x1` as a high-resolution detail feature map. The result is saved to `x`, and `x` is updated. Finally, the size and number of channels of `x` are the same as those of `x1`.

7. The method according to claim 6, characterized in that, Steps 2-6 include: x undergoes a 1*1 convolution through self.outc, which is an output layer built based on a custom OutConv module. In the OutConv module, a 1*1 convolution kernel is used to compress the number of channels of x from n_filters = 32 to n_classes, where n_classes is the final number of categories.

8. The method according to claim 7, characterized in that, Step 3 includes: Step 3-1, collect the information from step 2. The model generated Output tensor ~ The output of the first model is denoted as , No. The output of each model is denoted as... The Sigmoid activation function is used to transform the value of each point in the logistic regression algorithm. The formula is as follows: ， Where i is the row number and j is the column number. Let x be the value in the i-th row and j-th column. Let be the probability that the pixel in the i-th row and j-th column belongs to the blood vessel category, with a value range of [0,1]; e is the natural constant. based on get Two-dimensional probability matrices generated by each model; Step 3-2: Using the `torch.stack` function from the `torch` library, stack the two-dimensional probability matrices generated by the N1 pre-trained models along dimension 0 to form a shape of ( The integrated feature tensor of (1,1,H,W); Step 3-3: Using the `torch.mean` function from the `torch` library, perform a pixel-level arithmetic mean operation on the 0th dimension of the stacked tensor. ， in This represents the probability that the pixel in the i-th row and j-th column of the final result belongs to a blood vessel, which is the value at position (i,j) in the final blood vessel probability distribution map. This represents the value at position (s,1,1,i,j) of the stacked tensor. based on Generate the final blood vessel probability distribution map; Steps 3-4: For the prediction results of N1 models, calculate the standard deviation pixel-by-pixel using the torch.std function in the torch library. The formula is: ， in This represents the value at position (i,j) in the final uncertainty metric plot; based on Generate the final uncertainty metric plot.

9. The method according to claim 8, characterized in that, Step 4 includes: Step 4-1: Retrieve the original physical parameters of the image extracted and cached in Step 1. Use the PIL.Image.fromarray() function in the Pillow library to convert the floating-point channel of [0,1] into an 8-bit unsigned integer single-channel image of [0,255]. Use nearest neighbor interpolation and bilinear interpolation to restore the final blood vessel probability distribution map and the final uncertainty metric map to the original physical size. Step 4-2: Set the discrimination threshold. Based on the final blood vessel probability distribution map, identify the positions with a probability higher than the discrimination threshold as blood vessel regions and the positions with a probability lower than the discrimination threshold as non-blood vessel regions. Generate a blood vessel mask. Finally, convert the continuous probability distribution map into a binary blood vessel segmentation map and save it. Step 4-3: The binarization mask is superimposed on the original image with yellow and a certain transparency to form a blood vessel overlay image. Based on the uncertainty quantification results, an uncertainty threshold is set. Areas with a standard deviation exceeding the preset uncertainty threshold are identified as high-risk areas, while areas with a standard deviation not exceeding the preset uncertainty threshold are identified as low-risk areas. Areas with a standard deviation exceeding the preset uncertainty threshold are superimposed on the original image with red and a certain transparency to form an uncertainty overlay image.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 9.

Images