Spatial-frequency decoupled fluorescence-guided surgery image super-resolution reconstruction method

Abstract

This application relates to a spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method, belonging to the field of image analysis technology. The method includes: constructing a super-resolution reconstruction network, which comprises a generator and a discriminator; the generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer; training the super-resolution reconstruction network based on a training dataset to obtain a trained generator; and inputting the low-resolution FGS image to be processed into the trained generator to obtain a super-resolution image. This application achieves synergistic optimization of FGS image noise suppression, artifact reduction, and detail preservation through two-stream decoupling and adaptive gating mechanisms, overcoming the drawback of generative adversarial networks (GANs) easily generating artifacts in medical images, resulting in reconstruction results that better meet realism requirements.

Description

Technical Field

[0001] This application relates to the field of image analysis, and more specifically, to a method for super-resolution reconstruction of fluorescence-guided surgical images with spatial-frequency decoupling. Background Technology

[0002] Fluorescence-guided surgery (FGS) is a real-time navigation technology that combines near-infrared (NIR) fluorescence imaging with intraoperative video monitoring. It can accurately visualize the location and boundaries of key anatomical structures such as tumors, blood vessels, and lymph nodes during open or minimally invasive surgery, assisting surgeons in intraoperative localization and margin assessment. With the clinical widespread use of near-infrared fluorescent probes such as indocyanine green (ICG), FGS has achieved good results in scenarios such as gastrointestinal tumor resection, parathyroid imaging, and colorectal anastomosis blood flow assessment.

[0003] Currently, deep learning methods have been widely applied to medical image super-resolution and enhancement tasks, with Super-Resolution Generative Adversarial Networks (SRGANs) and their improved models being the mainstream techniques. However, existing methods mainly suffer from the following problems: feature learning coupling issues, artifact and noise amplification, inadequate loss function design, and poor training stability. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides a method for super-resolution reconstruction of fluorescence-guided surgical images with spatial-frequency decoupling.

[0005] Firstly, a method for super-resolution reconstruction of fluorescence-guided surgical images with spatial-frequency decoupling is provided, including: A super-resolution reconstruction network is constructed, which includes a generator and a discriminator. The generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer. The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; Each efficient two-stream decoupling structure includes a main path feature mixing module, a residual path feature mixing module, and an adaptive gated fusion module, all configured in parallel. The main path feature mixing module and the residual path feature mixing module are used to enhance the input features, resulting in main path enhanced features and residual path enhanced features, respectively. The main path enhanced features and residual path enhanced features are then fused by the adaptive gated fusion module to obtain the main path output features. The residual path enhanced features serve as the residual path output features. The main path output features and residual path output features from the previous efficient two-stream decoupling structure serve as the input to the next efficient two-stream decoupling structure. The input to the first efficient two-stream decoupling structure is shallow features. The main path output features of all efficient two-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output. The super-resolution reconstruction network is trained based on the training dataset to obtain the trained generator. During the training process, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator is used to distinguish the super-resolution images from real high-resolution images. The trained generator is obtained through adversarial training. The low-resolution FGS image to be processed is input into the trained generator to obtain the super-resolution image.

[0006] In one embodiment, the main path feature mixing module and the residual path feature mixing module have the same structure. The main path feature mixing module includes: a first-layer normalization unit, a parallel segmentation large kernel attention layer, a second-layer normalization unit, and an optimized gated feedforward network connected in sequence. The output of the parallel segmentation of the large kernel attention layer is expressed by the following formula:

[0007] in, To parallel segment the output of the large kernel attention layer, Indicates input, This indicates that the processing has been performed by parallel partitioning the large kernel attention layer. Presentation layer normalization processing, Indicates the scaling parameter; The output of the optimized gated feedforward network is expressed by the following formula:

[0008] in, For the output of the optimized gated feedforward network, This indicates that the signal has been processed by an optimized gated feedforward network. This represents the scaling parameter.

[0009] In one embodiment, the parallel segmentation of the large kernel attention layer includes an attention generation network, three parallel convolutional branches, and a channel splicing and fusion module; The input of the large kernel attention layer is split in parallel and passed through the attention generation network to generate the weights corresponding to the three convolutional branches; The input is split into three channels to obtain three inputs, which are then fed into three convolutional branches to perform convolution operations, resulting in three convolutional results. The three convolutional branches are a 5×5 ordinary convolution, a 7×7 ordinary convolution, and a 7×7 dilated convolution with a dilation rate of 3. The three convolution results are processed by the channel concatenation and fusion module, and then multiplied by their corresponding weights to obtain three multiplication results. The three multiplication results are concatenated and then subjected to convolution operations to obtain the output of the parallel segmented large kernel attention layer.

[0010] In one embodiment, the optimized gated feedforward network is used to achieve the following functions: The input of the optimized gated feedforward network is subjected to a 1×1 convolution, which expands the channel dimension by a factor of 2; then it is uniformly divided into two features in the channel dimension. The two features are multiplied element-wise and then convolved to obtain the output of the optimized gated feedforward network.

[0011] In one embodiment, the adaptive gating fusion module is used to implement the following functions: The main path enhancement features and residual path enhancement features are concatenated along the channel dimension, and a spatially aware gated map is generated using a convolutional layer and a sigmoid activation function, using the following formula:

[0012] in, For gating graphs, It is the Sigmoid activation function. For convolution operations, Indicates main path enhancement features and residual path enhancement features The splicing result; By using a gated graph, the main path enhancement features and residual path enhancement features are complementary and weighted element-wise fused to obtain the fused features, which are expressed by the following formula:

[0013] in, As a feature of fusion, This indicates element-wise multiplication; The fused features are subjected to a hybrid convolution operation, and the residual learning is fed back to the main path enhancement features to obtain the main path output features, which are expressed by the following formula:

[0014] in, The main path output feature of the k-th efficient two-stream decoupling structure is given. This represents the scaling parameter. This indicates a hybrid convolution operation.

[0015] In one embodiment, the PixelShuffle upsampling layer is used to achieve the following functions; The input to the PixelShuffle upsampling layer is sequentially subjected to 3×3 ordinary convolution, PixelShuffle operation, GELU activation function, 3×3 ordinary convolution, and 3×3 ordinary convolution to obtain the convolution result. The input to the PixelShuffle upsampling layer is bilinearly interpolated and then summed with the convolution result to obtain the output of the PixelShuffle upsampling layer.

[0016] In one embodiment, the training process employs a two-stage training strategy. In the first stage, the loss function used includes deep supervised loss. Edge-weighted Charbonnier pixel loss and structural similarity loss In the second phase, the loss employed also includes perceived loss. and combat losses .

[0017] In one embodiment, the deep supervision loss is:

[0018] in, To deeply supervise the losses, The number of efficient two-stream decoupling structures, For Charbonnier distance, For lightweight decoders, The residual path output characteristics of the k-th efficient two-stream decoupling structure are given. Ground truth value after spatial downsampling; The edge-weighted Charbonnier pixel loss is:

[0019] in, For edge-weighted Charbonnier pixel loss, For the height of the image, The width of the image, For the number of channels, For the generated super-resolution image, For ground truth, It is the stability constant. The edge reinforcement strength coefficient, This is a spatial weight map extracted from the gradient of the ground truth image; The structural similarity loss is:

[0020] in, For structural similarity loss, For structural similarity calculation, For the generated super-resolution image, This is the ground truth value.

[0021] Secondly, a space-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction device is provided, comprising: The network building module is used to build a super-resolution reconstruction network, which includes a generator and a discriminator. The generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer. The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; Each efficient two-stream decoupling structure includes a main path feature mixing module, a residual path feature mixing module, and an adaptive gated fusion module, all configured in parallel. The main path feature mixing module and the residual path feature mixing module are used to enhance the input features, resulting in main path enhanced features and residual path enhanced features, respectively. The main path enhanced features and residual path enhanced features are then fused by the adaptive gated fusion module to obtain the main path output features. The residual path enhanced features serve as the residual path output features. The main path output features and residual path output features from the previous efficient two-stream decoupling structure serve as the input to the next efficient two-stream decoupling structure. The input to the first efficient two-stream decoupling structure is the shallow features. The main path output features of all efficient two-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output. The network training module is used to train the super-resolution reconstruction network based on the training dataset to obtain the trained generator. During the training process, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator is used to distinguish the super-resolution images from real high-resolution images. The trained generator is obtained through adversarial training. The reconstruction module is used to input the low-resolution FGS image to be processed into the trained generator to obtain a super-resolution image.

[0022] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the aforementioned method for super-resolution reconstruction of fluorescence-guided surgical images with spatial-frequency decoupling.

[0023] Compared with existing technologies, this application has the following advantages: This application designs an efficient two-stream decoupling structure in the generator. This efficient two-stream decoupling structure employs a feature mixing module to achieve spatial feature mixing and channel feature modulation. An adaptive gating fusion module dynamically adjusts the contribution weights of the main path (details) and the residual path (scattered background) according to the spatial content. Through this adaptive adjustment, it ensures that the scattering components fitted by the residual path do not contaminate the sharp boundaries of the reconstructed main path, achieving deep decoupling and accurate fusion of features at the physical level. This application, through two-stream decoupling and adaptive gating mechanisms, achieves synergistic optimization of FGS image noise suppression, artifact reduction, and detail preservation, overcoming the drawback of Generative Adversarial Networks (GANs) easily generating artifacts in medical images, resulting in reconstruction results that better meet the requirements of realism. Attached Figure Description

[0024] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a fluorescence-guided surgical image super-resolution reconstruction method with spatial-frequency decoupling is shown. Figure 2 A schematic diagram of a super-resolution reconstruction network is shown; Figure 3 A schematic diagram of the generator is shown; Figure 4 A schematic diagram of the main path feature mixing module is shown; Figure 5 A schematic diagram of parallel segmentation of the large kernel attention layer is shown; Figure 6 A schematic diagram of an optimized gated feedforward network is shown; Figure 7 A schematic diagram of the adaptive gating fusion module is shown; Figure 8 A schematic diagram of the PixelShuffle upsampling layer is shown. Detailed Implementation

[0025] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0026] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0027] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0028] This application provides a method for super-resolution reconstruction of fluorescence-guided surgical images with spatial-frequency decoupling. Figure 1 A flowchart of a fluorescence-guided surgical image super-resolution reconstruction method with spatial-frequency decoupling is shown. See [link to relevant documentation]. Figure 1 The method mainly includes the following steps: Step S1: Construct a super-resolution reconstruction network, which includes a generator and a discriminator. The generator includes a shallow feature extraction module, multiple sequentially connected efficient dual-stream decoupling structures (ETDS), and a PixelShuffle upsampling layer. Figure 2 A schematic diagram of a super-resolution reconstruction network is shown. Figure 3 A schematic diagram of the generator is shown.

[0029] The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; here, the shallow feature extraction module is a 3×3 convolution.

[0030] See Figure 2 and Figure 3 Each efficient dual-stream decoupling structure includes a parallel main path feature mixing module (FMM), a residual path feature mixing module (FMM), and an adaptive gated fusion module. The main path feature mixing module and the residual path feature mixing module are used to enhance the input features, resulting in main path enhanced features and residual path enhanced features, respectively. The main path enhanced features and residual path enhanced features are fused by the adaptive gated fusion module to obtain the main path output features. The residual path enhanced features are used as residual path output features to calculate the deep supervision loss. Here, the main path output features and residual path output features of the previous efficient two-stream decoupling structure are used as inputs to the next efficient two-stream decoupling structure. Specifically, the main path output features of the previous efficient two-stream decoupling structure are used as inputs to the main path feature mixing module of the next efficient two-stream decoupling structure, and the residual path output features of the previous efficient two-stream decoupling structure are used as inputs to the residual path feature mixing module of the next efficient two-stream decoupling structure. The input to the first efficient two-stream decoupling structure is the shallow feature.

[0031] The main path output features of all efficient dual-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output.

[0032] Here, the generator takes a low-resolution FGS image (LR image) as input and outputs a super-resolution image (SR image).

[0033] Step S2 involves training the super-resolution reconstruction network based on the training dataset to obtain the trained generator. During training, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator distinguishes between the super-resolution images and the real high-resolution images (HR images). The trained generator is obtained through adversarial training. Here, the training dataset uses an existing publicly available fluorescence-guided surgery dataset, with samples consisting of low-resolution FGS images and their corresponding real high-resolution images.

[0034] Step S3: Input the low-resolution FGS image to be processed into the trained generator to obtain the super-resolution image.

[0035] In one embodiment, the main path feature mixing module and the residual path feature mixing module have the same structure. The following description focuses on the main path feature mixing module as an example, detailing its functionality. This embodiment reconstructs and optimizes the feature mixing module by serially and parallelly splitting the large kernel attention network (PLKA) and the optimized gated feedforward network (OptimizedGated-FFN) to achieve efficient spatial feature mixing and channel feature modulation, respectively. To improve training stability, layer normalization is applied before each sub-module.

[0036] Figure 4 A schematic diagram of the main path feature blending module is shown; see [link / reference]. Figure 4 The main path feature mixing module includes: a first-layer normalization unit, a parallel segmented large-kernel attention layer PLKA, a second-layer normalization unit, and an optimized gated feedforward network Gated-FFN, connected in sequence. The output of the parallel segmentation of the large kernel attention layer is expressed by the following formula:

[0037] in, To parallel segment the output of the large kernel attention layer, Indicates input, This indicates that the processing has been performed by parallel partitioning the large kernel attention layer. Presentation layer normalization processing, Indicates the learnable scaling parameter; The output of the optimized gated feedforward network is expressed by the following formula:

[0038] in, For the output of the optimized gated feedforward network, This indicates that the signal has been processed by an optimized gated feedforward network. This represents the learnable scaling parameters.

[0039] Specifically, to address the problem of diverse lesion feature scales and long-range scattering blurring in fluorescence images, a PLKA layer is proposed, employing a "segmentation-transformation-fusion" strategy to capture multi-scale spatial information.

[0040] Figure 5 A schematic diagram of parallel segmentation of the large kernel attention layer is shown. See [link / reference] Figure 5 The parallel segmentation of the large kernel attention layer includes the attention generation network AttNet, three parallel convolutional branches, and a channel splicing and fusion module; The input of the large kernel attention layer is split in parallel and then passed through the attention generation network AttNet to generate the weights corresponding to the three convolutional branches.

[0041] in, Indicates weight, This is used to segment the input of the large kernel attention layer in parallel.

[0042] The input is split into three channels to obtain three inputs, which are then fed into three convolutional branches to perform convolution operations, resulting in three convolutional results. The three convolutional branches are a 5×5 ordinary convolution, a 7×7 ordinary convolution, and a 7×7 dilated convolution with a dilation rate of 3. Here, the input feature channels are uniformly divided into three groups, X1, X2, and X3. The three branches perform convolution operations with different receptive fields in parallel: the first branch uses a 5×5 convolution to capture local details; the second branch uses a 7×7 convolution to extract mid-range texture; and the third branch uses a 7×7 dilated convolution with a dilation rate of 3 (equivalent to a receptive field of 19×19), specifically designed to capture long-range dependencies and large-scale tissue scattering patterns.

[0043] The three convolution results are processed by the channel concatenation and fusion module, and then multiplied by their corresponding weights to obtain three multiplication results. The three multiplication results are concatenated and then subjected to convolution operations to obtain the output of the parallel segmented large kernel attention layer.

[0044]

[0045] in, To parallel segment the output of the large kernel attention layer, This is a convolution operation, specifically a 1×1 convolution. This represents the convolution result of the i-th branch. For element-wise multiplication, Let be the weight of the i-th branch.

[0046] This design allows PLKA to simultaneously restore fine boundaries (small receptive field) and scattered halos (large receptive field).

[0047] The above embodiments address the problem that diagnostic targets in fluorescence images span a vast range of scales, encompassing both extremely fine vascular terminals (local features) and extensive tissue scattering patterns (global features). Existing technologies often employ single-scale or sequential convolutions, which struggle to efficiently capture both microscopic details and macroscopic background simultaneously, and incur a significant increase in computational cost when processing large receptive fields. This embodiment proposes a "segmentation-parallelism-fusion" strategy, splitting the channels and processing them through three parallel branches: using 5×5 convolutions to extract local details, and using expanded convolutions (response field reaching 19×19) to capture long-range scattering patterns. This design, without significantly increasing the number of parameters, enables the model to adaptively capture lesion features at different scales, solving the problem that a single receptive field cannot adapt to the diversity of medical targets.

[0048] Specifically, addressing the issue of strong "signal sparsity" (dark background, sparse bright target) in fluorescence images, traditional convolutional networks tend to amplify random noise in dark background regions, interfering with lesion identification. This embodiment introduces the SimpleGate mechanism to replace traditional nonlinear activation. This mechanism implements a "soft mask" function through element-wise multiplication, automatically identifying and amplifying informative sparse fluorescence features based on feature responses, while effectively suppressing dominant black background noise, significantly improving the reconstruction signal-to-noise ratio in sparse signal environments. Figure 6 A schematic diagram of an optimized gated feedforward network is shown. This optimized gated feedforward network is used to achieve the following functions: The input of the optimized gated feedforward network is subjected to a 1×1 convolution, which expands the channel dimension by a factor of 2; then it is uniformly divided into two features in the channel dimension. The two features are multiplied element-wise and then convolved to obtain the output of the optimized gated feedforward network.

[0049] Here, gated modulation based on SimpleGate is employed: element-wise multiplication is used to implement the gate logic, which acts as a "soft mask" to selectively amplify meaningful lesion features and suppress invalid background noise. Finally, channel dimensions are restored through convolution. Compared to traditional ReLU or GELU activation functions, this parameter-free nonlinear activation method achieves clearer boundary reconstruction with lower latency, and is particularly well-suited to the sparse distribution characteristics of fluorescence signals.

[0050] In one embodiment, a gating fusion mechanism is introduced to achieve efficient integration of the two features and prevent background noise from interfering with the details. Figure 7 A schematic diagram of the adaptive gating fusion module is shown. (See attached diagram) Figure 7 The adaptive gating fusion module is used to implement the following functions: The main path enhancement features and residual path enhancement features are concatenated along the channel dimension, and a spatially aware gated map is generated using a convolutional layer and a sigmoid activation function, using the following formula:

[0051] in, For gating graphs, It is the Sigmoid activation function. For convolution operations, Indicates main path enhancement features and residual path enhancement features The splicing result; By using a gated graph, the main path enhancement features and residual path enhancement features are complementary and weighted element-wise fused to obtain the fused features, which are expressed by the following formula:

[0052] in, As a feature of fusion, This indicates element-wise multiplication.

[0053] This mechanism is achieved through G and (1) The complementary design of G) dynamically adjusts the contribution of detail information and background information at different spatial locations, ensuring that the scattering background of the residual path fitting does not contaminate the sharp boundary of the main path reconstruction.

[0054] The fused features are subjected to a hybrid convolution operation, and the residual learning is fed back to the main path to enhance the features, resulting in the main path output features. This achieves iterative optimization of details, as expressed by the following formula:

[0055] in, The main path output feature of the k-th efficient two-stream decoupling structure is given. This represents the learnable scaling parameter, which is initialized to 0 to ensure stability during the initial training phase. This indicates a hybrid convolution operation.

[0056] It is worth noting that the residual path output features are independently passed to the next level module while remaining unchanged, thereby continuously maintaining the scattering prior information in the deep network and providing a stable physical constraint for feature decoupling.

[0057] The above embodiments address the problem that in medical fluorescence imaging, the "high-brightness halo" (low frequency) caused by tissue scattering is highly coupled with the lesion edge / vascular details (high frequency), making it difficult for traditional networks to maintain edge sharpness while suppressing scattering blur. Existing technologies employ "equivalent transformation constraints" to pursue faster inference speeds on mobile devices, sacrificing the flexibility of feature fusion. This results in incomplete decoupling when handling complex medical scattering interference, making details susceptible to background noise contamination. This embodiment abandons the equivalent transformation constraint and introduces a learnable Sigmoid-gated mapping. This mechanism dynamically adjusts the contribution weights of the main path (details) and the residual path (scattering background) based on spatial content. Through this adaptive adjustment, it ensures that the scattering components fitted by the residual path do not contaminate the sharp boundaries of the reconstructed main path, achieving deep decoupling and precise fusion of features at the physical level.

[0058] In one embodiment, Figure 8 A schematic diagram of the PixelShuffle upsampling layer is shown. The PixelShuffle upsampling layer is used to implement the following functions; The input to the PixelShuffle upsampling layer is sequentially subjected to 3×3 ordinary convolution, PixelShuffle operation, GELU activation function, 3×3 ordinary convolution, and 3×3 ordinary convolution to obtain the convolution result. The input to the PixelShuffle upsampling layer is bilinearly interpolated and then summed with the convolution result to obtain the output of the PixelShuffle upsampling layer.

[0059] In one embodiment, the training process employs a two-stage training strategy. In the first stage, the loss function used includes deep supervised loss. Edge-weighted Charbonnier pixel loss and structural similarity loss Learning anatomical structures and features decoupling, with an initial learning rate of 2×10⁻⁶. 4 In the second phase, the loss method also includes perceived loss. and combat losses The high-frequency fluorescent texture was restored while maintaining the correct structure.

[0060] Introducing deep supervision loss Content constraints are applied to the two-stream branches. Unlike the stepwise refinement strategy of traditional multi-stage networks, this embodiment uses this mechanism to force physical feature decoupling: that is, to force the residual path (low-frequency branch) to fit the scattering components in the image layer by layer.

[0061] Specifically, the losses from in-depth supervision are:

[0062] in, To deeply supervise the losses, The number of efficient two-stream decoupling structures, For Charbonnier distance, For lightweight decoders, The residual path output characteristics of the k-th efficient two-stream decoupling structure are given. Ground truth value after spatial downsampling; This loss explicitly guides the residual path to separate background structure from tissue scattering halos, allowing the main path to focus on the recovery of high-frequency diagnostic details.

[0063] To address the issue of insufficient sensitivity to edge blurring in standard pixel loss, this embodiment proposes an edge-weighted mechanism. This mechanism assigns higher penalty weights to high-frequency boundary regions, forcing the network to accurately delineate tumor edges and vascular details, preventing details from being obscured by background information. The edge-weighted Charbonnier pixel loss is as follows:

[0064] in, For edge-weighted Charbonnier pixel loss, For the height of the image, The width of the image, For the number of channels, For the generated super-resolution image, For ground truth, It is the stability constant. The edge reinforcement strength coefficient, This is a spatial weight map extracted from the gradient of the ground truth image; To overcome the limitation of simple gradient loss in its sensitivity to noise, SSIM loss is introduced to evaluate the consistency between the generated image and the ground truth in terms of brightness, contrast, and structural topology, ensuring the biological plausibility of the tissue structure. The structural similarity loss is:

[0065] in, For structural similarity loss, For structural similarity calculation, For the generated super-resolution image, This is the ground truth value.

[0066] Furthermore, a pre-trained VGG-19 network is used to extract features and minimize semantic bias in the feature space to reconstruct realistic tissue texture and eliminate blur. Its parameters are frozen (not used in training). Specifically, the super-resolution image SR and the high-resolution image HR are input into the VGG-19 network respectively to extract feature maps of specific layers. Then, the Euclidean distance or L1 distance between these two sets of feature maps is calculated as the perceptual loss.

[0067] The adversarial loss is calculated based on the local truth probability matrix output by the discriminator.

[0068] The discriminator not only evaluates the absolute realism of the image, but also estimates the relative probability that the real image is "more real" than the generated image. This mechanism provides a sharper gradient signal, which helps to stably recover fine fluorescence textures without causing mode collapse.

[0069] Specifically, traditional discriminators typically end with a fully connected layer, outputting a single scalar (0 or 1) to evaluate the truth value of the entire image. This embodiment, however, uses the PatchGAN architecture (combined with spectral normalization) to generate the local true probability matrix as follows: The fully convolutional structure is adopted: the discriminator removes the last fully connected layer and consists entirely of convolutional layers.

[0070] Receptive field mapping: After the input image undergoes layers of convolution and downsampling, its spatial size decreases, but it is not compressed into a single number. The final output is an N×N feature map (local true probability matrix).

[0071] Local correspondence: Each pixel in this N×N matrix (e.g., the value in the x-th row and y-th column) logically corresponds to a specific size region (Patch, e.g., a 70×70 pixel region) in the original image.

[0072] Matrix meaning: Each value in the matrix represents the probability that the corresponding local region in the original image is "real" or "generated". The final adversarial loss is the average of all values ​​in this matrix.

[0073] The loss function used in this embodiment addresses the following issues: Super-resolution algorithms, while improving sharpness, are prone to numerical shifts (distorting the meaning of physical light intensity) and "softening" of lesion edges, and adversarial training is highly unstable with small medical samples. Traditional SRGAN or ESRGAN often suffer from grayscale overflow due to a lack of physical constraints, and their discriminators are prone to mode collapse with small samples. In this embodiment, a deep supervised loss (DSL) is constructed and directly applied to the decoupled branch, forcing the model to learn the scattering degradation distribution. Spectral normalization is introduced into the discriminator and combined with the PatchGAN architecture, refining the evaluation dimension from "whole image authenticity" to "local texture authenticity," forcing the model to optimize every local anatomical detail, ensuring high-fidelity, artifact-free reconstruction results even with limited sample size.

[0074] To verify the effectiveness of the method in this application, quantitative tests were performed on a self-built surgical fluorescence dataset and a publicly available fluorescence-guided surgery (FGS) dataset.

[0075] 1) Definition of evaluation indicators The following two internationally recognized image quality evaluation metrics are adopted: Peak Signal-to-Noise Ratio (PSNR): This measures the pixel error between the reconstructed image and the ground truth (GT). A higher PSNR value indicates higher signal fidelity and better noise reduction.

[0076] Structural similarity (SSIM): Used to evaluate the consistency between the reconstructed image and the ground truth in terms of brightness, contrast and structural topology. The closer the value is to 1, the more complete the anatomical structures (such as blood vessel texture and tissue edges) are preserved.

[0077] 2) Analysis of quantitative experimental results ① Performance on self-built surgical dataset As shown in Table 1, in the test on the self-built surgical fluorescence dataset, the method of this application significantly outperforms the traditional bicubic interpolation and the classic deep learning methods (SRGAN, Real-ESRGAN) in both PSNR and SSIM.

[0078] The PSNR of the method in this application reaches 49.46 dB, which is an improvement of approximately 5.71 dB compared to SRGAN and approximately 10.36 dB compared to Real-ESRGAN.

[0079] Beneficial effects: Experimental results demonstrate that, for the specific scattering and noise characteristics of fluorescence images, the ETDS and FMM modules of this application can effectively filter out background interference and achieve feature recovery with extremely high fidelity.

[0080] Table 1

[0081] ② Generalization validation on the public FGS dataset To verify the generalization ability of the algorithm, further tests were conducted on the publicly available fluorescence-guided surgery dataset (using the fluorescence-guided surgery dataset from the literature Schmidt, A., Mohareri, O., DiMaio, SP, et al. Surgical tattoos in infrared: A dataset for quantifying tissue tracking and mapping. IEEE Transactions on Medical Imaging, 43(7), 2634-2645(2024), 10.1109 / TMI.2024.3372828). The results are shown in Table 2. The method in this application still maintains its leading position, with a PSNR of 32.90 dB and an SSIM of 0.8813.

[0082] Comparative analysis: Although the metrics of all models decreased in the public dataset with complex noise and more texture, the method of this application showed the smallest decrease, and the SSIM was significantly higher than that of other models, indicating that the method of this application is more robust in protecting the anatomical structure of the lesion.

[0083] Table 2

[0084] In summary, the method of this application has the following beneficial effects: Extremely high signal fidelity: High PSNR values ​​can be obtained on fluorescence images from different sources, effectively suppressing common intraoperative imaging noise.

[0085] Superior structural protection: The consistent leading SSIM index demonstrates the superiority of this application in restoring the boundaries of small lesions and vascular details, providing a clearer and more accurate visual reference than existing technologies.

[0086] High applicability: This application overcomes the drawback of Generative Adversarial Networks (GANs) being prone to artifacts in medical images by using dual-stream decoupling and adaptive gating mechanisms, and the reconstruction results are more in line with the requirements of realism.

[0087] Based on the same inventive concept as the space-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method, this embodiment also provides a corresponding space-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction device, including: The network building module is used to build a super-resolution reconstruction network, which includes a generator and a discriminator. The generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer. The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; Each efficient two-stream decoupling structure includes a main path feature mixing module, a residual path feature mixing module, and an adaptive gated fusion module, all configured in parallel. The main path feature mixing module and the residual path feature mixing module are used to enhance the input features, resulting in main path enhanced features and residual path enhanced features, respectively. The main path enhanced features and residual path enhanced features are then fused by the adaptive gated fusion module to obtain the main path output features. The residual path enhanced features serve as the residual path output features. The main path output features and residual path output features from the previous efficient two-stream decoupling structure serve as the input to the next efficient two-stream decoupling structure. The input to the first efficient two-stream decoupling structure is the shallow features. The main path output features of all efficient two-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output. The network training module is used to train the super-resolution reconstruction network based on the training dataset to obtain the trained generator. During the training process, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator is used to distinguish the super-resolution images from real high-resolution images. The trained generator is obtained through adversarial training. The reconstruction module is used to input the low-resolution FGS image to be processed into the trained generator to obtain a super-resolution image.

[0088] The spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction device of this embodiment has the same inventive concept as the spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method described above. Therefore, the specific implementation of this device can be found in the embodiment section of the spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method described above, and its technical effects correspond to the technical effects of the above method, so it will not be repeated here.

[0089] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method.

[0090] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for spatial-frequency decoupled fluorescence-guided surgery image super-resolution reconstruction, characterized in that, include: A super-resolution reconstruction network is constructed, which includes a generator and a discriminator; the generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer; The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; Each of the efficient dual-stream decoupling structures includes a main path feature mixing module, a residual path feature mixing module, and an adaptive gated fusion module arranged in parallel; the main path feature mixing module and the residual path feature mixing module are used to enhance the input features to obtain main path enhanced features and residual path enhanced features, respectively; The main path enhancement feature and the residual path enhancement feature are fused by the adaptive gated fusion module to obtain the main path output feature; the residual path enhancement feature is used as the residual path output feature; the main path output feature and the residual path output feature of the previous efficient two-stream decoupling structure are used as the input of the next efficient two-stream decoupling structure; the input of the first efficient two-stream decoupling structure is the shallow feature; The main path output features of all efficient two-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output. The super-resolution reconstruction network is trained based on the training dataset to obtain the trained generator; During training, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator is used to distinguish the super-resolution images from real high-resolution images. The trained generator is obtained through adversarial training. The low-resolution FGS image to be processed is input into the trained generator to obtain a super-resolution image.

2. The method as described in claim 1, characterized in that, The main path feature mixing module and the residual path feature mixing module have the same structure. The main path feature mixing module includes: a first-layer normalization unit, a parallel segmentation large kernel attention layer, a second-layer normalization unit, and an optimized gated feedforward network connected in sequence. The output of the parallel segmentation of the large kernel attention layer is expressed by the following formula: in, To parallel segment the output of the large kernel attention layer, Indicates input, This indicates that the processing has been performed by parallel partitioning the large kernel attention layer. Presentation layer normalization processing, Indicates the scaling parameter; The output of the optimized gated feedforward network is expressed by the following formula: in, For the output of the optimized gated feedforward network, This indicates that the signal has been processed by an optimized gated feedforward network. This represents the scaling parameter.

3. The method as described in claim 2, characterized in that, The parallel segmentation large kernel attention layer includes an attention generation network, three parallel convolutional branches, and a channel splicing and fusion module; The input of the large kernel attention layer is split in parallel and passed through the attention generation network to generate weights corresponding to the three convolutional branches; The input is split into three channels to obtain three inputs, which are then fed into three convolutional branches to perform convolution operations, resulting in three convolutional results. The three convolutional branches are a 5×5 ordinary convolution, a 7×7 ordinary convolution, and a 7×7 dilated convolution with a dilation rate of 3. The three convolution results are processed by the channel splicing and fusion module, and multiplied by their corresponding weights to obtain three multiplication results. The three multiplication results are spliced ​​together and then subjected to a convolution operation to obtain the output of the parallel segmentation large kernel attention layer.

4. The method as described in claim 2, characterized in that, The optimized gated feedforward network is used to achieve the following functions: The input of the optimized gated feedforward network is subjected to a 1×1 convolution, which expands the channel dimension by a factor of 2; then it is uniformly divided into two features in the channel dimension. The two features are multiplied element-wise and then convolved to obtain the output of the optimized gated feedforward network.

5. The method as described in claim 1, characterized in that, The adaptive gating fusion module is used to achieve the following functions: The main path enhancement features and the residual path enhancement features are concatenated along the channel dimension, and a spatially aware gated map is generated using a convolutional layer and a sigmoid activation function, using the following formula: in, For gating graphs, It is the Sigmoid activation function. For convolution operations, Indicates main path enhancement features and residual path enhancement features The splicing result; The main path enhancement features and the residual path enhancement features are complementary and weighted element-wise fused using the gating graph to obtain the fused features, which are expressed by the following formula: in, As a feature of fusion, This indicates element-wise multiplication; The fused features are subjected to a hybrid convolution operation, and the results are fed back to the main path enhancement features in the form of residual learning to obtain the main path output features, which are expressed by the following formula: in, The main path output feature of the k-th efficient two-stream decoupling structure is given. This represents the scaling parameter. This indicates a hybrid convolution operation.

6. The method as described in claim 1, characterized in that, The PixelShuffle upsampling layer is used to implement the following functions; The input of the PixelShuffle upsampling layer is sequentially subjected to 3×3 ordinary convolution, PixelShuffle operation, GELU activation function, 3×3 ordinary convolution, and 3×3 ordinary convolution to obtain the convolution result; The input to the PixelShuffle upsampling layer is bilinearly interpolated and then summed with the convolution result to obtain the output of the PixelShuffle upsampling layer.

7. The method as described in claim 1, characterized in that, The training process employs a two-stage training strategy. In the first stage, the loss function used includes deep supervised loss. Edge-weighted Charbonnier pixel loss and structural similarity loss In the second phase, the loss employed also includes perceived loss. and combat losses .

8. The method as described in claim 7, characterized in that, The deep supervision loss is: in, To deeply supervise the losses, The number of efficient two-stream decoupling structures, For Charbonnier distance, For lightweight decoders, The residual path output characteristics of the k-th efficient two-stream decoupling structure are given. Ground truth value after spatial downsampling; The edge-weighted Charbonnier pixel loss is: in, For edge-weighted Charbonnier pixel loss, For the height of the image, The width of the image, For the number of channels, For the generated super-resolution image, For ground truth, It is the stability constant. The edge reinforcement strength coefficient, This is a spatial weight map extracted from the gradient of the ground truth image; The structural similarity loss is: in, For structural similarity loss, For structural similarity calculation, For the generated super-resolution image, This is the ground truth value.

9. A spatially-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction device, characterized in that, include: A network construction module is used to construct a super-resolution reconstruction network, which includes a generator and a discriminator; the generator includes a shallow feature extraction module, multiple sequentially connected efficient two-stream decoupling structures, and a PixelShuffle upsampling layer; The shallow feature extraction module is used to perform convolution processing on the input to obtain shallow features; Each of the efficient dual-stream decoupling structures includes a main path feature mixing module, a residual path feature mixing module, and an adaptive gated fusion module arranged in parallel; the main path feature mixing module and the residual path feature mixing module are used to enhance the input features to obtain main path enhanced features and residual path enhanced features, respectively; The main path enhancement feature and the residual path enhancement feature are fused by the adaptive gated fusion module to obtain the main path output feature; the residual path enhancement feature is used as the residual path output feature; the main path output feature and the residual path output feature of the previous efficient two-stream decoupling structure are used as the input of the next efficient two-stream decoupling structure; the input of the first efficient two-stream decoupling structure is the shallow feature; The main path output features of all efficient two-stream decoupling structures are accumulated with the shallow features and input into the PixelShuffle upsampling layer to obtain upsampled features; the input is then accumulated with the upsampled features after bilinear interpolation to obtain the generator output. The network training module is used to train the super-resolution reconstruction network based on the training dataset to obtain the trained generator; During training, the generator reconstructs low-resolution FGS images into super-resolution images, and the discriminator is used to distinguish the super-resolution images from real high-resolution images. The trained generator is obtained through adversarial training. The reconstruction module is used to input the low-resolution FGS image to be processed into the trained generator to obtain a super-resolution image.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the spatial-frequency decoupled fluorescence-guided surgical image super-resolution reconstruction method according to any one of claims 1-8.

Images