Ovarian cancer intraoperative navigation method and system based on machine vision

Abstract

The invention discloses an ovarian cancer intraoperative navigation method and system based on machine vision. The method comprises the following steps: 1, obtaining a fluorescence signal; 2, obtaining a white light image and a fluorescence image; 3, obtaining a fluorescence correction image and a white light reliable region mask; 4, inputting the fluorescence correction image into the improved SAM2 model for tumor segmentation, and outputting a tumor area mask and a tumor confidence map; 5, generating a white light structure diagram and a fluorescent light structure diagram; 6, extracting a white light key point set and a fluorescent light key point set; step 7, key point matching is carried out by using a bidirectional consistency constraint rule of a SuperGlue algorithm, and a registration mapping relation from the fluorescent image to the white light image is generated; and 8, mapping the tumor area mask and the tumor confidence map to a white light image coordinate system to generate an intraoperative navigation image. According to the method, the improved SAM2 model and the SuperGlue algorithm are combined, so that precise navigation in the ovarian cancer operation is realized.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a machine vision-based intraoperative navigation method and system for ovarian cancer. Background Technology

[0002] With the increasing demand for precision surgery and real-time intraoperative navigation, high-precision localization and boundary recognition technologies for gynecological malignancies such as ovarian cancer have received widespread attention. Current intraoperative assessment of ovarian cancer primarily relies on the subjective experience of medical staff in judging tissue color, shape, and texture under white light imaging, sometimes supplemented by traditional near-infrared fluorescence imaging. However, in practical applications, the following problems commonly exist: Intraoperative white light and fluorescence images are usually acquired by different imaging systems. Their visual axes are not completely consistent and they lack a strict time synchronization mechanism, resulting in deviations in spatial location and time dimension between the two types of images. This affects the accurate superposition of tumor region information in the white light image. Fluorescence signals are easily affected by tissue scattering, blood absorption, and instrument occlusion in complex surgical field environments, resulting in significant fluctuations in signal intensity and low signal-to-noise ratio. Traditional simple filtering and background subtraction methods are difficult to effectively suppress noise while preserving detail information, leading to unstable tumor region segmentation results. For the spatial registration problem of cross-modal images, existing methods mostly rely on manual point selection or matching algorithms based on gray-level similarity. When the imaging mechanisms of white light and fluorescence are significantly different, mismatches are prone to occur, resulting in insufficient registration accuracy. At the same time, traditional segmentation networks are mostly based on single-frame images for inference and lack the ability to model the temporal consistency of continuous intraoperative images. Under changes in viewpoint, tissue deformation, and local reflection interference, segmentation results are prone to jumps, affecting the continuity and reliability of navigation images.

[0003] Therefore, how to provide a machine vision-based intraoperative navigation method and system for ovarian cancer is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a machine vision-based intraoperative navigation method and system for ovarian cancer. This invention combines an improved SAM2 model and the SuperGlue algorithm to construct a multimodal image collaborative processing flow, achieving accurate segmentation and cross-modal spatial registration of the tumor region during ovarian cancer surgery, and completing a stable mapping of fluorescence information to the white light field of view. It has the advantages of high segmentation stability, high registration accuracy, and intuitive and reliable navigation display, which can effectively improve the accuracy of intraoperative identification and the safety of operation.

[0005] A machine vision-based intraoperative navigation method for ovarian cancer according to an embodiment of the present invention includes the following steps: Step 1: Before ovarian cancer surgery, inject tumor microenvironment-responsive NIR-IIb composite nanoprobes into the patient to obtain fluorescence signals; Step 2: During the ovarian cancer surgery, simultaneously acquire white light images and fluorescence images corresponding to the fluorescence signals at the same time point to obtain white light images and fluorescence images; Step 3: Normalize and perform discrete wavelet transform on the fluorescence image to obtain a fluorescence-corrected image, and perform reliable region identification on the white light image to obtain a reliable region mask for white light; Step 4: Input the fluorescence-corrected image into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. Step 5: Generate a white light structure map based on the white light image and the white light reliable region mask, and generate a fluorescence structure map based on the fluorescence correction image, the tumor region mask, and the tumor confidence map; Step 6: Extract the set of white light key points based on the white light structure diagram, and extract the set of fluorescence key points based on the fluorescence structure diagram; Step 7: Based on the white light keypoint set and the fluorescence keypoint set, use the bidirectional consistency constraint rule of the SuperGlue algorithm to perform keypoint matching and generate the registration mapping relationship from the fluorescence image to the white light image; Step 8: Based on the registration mapping relationship from fluorescence image to white light image, map the tumor region mask and tumor confidence map to the white light image coordinate system to generate intraoperative navigation image.

[0006] Optionally, step one specifically includes: Injecting tumor microenvironment-responsive NIR-IIb composite nanoprobes into patients before ovarian cancer surgery; The NIR-IIb composite nanoprobe circulates and accumulates in the ovarian cancer tumor tissue region in vivo; Under the influence of the glutathione-enriched microenvironment in the tumor tissue region compared to the normal tissue region, the NIR-IIb composite nanoprobe undergoes structural and energy level changes, enabling it to regain fluorescence emission capability in the tumor tissue region and emit a fluorescence signal with a wavelength of 1650 nm.

[0007] Optionally, step two specifically includes: During ovarian cancer surgery, the white light imaging device and the fluorescence imaging device are set under the same surgical visual axis, and a spatial correspondence is established between the white light imaging device and the fluorescence imaging device so that the white light imaging device and the fluorescence imaging device can image the same surgical field area. A unified external trigger signal source is set up, and trigger signals are sent simultaneously to the white light imaging device and the fluorescence imaging device by the external trigger signal source in each acquisition cycle. Upon receiving the trigger signal, the white light imaging device acquires a white light image at the corresponding moment; Upon receiving the trigger signal, the fluorescence imaging device acquires the fluorescence signal and generates a fluorescence image by configuring an optical filter component that matches the wavelength of 1650nm. The same time stamp is assigned to the white light image and fluorescence image acquired by the same trigger signal to obtain the white light image and fluorescence image corresponding to the time stamp.

[0008] Optionally, step three specifically includes: The fluorescence image is converted into a single-channel grayscale image, and the pixel values ​​of the single-channel grayscale image are linearly normalized to obtain a normalized fluorescence image. Perform discrete wavelet transform on the normalized fluorescence image to decompose the normalized fluorescence image into a low-frequency approximate subband and multiple high-frequency detail subbands; For each high-frequency detail sub-band, calculate the absolute value of all wavelet coefficients in the high-frequency detail sub-band, and obtain the median of the absolute values ​​of the wavelet coefficients; Calculate the absolute deviation between the absolute value and the median of the wavelet coefficients, and obtain the median of the absolute deviation as the denoising threshold of the high-frequency detail subband; Wavelet coefficients whose absolute values ​​are less than the denoising threshold in the high-frequency detail subband are set to zero, while wavelet coefficients whose absolute values ​​are greater than or equal to the denoising threshold are retained. Perform inverse wavelet transform based on the processed wavelet coefficients to reconstruct the wavelet-denoised fluorescence image; In the fluorescence image after wavelet denoising, according to the set size of the sliding window, the sum of all pixel values ​​in each sliding window is divided by the total number of pixels in the sliding window to obtain the mean local background intensity of the corresponding sliding window. For each pixel in the wavelet-denoised fluorescence image, the pixel value is subtracted from the mean local background intensity corresponding to the sliding window where the pixel is located to obtain the corrected pixel value; When the difference result is less than zero, the corresponding corrected pixel value is set to zero, and a fluorescence-corrected image is generated based on all corrected pixel values. The brightness channel of the white light image is extracted, and pixels whose pixel values ​​exceed a set brightness threshold are marked as bright areas. Calculate the pixel gradient magnitude for the white light image, and mark pixels with gradient magnitudes lower than a set gradient threshold as low-texture regions; The bright areas and low-texture areas are merged at the pixel level to generate a set of unreliable region pixels. Pixels in the unreliable region pixel set are assigned a value of 0, and the remaining pixels are assigned a value of 1 to generate a white light reliable region mask, where pixels with a value of 0 represent unreliable regions and pixels with a value of 1 represent reliable regions.

[0009] Optionally, step four specifically includes: The fluorescence-corrected image is input into the encoder, which includes a multi-layer two-dimensional convolutional unit and a downsampling unit. The two-dimensional convolutional unit uses a 3×3 convolutional kernel and combines it with a convolutional operation with a stride of 1. The downsampling unit uses a convolutional operation with a stride of 2 to output feature maps at multiple resolution levels, forming the feature set of the current frame. The current frame feature set is input into a hierarchical memory stack, which includes a short-term memory layer and a long-term memory layer; the short-term memory layer stores the current frame feature set for a set number of frames in a first-in-first-out manner. For the current frame feature set corresponding to the same spatial location, calculate the feature mean along the time dimension, and calculate the element-wise difference between each time frame feature and the feature mean; When the element-wise difference is less than the set change threshold in multiple consecutive time frames, the feature mean of the corresponding spatial location is written into the long-term memory layer. Align the feature set stored in the long-term memory layer with the feature set of the current frame in spatial position, and combine them to form a memory feature set; Input the memory feature set into the bootstrap cue loop. Calculate the numerical variance of the corresponding feature in consecutive time frames for each spatial location in the memory feature set. When the numerical variance is less than the set variance threshold, set the value of the corresponding spatial location to 1; otherwise, set the value to 0 to generate a stable location map. The sum of absolute values ​​of features along the channel dimension is calculated for the memory feature set at each spatial location to obtain the response map; the response map is multiplied element-wise with the stable location map and min-max normalization is performed to obtain the cue weight map; The prompt weight map is copied and expanded along the channel dimension to a weight set with the same number of channels as the memory feature set, and the weight set is multiplied element-wise with the memory feature set to obtain the prompt enhancement feature set; The set of cue-enhanced features is input into the dual-domain representation decoder, which includes a region decoding branch and a structural response decoding branch. The region decoding branch includes a multi-layer deconvolution unit, which uses a 4×4 convolution kernel and an upsampling operation with a stride of 2, and outputs the region prediction value through a 1×1 convolution kernel. Pixels whose predicted regional values ​​are greater than or equal to the set prediction threshold are assigned a value of 1, and those that are not assigned a value of 0, thus obtaining the tumor region mask. The structural response decoding branch includes a 3×3 convolutional unit and a gradient enhancement unit. The gradient enhancement unit performs difference operations on the cue enhancement feature set in the horizontal and vertical directions respectively, and outputs boundary prediction values ​​through a 1×1 convolutional kernel. The region prediction value and the boundary prediction value are concatenated along the channel dimension to generate a joint prediction feature; The joint prediction features are calculated using Softmax normalization in the channel dimension to obtain the tumor confidence map.

[0010] Optionally, step five specifically includes: The white light image is converted to grayscale to obtain a white light grayscale image; The white light grayscale image is multiplied pixel by pixel with the white light reliable region mask to obtain the white light effective region image; Calculate the pixel gradient magnitude for the effective white light region image and generate a white light gradient response map; Based on the white light gradient response map, the gradient magnitude of each pixel position is compared with a set white light gradient threshold. When the pixel gradient magnitude is greater than or equal to the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set to the gradient magnitude. When the pixel gradient magnitude is less than the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set to 0. The white light structure map is composed of the values ​​of all pixel positions. The fluorescence-corrected image is multiplied pixel by pixel with the tumor region mask to obtain the fluorescence image after the tumor region mask is applied. The fluorescence image after the tumor region is masked is multiplied pixel by pixel with the tumor confidence map, and the pixel values ​​are weighted according to the corresponding tumor confidence level to obtain a confidence-weighted fluorescence image. Calculate the pixel gradient magnitude for the confidence-weighted fluorescence image and generate a fluorescence gradient response map; Based on the fluorescence gradient response map, the gradient magnitude of each pixel position is compared with a set fluorescence gradient threshold. When the pixel gradient magnitude is greater than or equal to the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set to the gradient magnitude. When the pixel gradient magnitude is less than the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set to 0. The fluorescence structure map is composed of the values ​​of all pixel positions.

[0011] Optionally, step six specifically includes: In the white light structure diagram, a local neighborhood of size 5×5 is constructed with each non-zero pixel position as the center; Compare the values ​​of each pixel position within the local neighborhood. When the value of the center pixel position is greater than the values ​​of the other pixel positions within the local neighborhood, it is determined that the center pixel position satisfies the local maximum condition, and the center pixel position is marked as a white light key point. By iterating through all pixel locations that satisfy the local maximum condition, a set of white light keypoints is obtained. In the fluorescence structure diagram, the same 5×5 local neighborhood as the white light structure diagram is used to traverse the non-zero pixel positions and select all pixel positions that satisfy the local maximum condition as fluorescence key points to obtain the fluorescence key point set.

[0012] Optionally, step seven specifically includes: Taking each key point in the white light key point set and the fluorescence key point set as the center, the pixel values ​​in the 5×5 local neighborhood are read in the white light structure map and the fluorescence structure map respectively in the order from the top left corner to the bottom right corner and in row priority, and arranged in order to form a one-dimensional numerical sequence, which serves as the feature vector of the corresponding key point. The feature vectors corresponding to the white light keypoint set and the fluorescent keypoint set are input into the SuperGlue algorithm. Element-wise multiplication is performed on each feature vector in the white light keypoint set and each feature vector in the fluorescent keypoint set, and the multiplication results are summed to obtain the corresponding matching score. A matching score matrix is ​​constructed based on the matching scores between all white light keypoints and fluorescent keypoints. Each matching score in the matching score matrix is ​​divided by the sum of the matching scores in its row to obtain a row-normalized matching score matrix. Each matching score in the row-normalized matching score matrix is ​​then divided by the sum of the matching scores in its column to obtain a normalized matching score matrix. In the normalized matching score matrix, the bidirectional consistency constraint rule of the SuperGlue algorithm is used for matching and filtering. The bidirectional consistency constraint rule is as follows: for any key point in the white light key point set, if the corresponding fluorescent key point has the maximum normalized matching score in the row corresponding to the white light key point, and the white light key point also has the maximum normalized matching score in the column corresponding to the fluorescent key point, then it is determined that the white light key point and the fluorescent key point constitute a matching pair. The key point correspondences that satisfy the bidirectional consistency constraint rules are traversed to obtain cross-modal key point matching results. Based on the cross-modal keypoint matching results, the pixel coordinates of each matching pair in the white light image coordinate system and the fluorescence image coordinate system are extracted. For each matching pair of pixel coordinates, the corresponding coordinate difference is calculated, and the coordinate differences are summed and averaged to obtain the translation parameter between the white light image coordinate system and the fluorescence image coordinate system. Divide the Euclidean distance of each matching pair in the white light image coordinate system by the Euclidean distance in the fluorescence image coordinate system to obtain the corresponding distance ratio. The distance ratios of all matching pairs are summed, and the summation is divided by the number of matching pairs to obtain the scale parameter between the white light image coordinate system and the fluorescence image coordinate system. The scale parameter is applied to the pixel coordinates in the fluorescence image coordinate system to perform a scale transformation on the pixel coordinates in the fluorescence image. The translation parameter is added to the pixel coordinates of the fluorescence image after scaling, so that the pixel coordinates in the fluorescence image are aligned with the coordinate system of the white light image, thereby generating a registration mapping relationship from the fluorescence image to the white light image.

[0013] Optionally, step eight specifically includes: Based on the registration mapping relationship from fluorescence image to white light image, for each pixel position in the tumor region mask, the pixel coordinates in the fluorescence image coordinate system are extracted, and scale transformation and translation transformation are performed sequentially according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor region mask and the white light image are added pixel by pixel at corresponding coordinate positions to obtain the tumor region overlay image; For each pixel position in the tumor confidence map, the pixel coordinates in the fluorescence image coordinate system are extracted, and scale transformation and translation transformation are performed according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor confidence map and the tumor region overlay map are added pixel by pixel at the corresponding coordinate positions. When the sum is greater than the maximum gray value, the pixel value is set to the maximum gray value to generate an intraoperative navigation image.

[0014] An intraoperative navigation system for ovarian cancer based on machine vision according to an embodiment of the present invention includes the following modules: A dual-channel image acquisition module is used to simultaneously acquire white light images and fluorescence images during ovarian cancer surgery using a white light imaging device and a fluorescence imaging device set under the same surgical visual axis, and to assign the same time stamp to the white light images and fluorescence images. The image preprocessing module is used to normalize and perform discrete wavelet transform on the fluorescence image to obtain a fluorescence-corrected image, and to perform reliable region identification on the white light image to obtain a reliable region mask for white light. The tumor segmentation module is used to input the fluorescence-corrected image into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. The structure building module is used to generate a white light structure map based on the white light image and the white light reliable region mask, and to generate a fluorescence structure map based on the fluorescence correction image, the tumor region mask and the tumor confidence map; The key point extraction module is used to extract a set of white light key points based on the white light structure diagram and to extract a set of fluorescent key points based on the fluorescence structure diagram. The cross-modal matching and registration module is used to perform key point matching based on the white light key point set and the fluorescence key point set, using the SuperGlue algorithm to generate a registration mapping relationship from the fluorescence image to the white light image. The navigation image generation module is used to map the tumor region mask and tumor confidence map to the white light image coordinate system based on the registration mapping relationship between the fluorescence image and the white light image, thereby generating an intraoperative navigation image.

[0015] The beneficial effects of this invention are: This invention utilizes a dual-channel image acquisition module to synchronously acquire white light and fluorescence images under the same time signature. Combined with the image preprocessing module's normalization and discrete wavelet transform of the fluorescence image, and the construction of a reliable white light region mask for the white light image, it addresses the issues of intraoperative fluorescence signals being susceptible to background interference and the presence of reflective and low-texture areas in the white light image. This achieves unified constraints on multimodal images at both the temporal and spatial effective region levels. In the tumor segmentation stage, by improving the encoder, hierarchical memory stack, bootstrap cue loop, and dual-domain representation decoder sequentially connected in the SAM2 model, multi-layer convolutional feature extraction is performed on the fluorescence-corrected image. Short-term memory and long-term memory layers are introduced to determine the stability of continuous frame features, and the cue enhancement feature set is strengthened through a stable position map and a cue weight map. This effectively suppresses instantaneous noise caused by intraoperative viewpoint changes and instrument occlusion, improving the continuity and reliability of the tumor region mask and tumor confidence map in the temporal dimension. In the structure construction module, white light structure maps and fluorescence... The unified structural representation of the structural map transforms tissue structure information from different imaging modalities into comparable gradient response forms. In the keypoint extraction module, white light keypoint sets and fluorescence keypoint sets are extracted based on the local maxima condition of a 5×5 local neighborhood, providing stable structural anchors for cross-modal matching. In the cross-modal matching and registration module, the bidirectional consistency constraint rule of the SuperGlue algorithm is used to match and filter keypoint feature vectors, and translation and scale parameters are calculated to generate a registration mapping relationship from fluorescence image to white light image, achieving high-precision spatial alignment without external calibration devices. Finally, in the navigation image generation module, the tumor region mask and tumor confidence map are mapped to the white light image coordinate system and superimposed pixel by pixel. While maintaining the integrity of the original anatomical structure information, tumor location information and confidence information are superimposed to generate intraoperative navigation images, thereby realizing real-time visualization, precise spatial positioning, and quantitative expression of the tumor region during ovarian cancer surgery, significantly improving the accuracy and safety of navigation in complex surgical field environments. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a machine vision-based intraoperative navigation method for ovarian cancer proposed in this invention. Figure 2 This is a schematic diagram of the structure of an intraoperative navigation system for ovarian cancer based on machine vision proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figure 1 A machine vision-based intraoperative navigation method for ovarian cancer includes the following steps: Step 1: Before ovarian cancer surgery, inject tumor microenvironment-responsive NIR-IIb composite nanoprobes into the patient to obtain fluorescence signals; Step 2: During ovarian cancer surgery, simultaneously acquire white light images and fluorescence images corresponding to the fluorescence signals at the same time point to obtain white light images and fluorescence images; Step 3: Normalize and perform discrete wavelet transform on the fluorescence image to obtain the fluorescence-corrected image, and perform reliable region identification on the white light image to obtain a reliable region mask for white light; Step 4: Input the fluorescence-corrected image into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. Step 5: Generate a white light structure map based on the white light image and the white light reliable region mask, and generate a fluorescence structure map based on the fluorescence-corrected image, the tumor region mask, and the tumor confidence map; Step 6: Extract the set of white light key points based on the white light structure map, and extract the set of fluorescence key points based on the fluorescence structure map; Step 7: Based on the white light keypoint set and the fluorescence keypoint set, use the bidirectional consistency constraint rule of the SuperGlue algorithm to perform keypoint matching and generate the registration mapping relationship from the fluorescence image to the white light image; Step 8: Based on the registration mapping relationship from fluorescence image to white light image, map the tumor region mask and tumor confidence map to the white light image coordinate system to generate intraoperative navigation image.

[0019] In this embodiment, step one specifically includes: Injecting tumor microenvironment-responsive NIR-IIb composite nanoprobes into patients before ovarian cancer surgery; The NIR-IIb composite nanoprobe circulates and accumulates in the ovarian cancer tumor tissue region in vivo; Under the influence of the glutathione-enriched microenvironment in the tumor tissue region compared to the normal tissue region, the NIR-IIb composite nanoprobe undergoes structural and energy level changes, enabling the NIR-IIb composite nanoprobe to regain its fluorescence emission capability in the tumor tissue region and emit a fluorescence signal with a wavelength of 1650 nm. In this invention, a tumor microenvironment-responsive NIR-IIb composite nanoprobe achieves highly specific imaging by responding to the characteristic biochemical environment of ovarian cancer tumor tissue. The glutathione concentration in ovarian cancer tumor tissue is significantly higher than that in normal tissue areas. In this high glutathione concentration microenvironment, the composite nanoprobe undergoes controlled structural and energy level changes, transforming from a low fluorescence state to a stable fluorescence emission state. This generates a fluorescence signal with a wavelength of 1650 nm in the tumor tissue area. This fluorescence signal is located in the NIR-IIb band and has the characteristics of low tissue scattering, large penetration depth, and low background interference. It can provide tumor-related optical information with a high signal-to-noise ratio in complex intraoperative environments, providing a reliable data foundation for tumor segmentation, spatial registration, and intraoperative navigation.

[0020] In this embodiment, step two specifically includes: During ovarian cancer surgery, the white light imaging device and the fluorescence imaging device are set under the same surgical visual axis, and a spatial correspondence is established between the white light imaging device and the fluorescence imaging device so that the white light imaging device and the fluorescence imaging device can image the same surgical field area. Set up a unified external trigger signal source, and send trigger signals to both the white light imaging device and the fluorescence imaging device simultaneously during each acquisition cycle. Upon receiving a trigger signal, the white light imaging device acquires a white light image at the corresponding moment. Upon receiving a trigger signal, the fluorescence imaging device acquires fluorescence signals and generates fluorescence images by configuring an optical filter component that matches the wavelength of 1650nm. By assigning the same time stamp to white light images and fluorescence images acquired from the same trigger signal, the white light images and fluorescence images corresponding to the time stamps are obtained. In this invention, by placing the white light imaging device and the fluorescence imaging device under the same surgical visual axis during ovarian cancer surgery and establishing a stable spatial correspondence between them in advance, it is possible to ensure that the white light image and the fluorescence image originate from the same surgical field area, avoiding spatial deviations introduced by differences in viewing angle. At the same time, a unified external trigger signal source is used to synchronously trigger the two imaging devices, ensuring that the white light image and the fluorescence image maintain consistency in the temporal dimension, eliminating registration errors caused by inconsistent acquisition timing from the source. By configuring an optical filter component in the fluorescence imaging device that matches the 1650nm wavelength, stable acquisition of the target fluorescence signal is achieved, which helps to suppress interference from non-target bands and provides a high-quality input data foundation with temporal and spatial consistency for tumor segmentation, cross-modal registration, and intraoperative navigation.

[0021] In this embodiment, step three specifically includes: The fluorescence image is converted into a single-channel grayscale image, and the pixel values ​​of the single-channel grayscale image are linearly normalized to obtain a normalized fluorescence image. Perform discrete wavelet transform on the normalized fluorescence image to decompose the normalized fluorescence image into a low-frequency approximate subband and multiple high-frequency detail subbands; For each high-frequency detail subband, calculate the absolute value of all wavelet coefficients in the high-frequency detail subband, and then find the median of the absolute values ​​of the wavelet coefficients. Calculate the absolute deviation between the absolute value and the median of the wavelet coefficients, and obtain the median of the absolute deviation as the denoising threshold for the high-frequency detail subband. Wavelet coefficients whose absolute values ​​are less than the denoising threshold in the high-frequency detail subband are set to zero, while wavelet coefficients whose absolute values ​​are greater than or equal to the denoising threshold are retained. Perform inverse wavelet transform based on the processed wavelet coefficients to reconstruct the wavelet-denoised fluorescence image; In the fluorescence image after wavelet denoising, according to the set size of the sliding window, the sum of all pixel values ​​in each sliding window is divided by the total number of pixels in the sliding window to obtain the mean local background intensity of the corresponding sliding window. For each pixel in the fluorescence image after wavelet denoising, the pixel value is subtracted from the mean local background intensity corresponding to the sliding window where the pixel is located to obtain the corrected pixel value; When the difference result is less than zero, the corresponding corrected pixel value is set to zero, and a fluorescence-corrected image is generated based on all corrected pixel values. Extract the brightness channel from the white light image and mark the pixels in the brightness channel whose pixel values ​​exceed the set brightness threshold as bright areas; Calculate the pixel gradient magnitude for the white light image and mark pixels with gradient magnitudes below a set gradient threshold as low-texture regions. The highlight areas and low-texture areas are merged at the pixel level to generate a set of unreliable region pixels. Pixels in the unreliable region pixel set are assigned a value of 0, and the rest are assigned a value of 1 to generate a white light reliable region mask, where pixels with a value of 0 represent unreliable regions and pixels with a value of 1 represent reliable regions. This invention introduces a multi-scale denoising and background correction mechanism based on discrete wavelet transform in the fluorescence image preprocessing stage, effectively improving the signal-to-noise ratio and stability of intraoperative fluorescence imaging. Specifically, the fluorescence image is first linearly normalized, mapping pixel values ​​to the range of 0 to 1 to eliminate differences in imaging intensity at different acquisition times. Then, discrete wavelet transform is used to decompose the normalized fluorescence image, with three wavelet decomposition layers to balance detail preservation and noise suppression. For each high-frequency detail subband, the denoising threshold is determined by calculating the median of the absolute values ​​of the wavelet coefficients and their median absolute deviation, achieving effective denoising of random noise. Effective suppression: During background correction, the sliding window size is set to 15×15 pixels. The local background intensity mean is obtained by calculating the arithmetic mean of the pixel values ​​within the window, and background subtraction is performed pixel by pixel. When the corrected pixel value is less than 0, it is uniformly truncated to 0, thereby avoiding negative value interference. On the white light image side, reliable region identification is achieved by extracting the brightness channel and calculating the gradient magnitude, and a binary reliable region mask is constructed using 0 and 1. Through the above processing, this invention effectively reduces the interference of high-brightness reflection, low-texture areas and fluorescent background in complex intraoperative environments, providing a stable and reliable data foundation for tumor segmentation and cross-modal registration.

[0022] In this embodiment, step four specifically includes: The fluorescence-corrected image is input into the encoder, which includes multi-layer two-dimensional convolutional units and downsampling units. The two-dimensional convolutional units use a 3×3 convolutional kernel and combine it with a convolutional operation with a stride of 1. The downsampling units use a convolutional operation with a stride of 2. The encoder outputs feature maps at multiple resolution levels to form the feature set of the current frame. The current frame feature set is input into the hierarchical memory stack, which includes a short-term memory layer and a long-term memory layer; the short-term memory layer stores the current frame feature set for a set number of frames in a first-in-first-out manner. For the current frame feature set corresponding to the same spatial location, calculate the feature mean along the time dimension, and calculate the element-wise difference between each time frame feature and the feature mean; When the element-wise difference is less than the set change threshold in multiple consecutive time frames, the feature mean of the corresponding spatial location is written into the long-term memory layer. Align the feature set stored in the long-term memory layer with the feature set of the current frame in spatial position, and combine them to form a memory feature set; Input the memory feature set into the bootstrap cue loop. Calculate the numerical variance of the corresponding feature in consecutive time frames for each spatial location in the memory feature set. When the numerical variance is less than the set variance threshold, set the value of the corresponding spatial location to 1; otherwise, set the value to 0 to generate a stable location map. The sum of absolute values ​​of features along the channel dimension is calculated for each spatial location of the memory feature set to obtain the response map; the response map is multiplied element-wise with the stable location map and min-max normalization is performed to obtain the cue weight map; The cue weight map is copied and expanded along the channel dimension to a weight set with the same number of channels as the memory feature set. The weight set and the memory feature set are then multiplied element by element to obtain the cue enhancement feature set. The set of cue-enhanced features is input into the dual-domain representation decoder, which includes a region decoding branch and a structural response decoding branch. The region decoding branch includes a multi-layer deconvolution unit, which uses a 4×4 convolution kernel and an upsampling operation with a stride of 2, and outputs the region prediction value through a 1×1 convolution kernel. Pixels whose predicted regional values ​​are greater than or equal to the set prediction threshold are assigned a value of 1, and those that are not assigned a value of 0, thus obtaining the tumor region mask. The structural response decoding branch includes a 3×3 convolutional unit and a gradient enhancement unit. The gradient enhancement unit performs difference operations on the cue enhancement feature set in the horizontal and vertical directions respectively, and outputs the boundary prediction value through a 1×1 convolutional kernel. The regional and boundary prediction values ​​are concatenated along the channel dimension to generate joint prediction features; The joint prediction features are calculated using Softmax normalization in the channel dimension to obtain the tumor confidence map. The improved SAM2 model used in this invention inherits the basic structure of existing SAM2 models in its overall network framework. Both models employ an encoder to perform multi-layer convolutional feature extraction on the input image and a decoder to output pixel-level segmentation results, ensuring basic performance consistency in segmentation accuracy and inference efficiency. Compared to existing SAM2 models, this invention addresses the problems of frequent viewpoint changes, significant tissue occlusion, and unstable fluorescence signals in intraoperative imaging of ovarian cancer by structurally improving the feature modeling method and segmentation output format.

[0023] Specifically, this invention sets up a hierarchical memory stack between the encoder and decoder to filter and integrate continuous frame features in the temporal dimension. The short-term memory layer stores the current frame feature set of the most recent 5 frames, covering a complete intraoperative visual field change process. The mean value of continuous frame features is calculated at the same spatial location, and the element-wise difference between each frame feature and the mean value is calculated. When the difference is less than 0.05 in 3 consecutive frames, the feature change at that spatial location is determined to be within a stable range, and the corresponding feature mean value is written into the long-term memory layer. In this way, stable tumor-related features that recur in multiple intraoperative images can be continuously retained, while unstable features caused by instrument occlusion, reflection, and transient noise are automatically excluded, thereby improving the consistency of segmentation results in the temporal dimension.

[0024] Building upon this foundation, the present invention further filters and strengthens the memory feature set through a bootstrap cue loop. It calculates the numerical variance of continuous time frame features for each spatial location and marks spatial locations with variances less than 0.02 as stable locations, forming a binary stable location map. This variance threshold matches the quantization noise level of the fluorescence imaging system, effectively distinguishing between real tissue structure changes and imaging noise fluctuations. Subsequently, the absolute values ​​of the memory feature set are summed along the channel dimension to obtain a response map. Under the constraint of the stable location map, minimum-maximum normalization is performed to generate a cue weight map, allowing stable and highly responsive regions to receive higher weights in subsequent decoding processes.

[0025] In the decoding stage, this invention employs a dual-domain representation decoder, simultaneously inputting the cue enhancement feature set into the region decoding branch and the structural response decoding branch. The region decoding branch recovers spatial resolution step by step through multi-layer deconvolution and outputs a 0 / 1 binary tumor region mask under the condition that the prediction threshold is fixed at 0.5, ensuring the spatial integrity of the segmentation result. The structural response decoding branch extracts the intensity of spatial changes through gradient enhancement operations, which is used to characterize the difference in structural response between the tumor region and its edge position. The outputs of the two branches are concatenated in the channel dimension and the tumor confidence map is obtained by calculating through Softmax normalization, so that the confidence of each pixel simultaneously reflects its regional prediction intensity and structural response consistency.

[0026] Through the above improvements, without increasing manual interaction or additional annotation input, the present invention enables the model to automatically enhance tumor features that stably appear in multiple frames of intraoperative images, and to quantify the confidence of uncertain regions in the segmentation results, effectively improving the stability and reliability of tumor region identification under small lesions, low-contrast regions and complex background conditions in intraoperative navigation scenarios for ovarian cancer.

[0027] In this embodiment, step five specifically includes: A white light image is converted to grayscale to obtain a white light grayscale image; The white light grayscale image is multiplied pixel by pixel with the white light reliable region mask to obtain the white light effective region image; Calculate the pixel gradient magnitude for the effective white light region image and generate a white light gradient response map; Based on the white light gradient response map, the gradient magnitude of each pixel position is compared with a set white light gradient threshold. When the pixel gradient magnitude is greater than or equal to the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set as the gradient magnitude; when the pixel gradient magnitude is less than the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set to 0. The white light structure map is composed of the values ​​of all pixel positions. The fluorescence-corrected image is multiplied pixel by pixel with the tumor region mask to obtain the fluorescence image after the tumor region mask is applied. The fluorescence image after the tumor region is masked is multiplied pixel by pixel with the tumor confidence map, and the pixel values ​​are weighted according to the corresponding tumor confidence level to obtain a confidence-weighted fluorescence image. Calculate the pixel gradient magnitude of the confidence-weighted fluorescence image to generate a fluorescence gradient response map; Based on the fluorescence gradient response map, the gradient magnitude of each pixel position is compared with a set fluorescence gradient threshold. When the pixel gradient magnitude is greater than or equal to the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set as the gradient magnitude; when the pixel gradient magnitude is less than the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set to 0. The fluorescence structure map is composed of the values ​​of all pixel positions. In this invention, by constructing structural maps for white light and fluorescence images respectively, a unified representation of multimodal images at the structural level is achieved. During the generation of the white light structural map, a reliable region mask is first used to spatially constrain the white light image, setting the pixel values ​​corresponding to unreliable regions to zero, and extracting structural information only within the reliable imaging region. In this invention, the pixel gradient magnitude is calculated by the difference in grayscale values ​​between adjacent pixels, with the pixel grayscale difference of the digital image serving as the measurement benchmark. Based on the gradient magnitude, the effective white light region image is processed. When the pixel gradient magnitude is greater than or equal to the white light gradient threshold of 20, the gradient magnitude of the corresponding pixel position is retained as the structural response. When the pixel gradient magnitude is less than the white light gradient threshold of 20, the value of the corresponding pixel position is set to zero, thereby highlighting the contours of the tissue's anatomical structure while suppressing background noise and non-structural information.

[0028] In the generation of fluorescence structure maps, the fluorescence-corrected image is first spatially constrained by a tumor region mask to avoid interference from fluorescence signals in non-tumor regions during structure extraction. Furthermore, the fluorescence signal is weighted pixel-by-pixel based on the tumor confidence map, enhancing structural information within high-confidence tumor regions. On this basis, pixel gradient magnitudes are calculated for the confidence-weighted fluorescence image, using pixel grayscale differences as the gradient metric. When the pixel gradient magnitude is greater than or equal to the fluorescence gradient threshold of 12, the gradient magnitude at the corresponding pixel location is retained as the fluorescence structure response; when the pixel gradient magnitude is less than the fluorescence gradient threshold of 12, the value at the corresponding pixel location is set to zero. The white light structure map and fluorescence structure map generated in this way maintain consistency in structural representation and retain continuous gradient response features within the structural region. This facilitates stable keypoint extraction and cross-modal keypoint matching based on the SuperGlue algorithm, thereby improving the accuracy and robustness of multimodal image registration during intraoperative navigation.

[0029] In this embodiment, step six specifically includes: In the white light structure diagram, a local neighborhood of size 5×5 is constructed with each non-zero pixel position as the center; Compare the values ​​of each pixel position within the local neighborhood. When the value of the center pixel position is greater than the values ​​of the other pixel positions within the local neighborhood, the center pixel position is determined to satisfy the local maximum condition, and the center pixel position is marked as the white light key point. By iterating through all pixel locations that satisfy the local maximum condition, a set of white light keypoints is obtained. In the fluorescence structure map, the same 5×5 local neighborhood as the white light structure map is used to traverse the non-zero pixel positions and select all pixel positions that satisfy the local maximum condition as fluorescence key points to obtain the fluorescence key point set. In this invention, the key point extraction process is directly based on the structural response intensity information already retained in the white light structure map and the fluorescence structure map, without performing additional feature transformations on the original image or structure map, thus ensuring the stability and consistency of key point localization. By constructing a local neighborhood of fixed size 5×5 at non-zero pixel locations and using the local maximum condition to determine the center pixel, the extracted key points are all located at the positions with the most significant local structural responses, effectively avoiding the misselection of weak response areas and noise areas as key points. Since the white light structure map and the fluorescence structure map are consistent in structural representation and key point determination rules, the obtained white light key point set and fluorescence key point set have a good correspondence in spatial distribution characteristics, which is conducive to the stable execution of the cross-modal key point matching process, thereby improving the accuracy and robustness of the registration results between the white light imaging modality and the fluorescence imaging modality.

[0030] In this embodiment, step seven specifically includes: Taking each key point in the white light key point set and the fluorescence key point set as the center, the pixel values ​​in the 5×5 local neighborhood are read in the white light structure map and the fluorescence structure map respectively in the order from the top left corner to the bottom right corner and in row priority, and arranged in order to form a one-dimensional numerical sequence, which serves as the feature vector of the corresponding key point. The feature vectors corresponding to the white light keypoint set and the fluorescent keypoint set are input into the SuperGlue algorithm. Element-wise multiplication is performed on each feature vector in the white light keypoint set and each feature vector in the fluorescent keypoint set, and the multiplication results are summed to obtain the corresponding matching score. A matching score matrix is ​​constructed based on the matching scores between all white light keypoints and fluorescent keypoints. Each matching score in the matching score matrix is ​​divided by the sum of the matching scores in its row to obtain a row-normalized matching score matrix. Each matching score in the row-normalized matching score matrix is ​​then divided by the sum of the matching scores in its column to obtain a normalized matching score matrix. In the normalized matching score matrix, the bidirectional consistency constraint rule of the SuperGlue algorithm is used for matching and filtering. The bidirectional consistency constraint rule is as follows: for any key point in the white light key point set, if the corresponding fluorescent key point has the maximum normalized matching score in the row corresponding to the white light key point, and the white light key point also has the maximum normalized matching score in the column corresponding to the fluorescent key point, then the white light key point and the fluorescent key point are determined to form a matching pair. The key point correspondences that satisfy the bidirectional consistency constraint rules are traversed to obtain the cross-modal key point matching results. Based on the cross-modal keypoint matching results, the pixel coordinates of each matching pair in the white light image coordinate system and the fluorescence image coordinate system are extracted; Calculate the corresponding coordinate difference for the pixel coordinates of all matching pairs, sum the coordinate differences and take the average to obtain the translation parameter between the white light image coordinate system and the fluorescence image coordinate system; Divide the Euclidean distance of each matching pair in the white light image coordinate system by the Euclidean distance in the fluorescence image coordinate system to obtain the corresponding distance ratio. The distance ratios of all matching pairs are summed, and the summation is divided by the number of matching pairs to obtain the scale parameter between the white light image coordinate system and the fluorescence image coordinate system. The scale parameter is applied to the pixel coordinates in the fluorescence image coordinate system to perform a scale transformation on the pixel coordinates in the fluorescence image. The translation parameter is added to the pixel coordinates of the fluorescence image after scaling, so that the pixel coordinates in the fluorescence image are aligned with the coordinate system of the white light image, thus generating a registration mapping relationship from the fluorescence image to the white light image. In the cross-modal keypoint matching process, this invention constructs a unified feature representation based on keypoint sets extracted from white light and fluorescence structure maps. By reading pixel values ​​in a fixed order within the local neighborhood of each keypoint to form feature vectors, structural information from different imaging modalities has a consistent expression in numerical space. The SuperGlue algorithm is used to perform element-wise multiplication and summation of the two types of keypoint feature vectors to construct a matching score matrix. Row and column normalization are combined to constrain the matching score, ensuring bidirectional consistency of the matching relationship between the white light and fluorescence keypoint sets. This effectively suppresses the influence of locally similar structures on matching results under cross-modal conditions. The invention addresses the mismatch caused by structural errors. After obtaining stable cross-modal keypoint matching results, the translation parameter is calculated by statistically analyzing the difference between the pixel coordinates of the matched keypoint pairs in the two coordinate systems. The scale parameter is obtained by averaging the pixel distance ratios between the matched keypoint pairs. When generating the registration mapping relationship, the scale transformation is first performed on the fluorescence image coordinates, and then the translation parameter is superimposed to achieve accurate alignment of the fluorescence image to the white light image coordinate system. Through the above method, the present invention achieves high-precision spatial registration between intraoperative white light imaging and fluorescence imaging without relying on external calibration devices, providing a reliable foundation for stable mapping and visual navigation of tumor region information in white light images.

[0031] In this embodiment, step eight specifically includes: Based on the registration mapping relationship from fluorescence image to white light image, for each pixel position in the tumor region mask, the pixel coordinates in the fluorescence image coordinate system are extracted, and scale transformation and translation transformation are performed sequentially according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor region mask and the white light image are added pixel by pixel at corresponding coordinate positions to obtain the tumor region overlay image; For each pixel location in the tumor confidence map, the pixel coordinates in the fluorescence image coordinate system are extracted, and scaling and translation transformations are performed according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor confidence map and the tumor region overlay map are added pixel by pixel at the corresponding coordinate positions. When the sum is greater than the maximum gray value, the pixel value is set to the maximum gray value to generate an intraoperative navigation image. After obtaining the registration mapping relationship from the fluorescence image to the white light image, this invention performs scale transformation and translation transformation on the tumor region mask and the tumor confidence map respectively, mapping them pixel by pixel to the white light image coordinate system to achieve spatial correspondence. By performing pixel-by-pixel addition on the mapped tumor region mask and the white light image, a tumor region overlay image is obtained, which enhances the display of the tumor region while maintaining the original anatomical structure information. At the same time, the mapped tumor confidence map is added pixel by pixel to the tumor region overlay image again, and truncation is performed when the pixel value exceeds the maximum gray value of 255 of the 8-bit grayscale image to ensure that the pixel value range of the output image is stable between 0 and 255. While preserving the integrity of the white light anatomical structure, tumor location information and confidence information are superimposed, improving the readability and safety of the intraoperative navigation image.

[0032] refer to Figure 2 A machine vision-based intraoperative navigation system for ovarian cancer includes the following modules: The dual-channel image acquisition module is used to simultaneously acquire white light and fluorescence images during ovarian cancer surgery using a white light imaging device and a fluorescence imaging device set under the same surgical visual axis, and to assign the same time stamp to the white light and fluorescence images. The image preprocessing module is used to normalize and perform discrete wavelet transform on the fluorescence image to obtain the fluorescence-corrected image, and to perform reliable region identification on the white light image to obtain a reliable region mask for the white light. The tumor segmentation module is used to input fluorescence-corrected images into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. The structure building module is used to generate white light structure maps based on white light images and white light reliable region masks, and to generate fluorescence structure maps based on fluorescence-corrected images, tumor region masks, and tumor confidence maps. The key point extraction module is used to extract a set of white light key points based on the white light structure map and to extract a set of fluorescent key points based on the fluorescence structure map. The cross-modal matching and registration module is used to perform keypoint matching based on the white light keypoint set and the fluorescence keypoint set, and generate the registration mapping relationship from the fluorescence image to the white light image. The navigation image generation module is used to map the tumor region mask and tumor confidence map to the white light image coordinate system based on the registration mapping relationship between the fluorescence image and the white light image, thereby generating intraoperative navigation images.

[0033] Example 1: To verify the feasibility of this invention in practice, it was applied to a laparoscopic ovarian cancer surgery scenario performed at a tertiary hospital's gynecologic oncology center, to verify intraoperative navigation-assisted surgery for patients undergoing standard debulking procedures. In this scenario, the surgical field was complex, with blood covering the abdominal tissue surface, instrument reflections, and mesenteric obstruction. Fluorescence signals were easily affected by tissue scattering and background noise. Scale variations and viewing angle disturbances existed between the white light and fluorescence images. Traditional single-modal observation methods struggled to accurately identify tiny lesions less than 3 mm in diameter, and medical staff relied on experience to judge diffuse implantation metastasis areas, leading to risks of missed or over-resection.

[0034] In practical applications, tumor microenvironment-responsive NIR-IIb composite nanoprobes are injected into patients before surgery, causing them to emit fluorescence signals with a wavelength of 1650 nm in the tumor tissue area. During surgery, white light images and fluorescence images are simultaneously acquired under the same surgical visual axis through a dual-channel image acquisition module, and both are assigned a unified time marker to ensure spatial and temporal consistency.

[0035] The image preprocessing module performs linear normalization on the fluorescence image and decomposes it into low-frequency approximate subbands and high-frequency detail subbands using discrete wavelet transform. After noise suppression, the fluorescence image is reconstructed to obtain the wavelet-denoised fluorescence image. Subsequently, the mean local background intensity is calculated using a fixed-size sliding window, and pixel-by-pixel subtraction is performed to generate a fluorescence-corrected image. Simultaneously, the brightness channel of the white light image is extracted and gradient magnitude is calculated. High-brightness and low-texture regions are marked as unreliable regions, generating a reliable white light region mask, effectively reducing the impact of reflective areas and background noise on subsequent segmentation and matching.

[0036] The tumor segmentation module inputs the fluorescence-corrected image into the improved SAM2 model. The encoder extracts multi-scale feature maps through multi-layer 3×3 convolutions and downsampling operations with a stride of 2, forming the feature set of the current frame; a hierarchical memory stack stores continuous frame feature information, filters out features that are stable in the time dimension, and writes them into the long-term memory layer; the bootstrap cue loop generates a stable position map based on the feature variance and constructs a cue weight map, strengthening stable and highly responsive features to generate a cue-enhanced feature set; the dual-domain representation decoder restores the spatial resolution and generates region prediction values ​​and structural response prediction values ​​in the region decoding branch and structural response decoding branch, respectively, and outputs the tumor region mask and tumor confidence map.

[0037] The structure building module generates a white light structure map using a white light image and a white light reliable region mask, and generates a fluorescence structure map using a fluorescence correction image, a tumor region mask, and a tumor confidence map. The gradient threshold in the white light structure map is set to 20, and the gradient threshold in the fluorescence structure map is set to 12, with the pixel grayscale difference used as the gradient measurement benchmark.

[0038] The keypoint extraction module constructs 5×5 local neighborhoods centered on non-zero pixels in both the white light and fluorescence structure maps, and extracts the sets of keypoints for both white light and fluorescence using local maxima criteria. The cross-modal matching and registration module arranges the 5×5 neighborhood pixel values ​​corresponding to the keypoints into a one-dimensional feature vector in a fixed order, inputs it into the SuperGlue algorithm for matching, filters matching pairs under bidirectional consistency constraints, calculates translation and scale parameters, and generates a registration mapping relationship from the fluorescence image to the white light image.

[0039] The navigation image generation module maps the tumor region mask and tumor confidence map to the white light image coordinate system according to the registration mapping relationship, performs pixel-by-pixel addition, and truncates the pixel value when it exceeds 255 to generate an intraoperative navigation image, which is then displayed in real time through an augmented reality display terminal.

[0040] To verify the performance of the improved SAM2 model, a comparative experiment was conducted with convolutional neural network models and existing SAM2 models. The results are shown in Table 1 below.

[0041] Table 1. Comparison of segmentation performance of different models Model Name Mean Intersection over Union (IoU) Inter-frame consistency index False positive rate (%) Convolutional Neural Network Model 0.821 0.884 9.6 Existing SAM2 model 0.856 0.912 6.8 Improved SAM2 model 0.893 0.947 4.1 As can be seen from the data in Table 1 above, the convolutional neural network model, the existing SAM2 model, and the improved SAM2 model show a progressively improving trend in terms of average intersection-over-union ratio (AUC), inter-frame consistency index, and false detection rate. Specifically, the AUC increased from 0.821 to 0.893, indicating that the improved SAM2 model has higher region overlap in terms of single-frame segmentation accuracy; the false detection rate decreased from 9.6% to 4.1%, indicating a significant reduction in incorrect identification of non-tumor regions; and the inter-frame consistency index improved from 0.884 to 0.947, reflecting the stability of the segmentation results across consecutive video frames.

[0042] The average intersection-over-union (IoU) ratio measures the degree of overlap between the tumor region mask predicted by the model and the manually annotated ground truth mask. It is calculated as follows: the number of pixels where both the predicted and ground truth masks are simultaneously set to 1 is counted as the intersection pixel count; the number of pixels where either the predicted or ground truth mask is set to 1 is counted as the union pixel count; and the intersection pixel count is divided by the union pixel count to obtain the IoU ratio for a single frame. The average IoU ratio is obtained by taking the arithmetic mean of the IoU ratios of all images in the test dataset. The closer this ratio is to 1, the higher the degree of overlap between the predicted and ground truth regions. The inter-frame consistency index measures the similarity between tumor region masks in two adjacent frames. The calculation method is as follows: after spatially aligning the tumor region mask of the current frame with the tumor region mask of the previous frame, calculate the ratio of the number of intersection pixels to the number of union pixels of the two frames, which is the average value of the crossover-union ratio of consecutive frames. When the segmentation result fluctuates little in the time dimension, this index is close to 1; when flickering, jumps or occlusion misjudgments occur, this index decreases. The improved SAM2 model strengthens the stable features through hierarchical memory stack and bootstrap cue loop, effectively suppressing the segmentation fluctuations caused by intraoperative viewpoint changes and instantaneous occlusion. Therefore, the inter-frame consistency index is significantly improved, indicating that the improved SAM2 model has better temporal continuity and stability in dynamic intraoperative scenarios.

[0043] In practical continuous surgical applications, after using the system of this invention, the detection rate of microlesions with a diameter of less than 3 mm increased from 72.4% to 91.6%, and the average tumor localization time per surgery was shortened from 8.5 minutes to 5.2 minutes, verifying the effectiveness of this invention in improving segmentation accuracy and registration accuracy in complex surgical field environments.

[0044] In this embodiment, a dual-channel image acquisition module enables the synchronous acquisition of white light and fluorescence images, ensuring consistency of input data in both time and space dimensions. An image preprocessing module normalizes the fluorescence image and performs discrete wavelet transform, while constructing a reliable white light region mask for the white light image, effectively suppressing intraoperative reflections, low-texture areas, and background noise interference. During tumor segmentation, a hierarchical memory stack and bootstrap cueing loop structure in the improved SAM2 model are used to screen and enhance the temporal consistency of continuous frame features, improving the stability of the tumor region mask and tumor confidence map in dynamic scenes. A structure construction module unifies the expression forms of the white light and fluorescence structure maps, and combined with a key point extraction module and cross-modal matching based on the SuperGlue algorithm, accurate registration from fluorescence to white light images is achieved. Finally, a stable mapping of tumor region information to the white light field of view is completed in the navigation image generation module, forming an intuitive and continuous intraoperative navigation image. This effectively solves the problems of intraoperative fluorescence signal fluctuations, difficulties in cross-modal registration, and unstable segmentation results, improving the accuracy and reliability of intraoperative navigation in ovarian cancer surgery.

[0045] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A machine vision-based intraoperative navigation method for ovarian cancer, characterized in that, Includes the following steps: Step 1: Before ovarian cancer surgery, inject tumor microenvironment-responsive NIR-IIb composite nanoprobes into the patient to obtain fluorescence signals; Step 2: During the ovarian cancer surgery, simultaneously acquire white light images and fluorescence images corresponding to the fluorescence signals at the same time point to obtain white light images and fluorescence images; Step 3: Normalize and perform discrete wavelet transform on the fluorescence image to obtain a fluorescence-corrected image, and perform reliable region identification on the white light image to obtain a reliable region mask for white light; Step 4: Input the fluorescence-corrected image into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. Step 5: Generate a white light structure map based on the white light image and the white light reliable region mask, and generate a fluorescence structure map based on the fluorescence correction image, the tumor region mask, and the tumor confidence map; Step 6: Extract the set of white light key points based on the white light structure diagram, and extract the set of fluorescence key points based on the fluorescence structure diagram; Step 7: Based on the white light keypoint set and the fluorescence keypoint set, use the bidirectional consistency constraint rule of the SuperGlue algorithm to perform keypoint matching and generate the registration mapping relationship from the fluorescence image to the white light image; Step 8: Based on the registration mapping relationship from fluorescence image to white light image, map the tumor region mask and tumor confidence map to the white light image coordinate system to generate intraoperative navigation image.

2. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step one specifically involves: Injecting tumor microenvironment-responsive NIR-IIb composite nanoprobes into patients before ovarian cancer surgery; The NIR-IIb composite nanoprobe circulates and accumulates in the ovarian cancer tumor tissue region in vivo; Under the influence of the glutathione-enriched microenvironment in the tumor tissue region compared to the normal tissue region, the NIR-IIb composite nanoprobe undergoes structural and energy level changes, enabling it to regain fluorescence emission capability in the tumor tissue region and emit a fluorescence signal with a wavelength of 1650 nm.

3. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step two specifically involves: During ovarian cancer surgery, the white light imaging device and the fluorescence imaging device are set under the same surgical visual axis, and a spatial correspondence is established between the white light imaging device and the fluorescence imaging device so that the white light imaging device and the fluorescence imaging device can image the same surgical field area. A unified external trigger signal source is set up, and trigger signals are sent simultaneously to the white light imaging device and the fluorescence imaging device by the external trigger signal source in each acquisition cycle. Upon receiving the trigger signal, the white light imaging device acquires a white light image at the corresponding moment; Upon receiving the trigger signal, the fluorescence imaging device acquires the fluorescence signal and generates a fluorescence image by configuring an optical filter component that matches the wavelength of 1650nm. The same time stamp is assigned to the white light image and fluorescence image acquired by the same trigger signal to obtain the white light image and fluorescence image corresponding to the time stamp.

4. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step three specifically involves: The fluorescence image is converted into a single-channel grayscale image, and the pixel values ​​of the single-channel grayscale image are linearly normalized to obtain a normalized fluorescence image. Perform discrete wavelet transform on the normalized fluorescence image to decompose the normalized fluorescence image into a low-frequency approximate subband and multiple high-frequency detail subbands; For each high-frequency detail sub-band, calculate the absolute value of all wavelet coefficients in the high-frequency detail sub-band, and obtain the median of the absolute values ​​of the wavelet coefficients; Calculate the absolute deviation between the absolute value and the median of the wavelet coefficients, and obtain the median of the absolute deviation as the denoising threshold of the high-frequency detail subband; Wavelet coefficients whose absolute values ​​are less than the denoising threshold in the high-frequency detail subband are set to zero, while wavelet coefficients whose absolute values ​​are greater than or equal to the denoising threshold are retained. Perform inverse wavelet transform based on the processed wavelet coefficients to reconstruct the wavelet-denoised fluorescence image; In the fluorescence image after wavelet denoising, according to the set size of the sliding window, the sum of all pixel values ​​in each sliding window is divided by the total number of pixels in the sliding window to obtain the mean local background intensity of the corresponding sliding window. For each pixel in the wavelet-denoised fluorescence image, the pixel value is subtracted from the mean local background intensity corresponding to the sliding window where the pixel is located to obtain the corrected pixel value; When the difference result is less than zero, the corresponding corrected pixel value is set to zero, and a fluorescence-corrected image is generated based on all corrected pixel values. The brightness channel of the white light image is extracted, and pixels whose pixel values ​​exceed a set brightness threshold are marked as bright areas. Calculate the pixel gradient magnitude for the white light image, and mark pixels with gradient magnitudes lower than a set gradient threshold as low-texture regions; The bright areas and low-texture areas are merged at the pixel level to generate a set of unreliable region pixels. Pixels in the unreliable region pixel set are assigned a value of 0, and the remaining pixels are assigned a value of 1 to generate a white light reliable region mask, where pixels with a value of 0 represent unreliable regions and pixels with a value of 1 represent reliable regions.

5. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step four specifically involves: The fluorescence-corrected image is input into the encoder, which includes a multi-layer two-dimensional convolutional unit and a downsampling unit. The two-dimensional convolutional unit uses a 3×3 convolutional kernel and combines it with a convolutional operation with a stride of 1. The downsampling unit uses a convolutional operation with a stride of 2 to output feature maps at multiple resolution levels, forming the feature set of the current frame. The current frame feature set is input into a hierarchical memory stack, which includes a short-term memory layer and a long-term memory layer; the short-term memory layer stores the current frame feature set for a set number of frames in a first-in-first-out manner. For the current frame feature set corresponding to the same spatial location, calculate the feature mean along the time dimension, and calculate the element-wise difference between each time frame feature and the feature mean; When the element-wise difference is less than the set change threshold in multiple consecutive time frames, the feature mean of the corresponding spatial location is written into the long-term memory layer. Align the feature set stored in the long-term memory layer with the feature set of the current frame in spatial position, and combine them to form a memory feature set; Input the memory feature set into the bootstrap cue loop. Calculate the numerical variance of the corresponding feature in consecutive time frames for each spatial location in the memory feature set. When the numerical variance is less than the set variance threshold, set the value of the corresponding spatial location to 1; otherwise, set the value to 0 to generate a stable location map. The sum of absolute values ​​of features along the channel dimension is calculated for the memory feature set at each spatial location to obtain the response map; the response map is multiplied element-wise with the stable location map and min-max normalization is performed to obtain the cue weight map; The prompt weight map is copied and expanded along the channel dimension to a weight set with the same number of channels as the memory feature set, and the weight set is multiplied element-wise with the memory feature set to obtain the prompt enhancement feature set; The cue-enhanced feature set is input into the dual-domain representation decoder, which includes a region decoding branch and a structural response decoding branch; The region decoding branch includes multiple deconvolution units. Each deconvolution unit uses a 4×4 convolution kernel and an upsampling operation with a stride of 2, and outputs the region prediction value through a 1×1 convolution kernel. Pixels whose predicted regional values ​​are greater than or equal to the set prediction threshold are assigned a value of 1, and those that are not assigned a value of 0, thus obtaining the tumor region mask. The structural response decoding branch includes a 3×3 convolutional unit and a gradient enhancement unit. The gradient enhancement unit performs difference operations on the cue enhancement feature set in the horizontal and vertical directions respectively, and outputs boundary prediction values ​​through a 1×1 convolutional kernel. The region prediction value and the boundary prediction value are concatenated along the channel dimension to generate a joint prediction feature; The joint prediction features are calculated using Softmax normalization in the channel dimension to obtain the tumor confidence map.

6. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step five specifically involves: The white light image is converted to grayscale to obtain a white light grayscale image; The white light grayscale image is multiplied pixel by pixel with the white light reliable region mask to obtain the white light effective region image; Calculate the pixel gradient magnitude for the effective white light region image and generate a white light gradient response map; Based on the white light gradient response map, the gradient magnitude of each pixel position is compared with a set white light gradient threshold. When the pixel gradient magnitude is greater than or equal to the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set as the gradient magnitude. When the pixel gradient magnitude is less than the set white light gradient threshold, the value of the corresponding pixel position in the white light structure map is set to 0, and the white light structure map is composed of the values ​​of all pixel positions. The fluorescence-corrected image is multiplied pixel by pixel with the tumor region mask to obtain the fluorescence image after the tumor region mask is applied. The fluorescence image after the tumor region is masked is multiplied pixel by pixel with the tumor confidence map, and the pixel values ​​are weighted according to the corresponding tumor confidence level to obtain a confidence-weighted fluorescence image. Calculate the pixel gradient magnitude for the confidence-weighted fluorescence image and generate a fluorescence gradient response map; Based on the fluorescence gradient response map, the gradient magnitude of each pixel position is compared with a set fluorescence gradient threshold. When the pixel gradient magnitude is greater than or equal to the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set as the gradient magnitude. When the pixel gradient magnitude is less than the set fluorescence gradient threshold, the value of the corresponding pixel position in the fluorescence structure map is set to 0, and the fluorescence structure map is composed of the values ​​of all pixel positions.

7. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step six specifically involves: In the white light structure diagram, a local neighborhood of size 5×5 is constructed with each non-zero pixel position as the center; Compare the values ​​of each pixel position within the local neighborhood. When the value of the center pixel position is greater than the values ​​of the other pixel positions within the local neighborhood, it is determined that the center pixel position satisfies the local maximum condition, and the center pixel position is marked as a white light key point. By iterating through all pixel locations that satisfy the local maximum condition, a set of white light keypoints is obtained. In the fluorescence structure diagram, the same 5×5 local neighborhood as the white light structure diagram is used to traverse the non-zero pixel positions and select all pixel positions that satisfy the local maximum condition as fluorescence key points to obtain the fluorescence key point set.

8. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step seven specifically involves: Taking each key point in the white light key point set and the fluorescence key point set as the center, the pixel values ​​in the 5×5 local neighborhood are read in the white light structure map and the fluorescence structure map respectively in the order from the top left corner to the bottom right corner and in row priority, and arranged in order to form a one-dimensional numerical sequence, which serves as the feature vector of the corresponding key point. The feature vectors corresponding to the white light keypoint set and the fluorescent keypoint set are input into the SuperGlue algorithm. Element-wise multiplication is performed on each feature vector in the white light keypoint set and each feature vector in the fluorescent keypoint set, and the multiplication results are summed to obtain the corresponding matching score. A matching score matrix is ​​constructed based on the matching scores between all white light keypoints and fluorescent keypoints. Each matching score in the matching score matrix is ​​divided by the sum of the matching scores in its row to obtain a row-normalized matching score matrix. Each matching score in the row-normalized matching score matrix is ​​then divided by the sum of the matching scores in its column to obtain a normalized matching score matrix. In the normalized matching score matrix, the bidirectional consistency constraint rule of the SuperGlue algorithm is used for matching and filtering. The bidirectional consistency constraint rule is as follows: for any key point in the white light key point set, if the corresponding fluorescent key point has the maximum normalized matching score in the row corresponding to the white light key point, and the white light key point also has the maximum normalized matching score in the column corresponding to the fluorescent key point, then it is determined that the white light key point and the fluorescent key point constitute a matching pair. The key point correspondences that satisfy the bidirectional consistency constraint rules are traversed to obtain cross-modal key point matching results. Based on the cross-modal keypoint matching results, the pixel coordinates of each matching pair in the white light image coordinate system and the fluorescence image coordinate system are extracted; For each matching pair of pixel coordinates, the corresponding coordinate difference is calculated, and the coordinate differences are summed and averaged to obtain the translation parameter between the white light image coordinate system and the fluorescence image coordinate system. Divide the Euclidean distance of each matching pair in the white light image coordinate system by the Euclidean distance in the fluorescence image coordinate system to obtain the corresponding distance ratio. The distance ratios of all matching pairs are summed, and the summation is divided by the number of matching pairs to obtain the scale parameter between the white light image coordinate system and the fluorescence image coordinate system. The scale parameter is applied to the pixel coordinates in the fluorescence image coordinate system to perform a scale transformation on the pixel coordinates in the fluorescence image. The translation parameter is added to the pixel coordinates of the fluorescence image after scaling, so that the pixel coordinates in the fluorescence image are aligned with the coordinate system of the white light image, thereby generating a registration mapping relationship from the fluorescence image to the white light image.

9. The intraoperative navigation method for ovarian cancer based on machine vision according to claim 1, characterized in that, Step eight specifically involves: Based on the registration mapping relationship from fluorescence image to white light image, for each pixel position in the tumor region mask, the pixel coordinates in the fluorescence image coordinate system are extracted, and scale transformation and translation transformation are performed sequentially according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor region mask and the white light image are added pixel by pixel at corresponding coordinate positions to obtain the tumor region overlay image; For each pixel position in the tumor confidence map, the pixel coordinates in the fluorescence image coordinate system are extracted, and scale transformation and translation transformation are performed according to the registration mapping relationship to obtain the corresponding mapped coordinates in the white light image coordinate system. The mapped tumor confidence map and the tumor region overlay map are added pixel by pixel at the corresponding coordinate positions. When the sum is greater than the maximum gray value, the pixel value is set to the maximum gray value to generate an intraoperative navigation image.

10. A machine vision-based intraoperative navigation system for ovarian cancer according to claim 1, performing a machine vision-based intraoperative navigation method for ovarian cancer according to any one of claims 1 to 9, characterized in that, Includes the following modules: A dual-channel image acquisition module is used to simultaneously acquire white light images and fluorescence images during ovarian cancer surgery using a white light imaging device and a fluorescence imaging device set under the same surgical visual axis, and to assign the same time stamp to the white light images and fluorescence images. The image preprocessing module is used to normalize and perform discrete wavelet transform on the fluorescence image to obtain a fluorescence-corrected image, and to perform reliable region identification on the white light image to obtain a reliable region mask for white light. The tumor segmentation module is used to input the fluorescence-corrected image into the improved SAM2 model for tumor segmentation. The improved SAM2 model includes an encoder, a hierarchical memory stack, a bootstrap cueing loop, and a dual-domain representation decoder connected in sequence, and outputs a tumor region mask and a tumor confidence map. The structure building module is used to generate a white light structure map based on the white light image and the white light reliable region mask, and to generate a fluorescence structure map based on the fluorescence correction image, the tumor region mask and the tumor confidence map; The key point extraction module is used to extract a set of white light key points based on the white light structure diagram and to extract a set of fluorescent key points based on the fluorescence structure diagram. The cross-modal matching and registration module is used to perform key point matching based on the white light key point set and the fluorescence key point set, using the SuperGlue algorithm to generate a registration mapping relationship from the fluorescence image to the white light image. The navigation image generation module is used to map the tumor region mask and tumor confidence map to the white light image coordinate system based on the registration mapping relationship between the fluorescence image and the white light image, thereby generating an intraoperative navigation image.

Images