Thyroid intraoperative real-time navigation method and system based on multi-mode optical fusion

Abstract

The invention discloses a thyroid intraoperative real-time navigation method and a thyroid intraoperative real-time navigation system based on multi-mode optical fusion. The thyroid intraoperative real-time navigation method comprises the following steps: outputting visible light through an endoscope and coupling light waves of a narrow-band multispectral light source to irradiate an operative field, exciting parathyroid glands to generate near-infrared fluorescence, and receiving reflected visible light, split light, near-infrared fluorescence and laser speckle signals through an endoscope probe. And separating the composite optical signal into four channels, and respectively generating an anatomical structure color image, a blood vessel spectroscopic image, a parathyroid gland near-infrared fluorescence image and a laser speckle image. Performing decorrelation processing on the laser speckle image to generate a blood flow dynamic pseudo-color decorrelation speckle image; an anatomical structure, a blood vessel center line, a parathyroid gland contour and blood flow dynamic feature points are extracted through a multi-modal registration technology, after affine transformation space alignment is conducted, a comprehensive imaging map containing the anatomical structure, blood vessel distribution and parathyroid gland function marking information is generated through a wavelet fusion algorithm, and intraoperative multi-dimensional real-time tissue navigation is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time navigation method and system during thyroid surgery based on multimodal optical fusion. Background Art

[0002] During thyroid surgery, accurate identification of parathyroid glands and cancerous tissue is a critical clinical requirement for avoiding postoperative complications such as hypoparathyroidism and residual tumor. Currently, mainstream methods for intraoperative tissue identification include visual identification, tissue sinking and floating methods, and frozen section pathological examination. However, these methods rely on the surgeon's experience and have inherent flaws such as high subjectivity and poor real-time performance. Visual identification is limited by subtle differences in tissue color and morphology, while frozen sections require 15-30 minutes to obtain pathological results, making them difficult to meet the demands of real-time intraoperative decision-making. Furthermore, they lack the ability to effectively identify early-stage, microscopic cancer lesions and parathyroid glands with abnormal functional status, resulting in a misdiagnosis rate as high as 20%-30%.

[0003] The nanocarbon dye tracing technology, which is widely used in clinical practice, assists in identifying tissues through color contrast, but its technical bottlenecks significantly restrict its clinical value: first, nanocarbon particles are non-absorbable, and long-term retention in the body may trigger chronic inflammatory reactions or granulomas, and there is a potential risk of allergies. According to statistics, approximately 3%-5% of patients have varying degrees of allergic reactions to carbon-based tracers; second, there is a lack of standardized operating procedures for injection dosage and method. Leaked nanocarbon will contaminate the surgical field of view and interfere with the surgeon's judgment of tissue boundaries, especially in complex anatomical areas (such as around the recurrent laryngeal nerve), which is prone to form artifacts and reduce the recognition of vascular or neural structures; third, for cases with severe lymph node metastasis, the blockage caused by tumor invasion of lymphatic vessels will prevent the effective enrichment of nanocarbon, increasing the risk of postoperative recurrence.

[0004] New technologies such as fluorescence laparoscopy and 4K high-definition laparoscopy that have emerged in recent years have improved some recognition effects by increasing image resolution or introducing exogenous fluorescent contrast agents, but have not yet broken through the core technical barriers: exogenous fluorescent contrast agents need to be injected in advance and rely on the body's metabolic process, there is a risk of allergies, and the clearance rate varies greatly from person to person, which may cause confusion between fluorescent signals and background noise; in terms of functional imaging of early cancerous tissue and adjacent tissues, existing technologies are limited by single modality signals (such as relying only on fluorescence intensity or anatomical structure), and have insufficient ability to capture functional characteristics such as abnormal blood supply and changes in metabolic activity, and cannot meet the needs of precision surgery for qualitative diagnosis of early lesions. Summary of the Invention

[0005] In order to solve at least one of the above-mentioned technical problems, the present invention provides a real-time navigation method and system for thyroid surgery based on multimodal optical fusion.

[0006] In a first aspect, the present invention provides a real-time navigation method for thyroid surgery based on multimodal optical fusion, the method comprising:

[0007] The endoscope outputs coupled light waves of visible light and narrow-band multispectral light source to illuminate the surgical field, stimulating the parathyroid gland to produce near-infrared fluorescence. The reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal are then received by the endoscope probe.

[0008] Separating the composite light signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, respectively generating an anatomical structure color image of the surgical field, a vascular spectroscopic image, a parathyroid gland near-infrared fluorescence image, and a laser speckle image;

[0009] performing decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing blood flow dynamics;

[0010] performing a blood vessel enhancement process on the blood vessel spectroscopic image to generate a blood vessel contour identification map;

[0011] The color image, the vascular contour identification map, the decorrelation speckle map and the decorrelation speckle map are spatially aligned and fused by multimodal registration technology to generate a comprehensive imaging atlas, which contains anatomical structure, vascular distribution and parathyroid function marker information.

[0012] Preferably, the performing blood vessel enhancement processing on the blood vessel spectroscopic image includes:

[0013] The image is normalized, and the blood vessels are segmented using a two-dimensional Gaussian matched filter. The segmentation results are binarized based on the maximum inter-class variance method, and small connected areas that are not blood vessels are eliminated to generate a blood vessel contour map.

[0014] Preferably, the color image, the vascular contour identification map, the decorrelation speckle map and the decorrelation speckle map are spatially aligned and fused by a multimodal registration technology to generate a comprehensive imaging atlas, including:

[0015] Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm;

[0016] Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering;

[0017] In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point;

[0018] performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points;

[0019] Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error;

[0020] The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

[0021] Preferably, performing decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing blood flow dynamics includes:

[0022] Laser speckle image sequences are continuously acquired at preset time intervals, and the light intensity autocorrelation function between adjacent frames is calculated. The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified in combination with the path length distribution model.

[0023] A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

[0024] Preferably, the narrow-band multispectral light source includes a visible light band with a wavelength range of 400 to 700 nm and a laser band of 785 nm. The spectroscopic channel separates the blood vessel spectroscopic signal through a tunable filter to generate a blood vessel spectroscopic image of the corresponding wavelength.

[0025] In a second aspect, the present invention further provides a real-time navigation system for thyroid surgery based on multimodal optical fusion, the system comprising:

[0026] The multi-source light wave irradiation module is used to illuminate the surgical field through the coupled light waves of visible light and narrow-band multi-spectral light source output by the endoscope, stimulate the parathyroid gland to produce near-infrared fluorescence, and receive the reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal through the endoscope probe;

[0027] a composite optical signal separation module, configured to separate the composite optical signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, and to generate a color image of the anatomical structure of the surgical field, a spectroscopic image of the blood vessels, a near-infrared fluorescence image of the parathyroid gland, and a laser speckle image, respectively;

[0028] a decorrelation speckle pattern generating module, configured to perform decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing the dynamics of blood flow;

[0029] a blood vessel contour identification map generating module, configured to perform blood vessel enhancement processing on the blood vessel spectroscopic image to generate a blood vessel contour identification map;

[0030] A multimodal image fusion module is used to spatially align and fuse the color image, vascular contour identification map, decorrelation speckle map and decorrelation speckle map through multimodal registration technology to generate a comprehensive imaging atlas, which contains anatomical structure, vascular distribution and parathyroid function marker information.

[0031] Preferably, the multimodal image fusion module is further used for:

[0032] Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm;

[0033] Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering;

[0034] In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point;

[0035] performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points;

[0036] Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error;

[0037] The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

[0038] Preferably, the decorrelation speckle pattern generating module is further configured to:

[0039] Laser speckle image sequences are continuously acquired at preset time intervals, and the light intensity autocorrelation function between adjacent frames is calculated. The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified in combination with the path length distribution model.

[0040] A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

[0041] In a third aspect, the present invention also provides an electronic device comprising a processor and a memory, wherein the memory is used to store computer program code, and the computer program code comprises computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation thereof.

[0042] In a fourth aspect, the present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor of an electronic device, the processor executes the method as described in the first aspect above and any possible implementation method thereof.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) This invention uses an endoscope to output a coupled lightwave of visible light and a narrowband multispectral light source to stimulate endogenous near-infrared fluorescence of the parathyroid glands and collect multimodal signals. After signal separation and processing, it generates a comprehensive imaging atlas that includes anatomical structure, vascular distribution, functional markers, and blood flow dynamics. This technology does not require exogenous dyes, avoiding the safety risks and non-standardized operation of nanocarbon. Through multimodal information fusion, it achieves real-time visualization of tissue morphology, functional status, and hemodynamics, addressing the subjective limitations of traditional methods that rely on visual identification. It significantly improves the accuracy of identifying parathyroid glands and cancerous tissue, and reduces the risk of missed diagnosis of lymph node metastasis through dynamic blood flow analysis, providing a multi-dimensional quantitative basis for precise intraoperative navigation.

[0045] 2) This invention performs normalization, Gaussian filtering, OTSU binarization, and connected domain screening on the vascular spectroscopic image to generate a clear vascular contour map. This process, using standardized image enhancement algorithms, effectively suppresses noise such as uneven illumination and tissue reflections in the surgical field of view, improving the contrast between blood vessels and the background by over 60%. It accurately extracts vascular structures with a diameter ≥ 0.5 mm, addressing the issues of visual field contamination caused by conventional nanocarbon dye leakage and blurred imaging due to lymphatic obstruction. This provides high-contrast vascular distribution information for avoiding critical vascular branches and planning safe resection paths during surgery, significantly reducing the risk of intraoperative bleeding and the probability of vascular injury.

[0046] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0048] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0049] Figure 1 A schematic diagram of a flow chart of a real-time navigation method during thyroid surgery based on multimodal optical fusion provided by an embodiment of the present invention;

[0050] Figure 2 A schematic structural diagram of a real-time navigation system for thyroid surgery based on multimodal optical fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0052] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] Existing tissue identification technology during thyroid surgery has the problems of strong subjectivity, poor real-time performance, safety risks of relying on exogenous tracers and non-standard operation. In addition, a single modality signal cannot take into account anatomical structure, functional markers and hemodynamic information, resulting in clinical defects such as misidentification of parathyroid glands, missed diagnosis of lymph nodes and low identification rate of early cancer.

[0054] See also Figure 1 , Figure 1 The flowchart of a real-time navigation method for thyroid surgery based on multimodal optical fusion provided by an embodiment of the present invention is as follows. Figure 1 As shown, the method includes:

[0055] S100 uses the coupled light waves of visible light and narrow-band multispectral light source output by the endoscope to illuminate the surgical field, stimulate the parathyroid gland to produce near-infrared fluorescence, and then receives the reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal through the endoscope probe;

[0056] S200, separating the composite light signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, and generating a color image of the anatomical structure of the surgical field, a spectral image of the blood vessels, a near-infrared fluorescence image of the parathyroid gland, and a laser speckle image, respectively;

[0057] S300, performing decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing blood flow dynamics;

[0058] S400, performing blood vessel enhancement processing on the blood vessel spectroscopic image to generate a blood vessel contour identification map;

[0059] S500, spatially aligning and fusing the color image, the vascular contour identification map, the decorrelated speckle map, and the decorrelated speckle map through a multimodal registration technology to generate a comprehensive imaging atlas, wherein the atlas includes anatomical structure, vascular distribution, and parathyroid function marker information.

[0060] The composite light source module at the front end of the endoscope outputs coupled light waves composed of visible light and a narrowband multispectral light source. The visible light covers the 400-700nm band to visualize the anatomical structures in the surgical field, while the narrowband multispectral light source includes a 785nm laser band to stimulate endogenous fluorescent substances (such as autofluorescent proteins or metabolites) in the parathyroid tissue, producing near-infrared fluorescence (wavelength range approximately 800-1000nm). After the coupled light waves illuminate the surgical area, the reflected mixed light signal (including visible light, narrowband spectral scattered light, near-infrared fluorescence, and laser speckle signals) is received by the multi-channel optical sensor of the endoscope probe. This sensor integrates a beam splitter prism and a wavelength-selective detector to initially separate the light signals from different bands.

[0061] The received composite light signal is separated into four independent channels by an optical spectrometer system: a visible light channel, which directly obtains a color image of the surgical field through a broadband filter (400-700nm), reflecting the anatomical morphology and surface texture of the tissue; a spectroscopic channel, which uses tunable filters (such as vascular-sensitive wavelengths such as 450nm and 540nm) to separate scattered light of specific wavelengths to generate a vascular spectroscopic image, highlighting the hemoglobin-rich vascular structure; a near-infrared fluorescence channel, which uses a long-pass filter (>750nm) to filter the near-infrared fluorescence signal excited by the parathyroid gland to generate a fluorescence image to mark the target tissue; and a laser speckle channel, which separately receives the speckle interference signal of the 785nm laser for subsequent blood flow dynamics analysis.

[0062] For the laser speckle channel signal, time series processing technology is used to continuously acquire 50 frames of speckle images at a preset time interval of 20ms. By calculating the light intensity autocorrelation function between adjacent frames, the light intensity fluctuations are converted into the electric field autocorrelation function based on the Siegert relationship. Combined with the light scattering path length distribution model in the tissue, the blood flow velocity parameters of the capillary network are quantified, and a decorrelated speckle pattern is generated through pseudo-color coding, where blue indicates low blood flow areas (such as adipose tissue) and red indicates high blood flow areas (such as the thyroid gland).

[0063] For the vascular spectroscopic images of the spectroscopic channel, grayscale normalization was performed to eliminate the influence of uneven illumination. A two-dimensional Gaussian matched filter (kernel size σ = 3 pixels) was used to perform a convolution operation on the image to enhance the contrast of tubular structures (blood vessels). The maximum between-class variance method (OTSU algorithm) was used to automatically calculate the threshold, and the image was binarized into vascular (white) and non-vascular (black) regions. Isolated noise regions with an area of less than 50 pixels were eliminated through a morphological opening operation (3×3 pixel structuring element) to generate a clear vascular contour map with superimposed vascular direction and branching information.

[0064] Anatomical structural feature points (such as the thyroid margin and tracheal cartilage contour) are extracted from visible light images, and feature descriptors containing scale and direction information are generated using the SIFT algorithm. The vascular contour identification map is skeletonized, and the intersection points of the vascular centerlines are extracted as vascular feature points. Hessian matrix filtering is used to generate a second-order directional descriptor of the vascular direction. In near-infrared fluorescence images, the parathyroid gland regions with high fluorescence intensity are located through adaptive threshold segmentation (based on the Otsu algorithm), and their contour vertices are extracted as functional marker feature points. After pseudo-color mapping of the decorrelated speckle pattern, the optical flow method is used to calculate the pixel motion vector between two adjacent frames to generate blood flow dynamic feature points (such as flow velocity gradient mutation points). The four types of feature points are input into the affine transformation model, and spatial alignment is achieved by minimizing the reprojection error (mean square error ≤ 0.5 pixels). The aligned image is then decomposed in the frequency domain using a weighted wavelet fusion algorithm. The overall outline of the anatomical structure is retained in the low-frequency band, and vascular details, fluorescent markers, and hemodynamic information are fused in the high-frequency band. The final output is a comprehensive imaging atlas that includes tissue morphology, vascular distribution, functional markers, and blood flow dynamics.

[0065] In this embodiment, an endoscope outputs a coupled lightwave of visible light and a narrowband multispectral light source to stimulate endogenous near-infrared fluorescence of the parathyroid glands and collect multimodal signals. After signal separation and processing, a comprehensive imaging atlas is generated that includes anatomical structure, vascular distribution, functional markers, and blood flow dynamics. This technology does not require exogenous dyes, avoiding the safety risks and non-standardized operation of nanocarbon. Through multimodal information fusion, it achieves real-time visualization of tissue morphology, functional status, and hemodynamics, addressing the subjective defects of traditional methods that rely on visual identification. It significantly improves the accuracy of identifying parathyroid glands and cancerous tissues, and reduces the risk of missed diagnosis of lymph node metastasis through dynamic blood flow analysis, providing a multi-dimensional quantitative basis for precise intraoperative navigation.

[0066] Preferably, the performing blood vessel enhancement processing on the blood vessel spectroscopic image includes:

[0067] The image is normalized, and the blood vessels are segmented using a two-dimensional Gaussian matched filter. The segmentation results are binarized based on the maximum inter-class variance method, and small connected areas that are not blood vessels are eliminated to generate a blood vessel contour map.

[0068] In this embodiment, the grayscale of the image is normalized and the pixel value range is mapped to [0, 255] to eliminate the influence of light source fluctuation. The mapping function is: ,in, is the original pixel value, and are the minimum and maximum grayscale values of the image respectively. A two-dimensional Gaussian matched filter (kernel function is The normalized image is convolved with the maximum inter-class variance method (OTSU) to enhance the contrast between the blood vessels and the background. The optimal threshold T is calculated based on the maximum inter-class variance method (OTSU), and the image is binarized into the blood vessel area T and the non-vascular area (I < T). The algorithm automatically determines the threshold by maximizing the grayscale variance of the two types of pixels. The formula is: ,in, is the ratio of the two types of pixels, is the average grayscale value of the two types of pixels. Through connected domain analysis, small connected regions with an area less than 100 pixels (usually noise or capillary ends) are eliminated, and continuous vascular contours are retained to generate a clear vascular contour identification map that can accurately reflect the distribution of blood vessels with a diameter ≥ 0.5 mm.

[0069] In this embodiment, the vascular spectroscopic image is normalized, enhanced with Gaussian filtering, binarized with OTSU, and filtered for connected domains to generate a clear vascular contour map. This process, using standardized image enhancement algorithms, effectively suppresses noise such as uneven illumination and tissue reflections in the surgical field of view, increasing the contrast between blood vessels and the background by over 60%. It accurately extracts vascular structures with a diameter ≥ 0.5 mm, addressing issues such as visual field contamination caused by conventional nanocarbon dye leakage and blurred imaging caused by lymphatic obstruction. This provides high-contrast vascular distribution information for avoiding critical vascular branches and planning safe resection paths during surgery, significantly reducing the risk of intraoperative bleeding and the probability of vascular injury.

[0070] Preferably, the color image, the vascular contour identification map, the decorrelation speckle map and the decorrelation speckle map are spatially aligned and fused by a multimodal registration technology to generate a comprehensive imaging atlas, including:

[0071] Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm;

[0072] Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering;

[0073] In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point;

[0074] performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points;

[0075] Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error;

[0076] The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

[0077] In this example, the SIFT algorithm is applied to visible light images. Through Gaussian pyramid construction and extreme point detection, 500-1000 stable anatomical structural feature points (such as corner points at the edge of the thyroid capsule and muscle texture features) are extracted. A 128-dimensional gradient direction histogram is generated as the first feature descriptor. The vascular contour map is skeletonized (using the Hilditch thinning algorithm). The intersections and bifurcations of the vascular centerlines are used as second feature points (approximately 200-300 in total). This is then filtered using the Hessian matrix (calculating the second-order derivative) to generate a directional descriptor (8-directional encoding) for the vascular orientation. In near-infrared fluorescence images, Otsu thresholding is used to locate regions where the fluorescence intensity exceeds three standard deviations above the background. The boundary points of the parathyroid contour (approximately 100-200 in total) are extracted, and contour curvature is used as an auxiliary descriptor. For the decorrelated speckle pattern, pseudo-color mapping (such as the jet color table) is first performed, and then the Lucas-Kanade optical flow method is used to calculate the pixel motion vector between two adjacent frames. The pixels with a flow velocity change rate > 10% are used as blood flow dynamic feature points (about 300-500).

[0078] Input the four types of feature points into the affine transformation model (including translation, rotation, and scaling parameters) and optimize the objective function using the least squares method. ,in, is the reference image feature point, Where is the feature point of the image to be registered, and H is the transformation matrix. This achieves spatial alignment of the four modal images with subpixel registration accuracy (root mean square error < 0.3 pixel). The aligned images are processed using a weighted wavelet fusion algorithm: First, a three-layer wavelet decomposition is performed to obtain low-frequency approximation components and high-frequency detail components (horizontally, vertically, and diagonally). In the low-frequency band, a weighted averaging method is used to fuse anatomical and functional marker information, with weights dynamically assigned based on the entropy of each modality (e.g., 0.4 for visible light images, 0.3 for fluorescence images, and 0.15 for vascular and speckle images). In the high-frequency band, the component with the largest absolute gradient in each modality is selected to preserve edge details. Finally, an inverse wavelet transform is used to reconstruct a comprehensive imaging atlas that simultaneously displays tissue morphology (e.g., thyroid lobes, trachea), vascular networks (arterial / venous orientation), parathyroid gland location (fluorescent highlights), and blood flow dynamics (pseudo-color velocity gradients).

[0079] In this example, SIFT feature extraction, vascular skeletonization, fluorescence region contour localization, and optical flow feature point detection are combined with affine transformation and wavelet fusion algorithms to achieve subpixel registration and frequency domain information fusion of four modal images. This technology addresses geometric deviations caused by endoscopic distortion and wavelength differences between different modal images, improving the spatial overlap accuracy of parathyroid fluorescence markers with anatomical structures and vascular distribution to within 0.3 pixels. It also preserves high-frequency details (such as vascular edges and fluorescence boundaries) and low-frequency contours (such as glandular morphology) in the fused image, significantly enhancing the functional imaging recognition rate of early-stage cancerous tissue (such as microscopic lesions) and adjacent tissue, while avoiding the blind spots caused by traditional techniques due to their reliance on exogenous contrast agents.

[0080] Preferably, performing decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing blood flow dynamics includes:

[0081] Laser speckle image sequences are continuously acquired at preset time intervals, and the light intensity autocorrelation function between adjacent frames is calculated. The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified in combination with the path length distribution model.

[0082] A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

[0083] In this embodiment, 200 frames of laser speckle images are continuously collected at a frequency of 50 Hz (single frame exposure time 10 ms) to form a time series Calculate the autocorrelation function of the light intensity of two adjacent frames (interval Δt = 20ms) , where <> represents the average value of the local area (3×3 pixel window). According to the Siegert relationship, the light intensity autocorrelation function and the electric field autocorrelation function satisfy , and then the electric field autocorrelation attenuation curve is obtained by inversion, and the blood flow velocity of each pixel is calculated by combining the Mie theory path length distribution model of light scattering in tissue ,in, The laser wavelength is 785nm, n is the tissue refractive index 1.37, is the autocorrelation decay time constant. Blood flow velocity is mapped to a pseudo-color image: 0-0.1 mm / s is set to blue (low blood flow, such as parathyroid glands), 0.1-0.5 mm / s to green (medium blood flow, such as normal thyroid glands), and >0.5 mm / s to red (high blood flow, such as diseased glands). This generates a decorrelated speckle pattern that intuitively reflects blood flow dynamics. This image can display tissue microcirculation status in real time and assist in determining tissue activity.

[0084] By calculating the autocorrelation function of laser speckle image sequences and quantifying blood flow velocity, a pseudo-color-coded decorrelation speckle image is generated, reflecting the tissue microcirculatory status in real time. This technology utilizes endogenous laser speckle signals, eliminating the need for exogenous tracers. By distinguishing blood flow velocity by color gradient (low blood flow blue / high blood flow red), it effectively addresses the problem of missed diagnosis caused by lymphatic obstruction in severe lymph node metastasis. Even if lymph nodes lose their ability to visualize lymphatic drainage due to metastasis, their abnormal hemodynamic characteristics (such as turbulent flow velocity and gradient abrupt changes) can still be accurately captured. In clinical application, this method has reduced the lymph node missed diagnosis rate from 12% to below 3%. It also provides a real-time dynamic indicator for assessing parathyroid blood supply status, preventing postoperative functional impairment caused by inadvertent ischemic resection.

[0085] Preferably, the narrow-band multispectral light source includes a visible light band with a wavelength range of 400 to 700 nm and a laser band of 785 nm. The spectroscopic channel separates the blood vessel spectroscopic signal through a tunable filter to generate a blood vessel spectroscopic image of the corresponding wavelength.

[0086] The narrowband multispectral light source consists of two components: a visible light source and a laser light source. The visible light source utilizes an LED array, covering the entire visible light band of 400-700nm, with peak wavelengths of 450nm (blue light), 550nm (green light), and 630nm (red light). This three-color mixing generates natural-color illumination for acquiring color images of anatomical structures within the surgical field. The laser light source utilizes a single-wavelength 785nm semiconductor laser, with power controlled at 50-100mW to avoid thermal damage to tissue. This wavelength effectively stimulates parathyroid autofluorescence (primarily generated by intracellular NADH and other substances) while reducing interference from hemoglobin light absorption.

[0087] The spectroscopic channel is equipped with a tunable filter (such as a liquid crystal tunable filter, LCTF) that dynamically selects vessel-sensitive wavelengths such as 450nm and 570nm. The 450nm wavelength is sensitive to deoxyhemoglobin, while the 570nm wavelength is sensitive to oxyhemoglobin. By switching the filter wavelengths in a time-sharing manner, spectroscopic images of blood vessels are generated under different blood oxygenation conditions. For example, at a wavelength of 540nm, the light absorption coefficient of intravascular hemoglobin is high, creating a significant contrast with surrounding tissue, facilitating the extraction of vessel contours. The wavelength accuracy of the filter is controlled within ±2nm, ensuring the accuracy of the spectroscopic signal.

[0088] This embodiment utilizes a narrowband multispectral light source consisting of 400-700nm visible light and 785nm laser light, and utilizes a tunable filter to isolate the vascular-sensitive wavelength signal. The 785nm laser excites endogenous fluorescence in the parathyroid glands, completely eliminating the need for exogenous dyes and the potential risk of inabsorbability and allergic reactions caused by nanocarbon. The visible light band provides natural color anatomical images, addressing the subjectivity inherent in visual identification. The tunable filter (with an accuracy of ±2nm) dynamically selects vascular-sensitive wavelengths (e.g., 450nm and 540nm) to generate high-contrast vascular spectroscopic images, avoiding the issues of inconsistent injection doses and missed dyes that interfere with the visual field associated with traditional techniques. The optimized design of this light source and spectroscopic system provides a secure and accurate foundation for multimodal signal acquisition, mitigating the medical risks of exogenous substances while enhancing the spectrally specific identification of blood vessels and functional tissues.

[0089] In summary, the method provided in this embodiment can at least achieve the following effects:

[0090] 1) This invention uses an endoscope to output a coupled lightwave of visible light and a narrowband multispectral light source to stimulate endogenous near-infrared fluorescence of the parathyroid glands and collect multimodal signals. After signal separation and processing, it generates a comprehensive imaging atlas that includes anatomical structure, vascular distribution, functional markers, and blood flow dynamics. This technology does not require exogenous dyes, avoiding the safety risks and non-standardized operation of nanocarbon. Through multimodal information fusion, it achieves real-time visualization of tissue morphology, functional status, and hemodynamics, addressing the subjective limitations of traditional methods that rely on visual identification. It significantly improves the accuracy of identifying parathyroid glands and cancerous tissue, and reduces the risk of missed diagnosis of lymph node metastasis through dynamic blood flow analysis, providing a multi-dimensional quantitative basis for precise intraoperative navigation.

[0091] 2) This invention performs normalization, Gaussian filtering, OTSU binarization, and connected domain screening on the vascular spectroscopic image to generate a clear vascular contour map. This process, using standardized image enhancement algorithms, effectively suppresses noise such as uneven illumination and tissue reflections in the surgical field of view, improving the contrast between blood vessels and the background by over 60%. It accurately extracts vascular structures with a diameter ≥ 0.5 mm, addressing the issues of visual field contamination caused by conventional nanocarbon dye leakage and blurred imaging due to lymphatic obstruction. This provides high-contrast vascular distribution information for avoiding critical vascular branches and planning safe resection paths during surgery, significantly reducing the risk of intraoperative bleeding and the probability of vascular injury.

[0092] See also Figure 2 In one embodiment, a real-time navigation system for thyroid surgery based on multimodal optical fusion is provided, the system comprising:

[0093] The multi-source light wave irradiation module 100 is used to illuminate the surgical field through the endoscope output of coupled light waves of visible light and narrow-band multi-spectral light source, stimulate the parathyroid gland to produce near-infrared fluorescence, and receive the reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal through the endoscope probe;

[0094] The composite optical signal separation module 200 is used to separate the composite optical signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, and respectively generate an anatomical structure color image of the surgical field, a vascular spectroscopic image, a parathyroid gland near-infrared fluorescence image, and a laser speckle image;

[0095] A decorrelation speckle pattern generating module 300 is configured to perform decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing the dynamics of blood flow;

[0096] A blood vessel contour identification map generating module 400 is configured to perform blood vessel enhancement processing on the blood vessel spectroscopic image to generate a blood vessel contour identification map;

[0097] The multimodal image fusion module 500 is used to spatially align and fuse the color image, the vascular contour identification map, the decorrelation speckle map and the decorrelation speckle map using a multimodal registration technique to generate a comprehensive imaging atlas containing anatomical structure, vascular distribution and parathyroid function marker information.

[0098] Preferably, the multimodal image fusion module 500 is further configured to:

[0099] Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm;

[0100] Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering;

[0101] In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point;

[0102] performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points;

[0103] Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error;

[0104] The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

[0105] Preferably, the decorrelation speckle pattern generating module 400 is further configured to:

[0106] Laser speckle image sequences are continuously acquired at preset time intervals, and the light intensity autocorrelation function between adjacent frames is calculated. The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified in combination with the path length distribution model.

[0107] A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

[0108] It can be understood that the functions or modules included in the system provided in this embodiment can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0109] The present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes a method as described in any one of the possible implementation modes.

[0110] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes a method as described in any one of the possible implementation methods described above.

[0111] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0112] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of the present invention have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.

Claims

1. A real-time navigation method for thyroid surgery based on multimodal optical fusion, characterized in that: The method comprises: The endoscope outputs coupled light waves of visible light and narrow-band multispectral light source to illuminate the surgical field, stimulating the parathyroid gland to produce near-infrared fluorescence. The reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal are then received by the endoscope probe. Separating the composite light signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, respectively generating an anatomical structure color image of the surgical field, a vascular spectroscopic image, a parathyroid gland near-infrared fluorescence image, and a laser speckle image; performing decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing blood flow dynamics; performing a blood vessel enhancement process on the blood vessel spectroscopic image to generate a blood vessel contour identification map; The color image, the vascular contour identification map, the decorrelation speckle map and the decorrelation speckle map are spatially aligned and fused by multimodal registration technology to generate a comprehensive imaging atlas, which contains anatomical structure, vascular distribution and parathyroid function marker information.

2. The method for real-time navigation during thyroid surgery based on multimodal optical fusion according to claim 1, characterized in that: The performing blood vessel enhancement processing on the blood vessel spectroscopic image includes: The image is normalized, and the blood vessels are segmented using a two-dimensional Gaussian matched filter. The segmentation results are binarized based on the maximum inter-class variance method, and small connected areas that are not blood vessels are eliminated to generate a blood vessel contour map.

3. The method for real-time navigation during thyroid surgery based on multimodal optical fusion according to claim 1, characterized in that: The color image, the blood vessel contour identification map, the decorrelation speckle map and the decorrelation speckle map are spatially aligned and fused by the multimodal registration technology to generate a comprehensive imaging atlas, including: Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm; Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering; In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point; performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points; Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error; The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

4. The method for real-time navigation during thyroid surgery based on multimodal optical fusion according to claim 1, characterized in that: The decorrelation processing is performed on the laser speckle image to generate a decorrelation speckle pattern representing the dynamics of blood flow, including: Continuously collect laser speckle image sequences at preset time intervals and calculate the light intensity autocorrelation function between adjacent frames; The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified by combining the path length distribution model; A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

5. The method for real-time navigation during thyroid surgery based on multimodal optical fusion according to claim 1, characterized in that: The narrowband multi-spectral light source includes a visible light band with a wavelength range of 400 to 700 nm and a laser band of 785 nm. The spectroscopic channel separates the blood vessel spectroscopic signal through a tunable filter to generate a blood vessel spectroscopic image of the corresponding wavelength.

6. A real-time navigation system for thyroid surgery based on multimodal optical fusion, characterized in that: The system comprises: The multi-source light wave irradiation module is used to illuminate the surgical field through the coupled light waves of visible light and narrow-band multi-spectral light source output by the endoscope, stimulate the parathyroid gland to produce near-infrared fluorescence, and receive the reflected visible light, spectroscopic signal, near-infrared fluorescence and laser speckle signal through the endoscope probe; a composite optical signal separation module, configured to separate the composite optical signal received by the endoscopic probe into a visible light channel, a spectroscopic channel, a near-infrared fluorescence channel, and a laser speckle channel, and to generate a color image of the anatomical structure of the surgical field, a spectroscopic image of the blood vessels, a near-infrared fluorescence image of the parathyroid gland, and a laser speckle image, respectively; a decorrelation speckle pattern generating module, configured to perform decorrelation processing on the laser speckle image to generate a decorrelation speckle pattern representing the dynamics of blood flow; a blood vessel contour identification map generating module, configured to perform blood vessel enhancement processing on the blood vessel spectroscopic image to generate a blood vessel contour identification map; A multimodal image fusion module is used to spatially align and fuse the color image, vascular contour identification map, decorrelation speckle map and decorrelation speckle map through multimodal registration technology to generate a comprehensive imaging atlas, which contains anatomical structure, vascular distribution and parathyroid function marker information.

7. The real-time navigation system for thyroid surgery based on multimodal optical fusion according to claim 6, characterized in that: The multimodal image fusion module is further used for: Extracting anatomical structure feature points from the color image and generating a first feature descriptor based on a SIFT algorithm; Performing blood vessel skeletonization on the spectroscopic image, extracting the blood vessel centerline as a second feature point, and generating a blood vessel direction descriptor through Hessian matrix filtering; In the near-infrared fluorescence image, the parathyroid gland region is located by threshold segmentation, and its contour boundary is extracted as a third feature point; performing pseudo color mapping on the decorrelated speckle pattern, and calculating motion vectors between adjacent speckle patterns based on an optical flow method to generate blood flow dynamic feature points; Inputting the first feature descriptor, the second feature point, the third feature point and the blood flow dynamic feature point into an affine transformation model, and achieving spatial alignment of the four-modal images by minimizing the reprojection error; The weighted wavelet fusion algorithm is used to decompose the aligned images in the frequency domain and output a comprehensive imaging spectrum.

8. The real-time navigation system for thyroid surgery based on multimodal optical fusion according to claim 6, characterized in that: The decorrelation speckle pattern generating module is further used for: Continuously collect laser speckle image sequences at preset time intervals and calculate the light intensity autocorrelation function between adjacent frames; The light intensity autocorrelation function is converted into the electric field autocorrelation function based on the Siegert relationship, and the blood flow velocity is quantified by combining the path length distribution model; A pseudo-color-coded decorrelation speckle pattern is generated based on blood flow velocity, where low blood flow areas are distinguished from high blood flow areas by color gradients.

9. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store computer program code, the computer program code comprising computer instructions, and when the processor executes the computer instructions, the electronic device executes the real-time navigation method for thyroid surgery based on multimodal optical fusion as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes the real-time navigation method for thyroid surgery based on multimodal optical fusion according to any one of claims 1 to 5.

Images