A multi-modal fluorescence imaging method and system applied to thyroid surgery

Abstract

The application provides a multimodal fluorescence imaging method and system applied to thyroid surgery, and the method comprises the following steps: collecting image data by using a multi-wavelength endoscope system, and generating an image set; performing Fourier transform on the image set, performing fine enhancement processing on a target region based on a local contrast adjustment enhancement technology, and obtaining an enhanced image; monitoring the blood flow state of parathyroid glands in real time, constructing a parathyroid gland function protection and blood flow change evaluation model to evaluate the function risk of parathyroid glands in real time; comparing the real-time blood flow change with the existing standard model of electrical impedance change, and when the monitoring data exceeds the set threshold, an alarm is triggered to remind the operation team of the possible damage risk of the parathyroid gland region, and meanwhile, the protection measures are automatically activated according to the parathyroid gland function risk evaluation model. Through spectral navigation, the tumor positioning in the operation process and the protection of important surrounding tissues are effectively guaranteed, and the problems of incomplete cutting and accidental injury are avoided.

Description

Technical Field

[0001] This invention belongs to the field of multimodal fluorescence imaging, and particularly relates to a multimodal fluorescence imaging method and system for thyroid surgery. Background Technology

[0002] Currently, commonly used image-aided techniques in thyroid surgery include traditional endoscopic imaging, fluorescence imaging, and frozen section techniques. However, each method has its limitations. First, traditional endoscopic systems can only provide routine images of organs and tissues in the surgical field. Surgeons typically rely on visual judgment of tissue morphology, color, and anatomical structure, but their ability to identify small tumors and metastatic lymph nodes is limited. In particular, early-stage cancerous or micrometastatic lymph nodes are often missed, leading to incomplete resection. Second, current fluorescence imaging techniques mainly rely on exogenous fluorescent contrast agents. While these agents can enhance image contrast to some extent, they may cause allergic reactions, toxicity, retention, and contamination of the surgical field. Furthermore, the effectiveness of exogenous fluorescent contrast agents in identifying certain tissues (such as parathyroid glands and small tumors) is low, especially in the early identification of small tumors. Third, while frozen section techniques can analyze tissue samples in real time, they require a certain amount of operation time and may cause interference during surgery, failing to provide sufficiently accurate immediate feedback. While some advanced technologies (such as nanocarbon tracers) provide intraoperative tissue color contrast, their safety, procedural standardization, and contamination issues remain highly controversial, limiting their clinical application. Therefore, existing endoscopic systems are poorly suited for tasks such as protecting small thyroid tumors, metastatic lymph nodes, parathyroid glands, and monitoring blood vessels, easily leading to serious problems such as accidental damage to normal tissue and missed metastatic lymph nodes during surgery.

[0003] Furthermore, although spectral imaging techniques (such as narrowband spectral imaging and multispectral imaging) have seen some application in thyroid surgery in recent years, these techniques mostly rely on a single imaging method and lack the ability to flexibly adapt to different tissue characteristics. Existing systems largely employ fixed imaging techniques, making it difficult to adaptively adjust to the spectral characteristics of different tissues during surgery. They also lack the ability to perform multi-dimensional joint identification and real-time assessment of multiple tissue types (such as tumors, parathyroid glands, and adipose tissue). This limitation leads to shortcomings in intraoperative localization, precise resection, and functional preservation of existing surgical endoscopic systems. Therefore, it is crucial to address these issues through more intelligent, multimodal image fusion and navigation systems. Summary of the Invention

[0004] The purpose of this invention is to propose a multimodal fluorescence imaging method and system for thyroid surgery. By integrating adaptive intelligent spectral imaging, multispectral collaborative navigation and real-time functional assessment, it has achieved a breakthrough in the precise localization of parathyroid function.

[0005] To achieve the above objectives, the present invention provides a multimodal fluorescence imaging method for thyroid surgery, the method comprising:

[0006] S1. Simultaneously acquire image data in the infrared, near-infrared and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, perform preliminary synthesis of the preprocessed images to generate a multispectral image set, and optimize the multispectral image set to output the optimized image set.

[0007] S2. Perform Fourier transform on the optimized image set to obtain the frequency domain representation. Then, design weighting strategies for the low-frequency and high-frequency parts of each band to perform frequency domain image fusion. Based on the fused frequency domain image, introduce a deep convolutional neural network model to perform target detection and localization on the fused image. After localization, perform fine enhancement processing on the target area based on local contrast adjustment enhancement technology to further improve the contrast and clarity of the target area and obtain the enhanced image.

[0008] S3. Real-time monitoring of parathyroid blood flow status, construction of parathyroid function protection and blood flow change assessment model. Blood flow change is assessed based on blood flow change measurement and the rate of change of electrical impedance is used to assess parathyroid function status. Based on blood flow change and parathyroid function status, a multispectral image and fluid dynamics model is constructed to calculate local blood flow velocity field and local blood flow in real time. Based on blood flow change measurement, rate of change of electrical impedance and local blood flow, a parathyroid function risk assessment model is established to assess parathyroid function risk in real time.

[0009] S4. Based on the parathyroid function risk assessment model, combined with the impact of parathyroid blood flow change measurement, electrical impedance change rate and local blood flow on parathyroid injury, a comprehensive risk assessment index is designed to comprehensively assess the risk of parathyroid injury. An automatic decision-making mechanism is established based on the comprehensive risk assessment index to determine whether the parathyroid gland needs protection. If the comprehensive risk assessment index exceeds the preset threshold, a protection recommendation is fed back to the surgical team.

[0010] S5. The real-time blood flow changes and electrical impedance changes during surgery are compared with existing standard models. An alarm is triggered by setting a threshold. When the monitored data exceeds the set threshold, an alarm is triggered to remind the surgical team of the possible risk of damage to the parathyroid gland area. At the same time, protective measures are automatically activated based on the parathyroid function risk assessment model.

[0011] Furthermore, the multimodal fluorescence generation method further includes:

[0012] S6. Based on real-time data and surgical results obtained during multiple surgeries on the same person, machine learning algorithms are used to optimize the decision support system and generate customized surgical plans for different patients.

[0013] S7. Design an adaptive adjustment mechanism to dynamically adjust the parathyroid function risk assessment model based on real-time monitoring data and feedback information during the surgical procedure.

[0014] Furthermore, the image preprocessing includes denoising using wavelet transform to ensure the smoothness and detail of the image, and then performing wavelength normalization on the denoised image to ensure data comparability between different bands.

[0015] The optimization process further includes:

[0016] An adaptive spectral selection mechanism based on a deep learning model is introduced. By automatically learning image features, the laser band is adjusted in real time to calculate the optimal wavelength to maximize the imaging effect of the target area. At the same time, a regularization term is introduced into the adaptive spectral selection mechanism. The regularization term reduces the risk of overfitting in band selection, ensuring the stability and robustness of band selection. Finally, the output wavelength of the endoscope laser source is adjusted through a mapping function based on the optimal wavelength, so that the image has the best contrast and resolution in this band.

[0017] The mapping function is an optimized mapping function based on the spectral characteristics of the target tissue.

[0018] Furthermore, the optimized image set is subjected to Fourier transform to obtain a frequency domain representation. Then, a weighting strategy is designed for the low-frequency and high-frequency components of each band to perform frequency domain image fusion, specifically as follows:

[0019]

[0020] Among them, F fused F represents the fused frequency domain image. low (λ i ) and F high (λ i ) represent the frequency transformations of the low-frequency and high-frequency components, respectively, w low (λ i ) and w high (λ i ) is the weighting coefficient for the low-frequency and high-frequency components, N is the total number of bands, and i is the i-th band;

[0021] Specifically, based on the spatial frequency distribution of each band image in a specific region, weighting coefficients w are designed for the low-frequency and high-frequency components. low (λ i ) and w high (λ i ), represented as:

[0022]

[0023] in, and These represent the local contrast and information entropy of the image in band i, respectively, and are used to measure the feature intensity of the image region.

[0024] Furthermore, the step of introducing a deep convolutional neural network model to perform target detection and localization on the fused image based on the fused frequency domain image specifically includes:

[0025] The fused image is input into a pre-trained convolutional neural network, which consists of multiple convolutional layers and pooling layers; the output of the pre-trained convolutional neural network is the bounding box coordinates of the target region, i.e., the precise location and size of the region of interest.

[0026] Meanwhile, a spatial consistency regularization term is introduced during the training of the convolutional neural network to encourage the network to maintain consistent localization results when processing images of multiple bands, thereby avoiding inaccurate localization due to differences between different bands.

[0027] Furthermore, after image localization, the target region is refined and enhanced using adaptive local histogram equalization technology, specifically as follows:

[0028] I enhanced (P target ) = I fused (P target )·ALHE(P target ),

[0029] Among them, I enhanced (P target ) represents the enhanced image, ALHE(P) target ) is based on the target region P target The calculation of local histogram equalization operation.

[0030] Furthermore, the blood flow change measurement is quantified by a blood flow change index, namely the blood flow change index F. bio The calculation is as follows:

[0031]

[0032] Among them, Z pre and Zpost It refers to the electrical impedance values ​​of the parathyroid region before and after surgery; w i These are the weighting coefficients for each vascular region; v i dA is the velocity component of the blood flow velocity field in a specific region; dA is the area of ​​a small element on the surface of the blood vessel; N is the number of blood vessel regions.

[0033] The rate of change of electrical impedance ΔZ bio The calculation is as follows:

[0034]

[0035] Among them, Z max and Z min These are the maximum and minimum electrical impedance values ​​during the monitoring period, respectively;

[0036] The method is based on multispectral images and fluid dynamics models.

[0037]

[0038] Among them, Q local For local blood flow, v local (x) represents the local blood flow velocity field; dA represents the area of ​​a small element on the cross-section of the blood vessel; V represents the volume of the blood vessel within the parathyroid region;

[0039] Simultaneously, a blood flow weighting function w is introduced. flow (λ), adjusting the blood flow signal in the image according to the vascular display characteristics of different wavelength bands:

[0040]

[0041] Among them, Contrast λ The local contrast of each band image; λ is the band number;

[0042] The parathyroid function risk assessment model is expressed as follows:

[0043] R risk =γ1·F bio +γ2·Q flow +γ3·ΔZ bio

[0044] Among them, R risk γ1, γ2, and γ3 are risk scores and weighting coefficients, respectively.

[0045] Furthermore, the automatic activation of protection measures based on the decision support model specifically includes:

[0046] The surgical procedure is dynamically adjusted through feedback control by combining blood flow and electrical impedance data. The specific control formula is as follows:

[0047]

[0048] Where u(t) is the control signal, representing the surgical strategy that needs to be adjusted; e(t) is the error function, representing the deviation between the monitoring data and the set standard; K p ,K d ,K i These are the proportional, derivative, and integral gain coefficients, reflecting the adjustment strength of the feedback system.

[0049] Furthermore, the customized surgical plan uses a deep learning model to automatically recommend surgical strategies based on the patient's specific characteristics.

[0050] In a second aspect of the invention, a multimodal fluorescence imaging system for thyroid surgery is provided, the system comprising:

[0051] The multimodal image acquisition unit is used to simultaneously acquire image data in the infrared, near-infrared and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, perform preliminary synthesis of the preprocessed images to generate a multispectral image set, and perform optimization processing on the multispectral image set to output the optimized image set.

[0052] The image enhancement unit performs Fourier transform on the optimized image set to obtain a frequency domain representation. Then, it designs weighting strategies for the low-frequency and high-frequency components of each band to perform frequency domain image fusion. Based on the fused frequency domain image, a deep convolutional neural network model is introduced to perform target detection and localization on the fused image. After localization is completed, enhancement technology based on local contrast adjustment is used to perform fine enhancement processing on the target area to further improve the contrast and clarity of the target area, resulting in an enhanced image.

[0053] The real-time data monitoring unit for thyroid surgery is used to monitor the blood flow status of the parathyroid glands in real time. It constructs a parathyroid function protection and blood flow change assessment model. It assesses blood flow changes based on blood flow change measurement and evaluates parathyroid function status based on impedance change rate. Based on blood flow changes and parathyroid function status, it constructs a multispectral image and fluid dynamics model to calculate the local blood flow velocity field and local blood flow in real time. Based on blood flow change measurement, impedance change rate, and local blood flow, it establishes a parathyroid function risk assessment model to assess parathyroid function risk in real time.

[0054] The thyroid surgery risk assessment unit is used to design comprehensive risk assessment indicators based on the parathyroid function risk assessment model, combined with the impact of parathyroid blood flow change measurement, electrical impedance change rate and local blood flow on parathyroid injury. It is used to comprehensively assess the risk of parathyroid injury, establish an automatic decision-making mechanism based on the comprehensive risk assessment indicators, determine whether the parathyroid gland needs protection, and if the comprehensive risk assessment indicators exceed the preset threshold, it will provide protection recommendations to the surgical team.

[0055] The early warning unit is used to compare real-time blood flow changes and electrical impedance changes during surgery with existing standard models. It triggers an alarm by setting a threshold. When the monitored data exceeds the set threshold, an alarm is triggered to remind the surgical team of the potential risk of damage to the parathyroid gland area. At the same time, it automatically activates protective measures based on the parathyroid function risk assessment model.

[0056] The beneficial technical effects of the present invention are at least as follows:

[0057] This invention achieves groundbreaking progress in the precise localization of tumors, metastatic lymph nodes, and parathyroid function protection by integrating adaptive intelligent spectral imaging, multispectral collaborative navigation, and real-time functional assessment. The core innovation of the system lies in its ability to intelligently select and adjust the optimal spectral excitation band based on real-time intraoperative images and tissue characteristics, thereby providing high-quality multispectral imaging for different tissue types and significantly improving the resolution and contrast between the tumor and surrounding normal tissues. Furthermore, the system combines near-infrared imaging and photoacoustic imaging technologies to monitor the blood flow of the thyroid and parathyroid glands in real time and assess their functional status through dynamic imaging, providing physicians with real-time decision support and reducing parathyroid damage and postoperative complications.

[0058] The adaptive spectral imaging technology of this invention significantly improves the differentiation between tumors and normal tissues by dynamically selecting the excitation source through real-time analysis of the spectral response of different tissues, making it particularly suitable for the precise identification of small lesions. Compared with existing fluorescence imaging technologies, it can obtain clearer imaging results through endogenous fluorescence and its own spectral characteristics without relying on exogenous contrast agents, thereby overcoming the safety, standardized operation, and contamination problems of traditional contrast agents.

[0059] Furthermore, the system's multispectral collaborative navigation function, by integrating near-infrared and photoacoustic imaging technologies, precisely locates the thyroid and parathyroid glands and monitors the blood flow status of the parathyroid glands in real time. This ensures that the function of the parathyroid glands is effectively protected during tumor resection, reducing the risk of hypoparathyroidism. Through this multispectral navigation system, tumor localization and the protection of surrounding vital tissues during surgery are effectively guaranteed, avoiding problems such as missed resections and accidental damage. Attached Figure Description

[0060] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0061] Figure 1 This is a flowchart of a multimodal fluorescence imaging method for thyroid surgery according to the present invention. Detailed Implementation

[0062] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0063] like Figure 1 As shown in the embodiment of the present invention, a multimodal fluorescence imaging method for thyroid surgery is provided, the method comprising the following:

[0064] S1. Simultaneously acquire image data in the infrared, near-infrared and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, perform preliminary synthesis of the preprocessed images to generate a multispectral image set, and optimize the multispectral image set to output the optimized image set.

[0065] Specifically, the input for this step is the real-time image data captured by the endoscopic system during the procedure. We first use a multi-wavelength endoscopic system to simultaneously acquire image data in the infrared, near-infrared, and visible light bands, denoted as I(λ). i ), where λ i Here, λ represents the wavelength of different bands, and λi denotes the wavelength index (e.g., λ1 represents the infrared band, λ2 represents the near-infrared band, and λ3 represents the visible light band). An image fusion algorithm is used to initially synthesize the images from each band, generating a multispectral image set. And perform the following image preprocessing:

[0066] Denoising: Wavelet transform is used to remove high-frequency noise from the image, ensuring smoothness and detail. The image is then processed using the following formula:

[0067] I′(λ i ) = Wavelet(I(λ) i ))

[0068] Where, I′(λ) i ) represents the denoised image, and Wavelet(·) is the wavelet transform function.

[0069] Normalization: Wavelength normalization is performed on the denoised image to ensure data comparability between different bands. Normalization is performed using the following formula:

[0070]

[0071] Where, μ λi For band λ i The mean, For band λ i The standard deviation, I′(λ) i () is the normalized image.

[0072] The purpose of this preprocessing step is to improve data quality and lay the foundation for subsequent image optimization and tissue identification.

[0073] Furthermore, during thyroid cancer surgery, different tissues (such as tumors, parathyroid glands, and lymph nodes) exhibit significant differences in their reflection or emission spectral characteristics across different spectral bands. Existing imaging methods typically rely on fixed spectral bands, which can lead to poor identification or even omission of some key tissues (such as small tumors and lymph nodes).

[0074] To address this issue, we introduce an adaptive spectral selection mechanism. This mechanism, based on a deep learning model, automatically learns image features and adjusts the laser band in real time to maximize the imaging effect on the target area. We define a deep neural network model M(·), which outputs an optimal wavelength λ based on the spectral characteristics of each image band. opt :

[0075] λ opt =M(I′(λ) i ))

[0076] Where M(·) is a trained deep learning model that can process the input image data Features of the target tissue (such as tumors, parathyroid glands, etc.) are extracted, and the most suitable laser wavelength λ is calculated. opt .

[0077] To further optimize the laser effect, we introduced a regularization term during the band selection process, taking into account the similarity of different tissues and the impact of data noise. This regularization term... By reducing the risk of overfitting in band selection, the stability and robustness of band selection are ensured. The formula is expressed as:

[0078]

[0079] Among them, I target It is an ideal image of the target organization. It is a regularization term, and the specific design is as follows:

[0080]

[0081] in, Indicates band λ i The gradient of the image, α is the regularization strength parameter. The introduction of this regularization term can effectively reduce noise, enhance the contrast between the real target and the background, and thus improve the recognition accuracy of target tissues.

[0082] Furthermore, the optimal band λ calculated based on the above steps... opt We further adjusted the output wavelength of the endoscopic laser source to achieve optimal contrast and resolution in this wavelength band. This optimization step was achieved through the following mapping function:

[0083] I opt (λ opt )=f(I′(λ opt ))

[0084] Among them, I opt (λ opt ) is the optimized image, and f(·) is the mapping function optimized based on the spectral characteristics of the target tissue (such as tumor, thyroid, etc.).

[0085] Through this optimization process, we can obtain high-quality images optimized for different tissue types in real time during surgery, especially in the identification of small tumors, metastatic lymph nodes, and parathyroid glands, thereby improving the accuracy and safety of the surgery.

[0086] S2. Perform Fourier transform on the optimized image set to obtain the frequency domain representation. Then, design weighting strategies for the low-frequency and high-frequency parts of each band to perform frequency domain image fusion. Based on the fused frequency domain image, introduce a deep convolutional neural network model to perform target detection and localization on the fused image. After localization is completed, perform fine enhancement processing on the target area based on local contrast adjustment enhancement technology to further improve the contrast and clarity of the target area and obtain the enhanced image.

[0087] Specifically, multispectral image fusion is the core step in this process. Its purpose is to improve the overall visual quality and information content of the image by fusing information from different spectral bands, especially enhancing the features of the target region. To achieve this goal, we employ a method called Weighted Frequency Domain Optimized Fusion (WFD-OF).

[0088] Traditional image fusion methods typically rely on optimization in either the spatial or frequency domains, but these methods often fail to adequately balance the fusion accuracy across different spectral bands. Our proposed scheme considers the local correlation of spectral information and the complementarity of information between different bands. Therefore, we weight the low-frequency and high-frequency components of each band image separately in the frequency domain.

[0089] Specifically, firstly, for each band image I opt (λ i Perform a Fourier transform to obtain the frequency domain representation F. opt (λ i Then, weighting strategies are designed for the low-frequency and high-frequency components of each band, comprehensively considering the spatial and spectral information of each band image to obtain the weighted frequency domain expression. The mathematical formula for fusion is expressed as:

[0090]

[0091] Among them, F fused F represents the fused frequency domain image. low (λ i ) and F high (λ i ) represent the frequency transformations of the low-frequency and high-frequency components, respectively, w low (λ i ) and w high (λ i () is a weighting coefficient for the low-frequency and high-frequency components.

[0092] Weighting coefficient w low (λ i ) and w high (λ i The design takes into account the spatial frequency distribution of each band image in a specific region. We introduce local contrast of the band images. and local information entropy λi To calculate the weighting coefficients, this effectively emphasizes high-frequency information in information-dense regions of the image while maintaining the smoothness of low-frequency regions.

[0093]

[0094] in, and These represent the local contrast and information entropy of the image in band i, respectively, and are used to measure the feature intensity of the image region.

[0095] By using these weighting schemes, we not only ensure the accuracy of image fusion, but also effectively avoid information loss and enhance the feature expression of target regions (such as tumors, thyroid glands, etc.).

[0096] Furthermore, after image fusion is completed, the next step is to automatically locate the target region in the fused image. For this purpose, we introduce a deep convolutional neural network (CNN) model L... loc (·), This model is used for target detection and localization in fused images. This model is specifically designed and trained to automatically identify and label key regions in images, especially target regions with complex structures and morphologies, such as tumors or thyroid glands.

[0097] First, we will fuse the image I fused The input is fed into a pre-trained convolutional neural network, which consists of multiple convolutional and pooling layers. The network output is the bounding box coordinates P of the target region. target This refers to the precise location and size of the region of interest (ROI).

[0098] The specific positioning process can be represented as follows:

[0099] P target =L loc (I fused )

[0100] Among them, P target =(x min ,y min ,x max ,y max ) represents the four coordinates of the output target region bounding box, L loc (·) is a deep learning model used to locate target regions.

[0101] To improve positioning accuracy and stability, we introduced a spatial consistency regularization term during training. This regularization term encourages the network to maintain consistent localization results when processing images across multiple bands, thereby avoiding inaccurate localization due to differences between different bands. This regularization term can be expressed as:

[0102]

[0103] Where λ is the regularization strength, This is the gradient of the bounding box coordinates of the target region. The purpose of this regularization term is to improve the accuracy of localization by constraining the smoothness of the target region boundary.

[0104] Furthermore, after image localization, we perform refined enhancement processing on the target region to further improve its contrast and clarity. This step employs an enhancement technique based on local contrast adjustment, which makes important target regions in the image stand out more while reducing background noise interference. Specifically, we use the "Adaptive Local Histogram Equalization" (ALHE) technique, which automatically adjusts the contrast according to the spatial characteristics of the target region.

[0105] This enhancement process is implemented using a local region histogram equalization algorithm, and can be represented as:

[0106] I enhanced (P target ) = I fused (P target )·ALHE(P target )

[0107] Among them, I enhanced (P target ) represents the enhanced image, ALHE(P) target ) is based on the target region P target The calculation involves local histogram equalization. This step significantly improves the visualization of the target region, providing clearer information for the surgeon's decision-making during the procedure.

[0108] S3. Real-time monitoring of parathyroid blood flow status, construction of a parathyroid function protection and blood flow change assessment model. Blood flow change is assessed based on blood flow change measurement and the rate of change of electrical impedance is used to assess parathyroid function status. Based on blood flow change and parathyroid function status, a multispectral image and fluid dynamics model is constructed to calculate local blood flow velocity field and local blood flow in real time. Based on blood flow change measurement, rate of change of electrical impedance and local blood flow, a parathyroid function risk assessment model is established to assess parathyroid function risk in real time.

[0109] Specifically, in the preceding steps, through image fusion and precise localization, we have obtained multispectral images and spatial information of the parathyroid gland and accurately located the target area. Next, our task is to monitor the blood flow status of the parathyroid gland in real time and assess its functional changes in order to provide timely decision support during the procedure and prevent parathyroid gland dysfunction or insufficient blood perfusion.

[0110] Furthermore, a model for assessing parathyroid function protection and blood flow changes:

[0111] Input: Multispectral image data and electrical impedance monitoring data output from the previous step.

[0112] Output: Measurement of blood flow changes and electrical impedance changes in the parathyroid region.

[0113] Specifically, blood flow changes are assessed by combining electrical impedance values ​​with blood flow velocity fields. A blood flow change index F is defined using vascular features extracted from multispectral images and real-time blood flow velocity fields. bio :

[0114]

[0115] Among them, Z pre and Z post These are the electrical impedance values ​​of the parathyroid gland region before and after surgery. i These are the weighting coefficients for each vascular region. i dA is the velocity component of the blood flow velocity field in a specific region. dA is the area of ​​a small element on the surface of the blood vessel. N is the number of vascular regions. This formula combines vascular features from an image with electrical impedance data to accurately assess changes in parathyroid blood flow.

[0116] Furthermore, the rate of change of electrical impedance ΔZ bio Used to assess parathyroid function status:

[0117]

[0118] Among them, Z max and Z min These represent the maximum and minimum electrical impedance values ​​during the monitoring period. Changes in electrical impedance are directly related to the blood flow and tissue state of the parathyroid gland, and can reflect its health status in real time.

[0119] Furthermore, a real-time blood flow monitoring and feedback mechanism:

[0120] Input: Blood flow change data and electrical impedance change data from the previous step.

[0121] Output: Real-time blood flow monitoring results and feedback.

[0122] Specifically, based on multispectral images and fluid dynamics models, the local blood flow velocity field v is calculated in real time. local and traffic Q local The definition is as follows:

[0123]

[0124] Among them, v local (x) represents the local blood flow velocity field; dA is the area of ​​a small element on the cross-section of the blood vessel; V is the volume of the blood vessel within the parathyroid region. Using the fluid dynamics equations, we can accurately describe the dynamic changes in blood flow, especially the blood flow state in the parathyroid region.

[0125] Furthermore, a blood flow weighting function w is introduced. flow (λ), adjusting the blood flow signal in the image according to the vascular display characteristics of different wavelength bands:

[0126]

[0127] Among them, Contrast λ This represents the local contrast of each band image; λ is the band number. This weighting function helps us accurately detect blood flow signals and enhances the visualization of blood flow information.

[0128] Furthermore, based on the blood flow and impedance change data from the first two steps, a parathyroid function protection decision and real-time feedback are output.

[0129] Specifically, based on blood flow change measurements, electrical impedance change measurements, and real-time blood flow data, a parathyroid function risk assessment model R is established. risk :

[0130] R risk =γ1·F bio +γ2·Q flow +γ3·ΔZ bio

[0131] Where γ1, γ2, γ3 are weighting coefficients; F bio Q is a comprehensive index of electrical impedance and blood flow velocity field; flow Blood flow; ΔZ bio R is the rate of change of electrical impedance. Through analysis of real-time data, R... risk It can assess the risk of parathyroid dysfunction in real time and provide doctors with decision support for protective measures.

[0132] Furthermore, if R risk If the threshold is exceeded, the system will automatically provide real-time feedback, prompting the doctor whether protective measures are needed. By combining blood flow data, impedance changes, and image information, the system provides efficient real-time monitoring and decision support for surgery.

[0133] S4. Based on the parathyroid function risk assessment model, and combined with the effects of parathyroid blood flow change measurement, electrical impedance change rate and local blood flow on parathyroid injury, a comprehensive risk assessment index is designed to comprehensively assess the risk of parathyroid injury. An automatic decision-making mechanism is established based on the comprehensive risk assessment index to determine whether the parathyroid glands need protection. If the comprehensive risk assessment index exceeds the preset threshold, a protection recommendation is fed back to the surgical team.

[0134] Specifically, based on the blood flow and impedance change data obtained in the previous stage, and combined with the protection requirements of the parathyroid glands, a comprehensive risk assessment model was designed. This model considers the impact of changes in blood flow, impedance, and local blood flow in the parathyroid region on parathyroid gland injury.

[0135] We define a comprehensive risk assessment index R. total Its form is a weighted sum:

[0136] R total =α1·F bio +α2·ΔZ bio +α3·Q local +α4·w flow

[0137] Where α1, α2, α3, and α4 are weighting coefficients, representing the importance of different input data to the overall risk assessment; F bio ΔZ is a measure of blood flow changes in the parathyroid region. bio Q is a measure of the change in electrical impedance in the parathyroid region. local Local blood flow in the parathyroid region; w flow The blood flow weighting coefficient reflects the overall importance of blood flow. This risk assessment model comprehensively evaluates the risk of parathyroid gland injury by weighting the importance of different data sources. This comprehensive assessment index provides a quantitative basis for subsequent decision support systems.

[0138] Furthermore, based on the comprehensive risk assessment index R total We established an automated decision-making mechanism to determine whether the parathyroid glands need protection. If R total Exceeding the preset threshold R threshold The system will provide protection suggestions to the surgical team:

[0139]

[0140] Among them, R total The comprehensive risk assessment value is calculated in real time; R threshold This threshold, determined through historical surgical data and clinical experience, represents the risk threshold for parathyroid gland injury. This decision support system uses real-time calculations to determine whether parathyroid protection measures are necessary, providing the surgical team with scientific decision support based on real-time data.

[0141] S5. The real-time blood flow changes and electrical impedance changes during surgery are compared with existing standard models. An alarm is triggered by setting a threshold. When the monitored data exceeds the set threshold, an alarm is triggered to remind the surgical team of the possible risk of damage to the parathyroid gland area. At the same time, protective measures are automatically activated based on the parathyroid function risk assessment model.

[0142] Specifically, based on the decision support results from the previous stage and real-time data during the operation (such as impedance and blood flow changes), real-time feedback is calculated to ensure that protective measures for the parathyroid glands are implemented during the operation.

[0143] Furthermore, by comparing real-time blood flow changes with electrical impedance changes using existing standard models, an alarm is triggered based on a set threshold.

[0144]

[0145] Among them, F ref The standard blood flow change value is used as a reference value; ΔZ ref The standard impedance change value is used as a reference value; ∈ bio and ∈ Z The set tolerance threshold represents the normal fluctuation range of blood flow and electrical impedance.

[0146] When monitoring data exceeds a set threshold, the system triggers an alarm, alerting the surgical team to potential damage risks to the parathyroid gland area. This mechanism provides precise feedback for real-time monitoring during the surgical procedure.

[0147] Furthermore, if an alarm is triggered, the system will automatically activate protective measures based on a decision support model. For example, if the impedance value is too high or the blood flow changes abnormally, the system can reduce the risk of damage to the parathyroid glands by controlling surgical instruments or adjusting medications in real time.

[0148] To respond quickly and automatically execute protective measures, the system dynamically adjusts the surgical procedure through feedback control, combining blood flow and electrical impedance data. The specific control formula is as follows:

[0149]

[0150] Where u(t) is the control signal, representing the surgical strategy that needs to be adjusted (such as adjusting the current, drug delivery volume, etc.); e(t) is the error function, representing the deviation between the monitoring data and the set standard; K p ,K d ,K i These are the proportional, derivative, and integral gain coefficients, reflecting the adjustment strength of the feedback system.

[0151] S6. Based on real-time data and surgical results obtained during multiple surgeries on the same person, machine learning algorithms are used to optimize the decision support system and generate customized surgical plans for different patients.

[0152] Specifically, based on real-time data acquired during the surgery and the surgical outcome, machine learning algorithms (such as deep neural networks or reinforcement learning) are used to optimize the decision support system. By learning from historical data, the weighting coefficients in the system are adjusted to better predict the risk of parathyroid gland damage.

[0153] We update the weighting coefficients using the following optimized formula:

[0154]

[0155] Where, α i η is the weighting coefficient used to adjust the weights of different data sources; η is the learning rate. The loss function is calculated based on the error between the predicted and actual results. This is the gradient of the loss function with respect to the weighting coefficients.

[0156] Through continuous optimization, the weighting coefficients are adjusted based on surgical feedback from different patients, thereby improving the predictive accuracy of the system.

[0157] Furthermore, each patient's situation is different, therefore the decision support system needs to generate customized surgical plans based on the patient's individual data (such as parathyroid gland location, blood flow patterns, etc.). We use a deep learning model to automatically recommend surgical strategies based on the patient's specific characteristics. Personalized recommendations can be modeled using the following formula:

[0158]

[0159] in, Personalized surgical recommendations; x patient The model incorporates individual patient characteristics (such as parathyroid gland location, blood flow pattern, and electrical impedance); θ represents the model parameters, trained based on historical surgical data. Personalized recommendations not only improve the efficiency of parathyroid gland protection but also optimize surgical treatment plans for each patient.

[0160] S7. Design an adaptive adjustment mechanism to dynamically adjust the parathyroid function risk assessment model based on real-time monitoring data and feedback information during the surgical procedure.

[0161] To address unforeseen circumstances in various surgical scenarios, the system employs an adaptive adjustment mechanism, dynamically adjusting the decision support model based on real-time monitoring data and feedback during the surgical procedure. This mechanism ensures that the system can respond quickly and optimize operations in dynamic surgical environments.

[0162] Adaptive adjustment is achieved using the following feedback control formula:

[0163]

[0164] Where u(t) is the adjusted control signal, indicating the surgical protection strategy to be taken; e(t) is the current error, i.e., the difference between the real-time data and the expected target; K p ,K d ,K i The proportional, derivative, and integral gains determine the controller's response speed and sensitivity.

[0165] In another embodiment of the present invention, a multimodal fluorescence imaging system for thyroid surgery is provided, the system comprising:

[0166] The multimodal image acquisition unit is used to simultaneously acquire image data in the infrared, near-infrared and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, perform preliminary synthesis of the preprocessed images to generate a multispectral image set, and perform optimization processing on the multispectral image set to output the optimized image set.

[0167] The image enhancement unit performs Fourier transform on the optimized image set to obtain a frequency domain representation. Then, it designs weighting strategies for the low-frequency and high-frequency components of each band to perform frequency domain image fusion. Based on the fused frequency domain image, a deep convolutional neural network model is introduced to perform target detection and localization on the fused image. After localization is completed, enhancement technology based on local contrast adjustment is used to perform fine enhancement processing on the target area to further improve the contrast and clarity of the target area, resulting in an enhanced image.

[0168] The real-time data monitoring unit for thyroid surgery is used to monitor the blood flow status of the parathyroid glands in real time. It constructs a parathyroid function protection and blood flow change assessment model. It assesses blood flow changes based on blood flow change measurement and evaluates parathyroid function status based on impedance change rate. Based on blood flow changes and parathyroid function status, it constructs a multispectral image and fluid dynamics model to calculate the local blood flow velocity field and local blood flow in real time. Based on blood flow change measurement, impedance change rate, and local blood flow, it establishes a parathyroid function risk assessment model to assess parathyroid function risk in real time.

[0169] The thyroid surgery risk assessment unit is used to design comprehensive risk assessment indicators based on the parathyroid function risk assessment model, combined with the impact of parathyroid blood flow change measurement, electrical impedance change rate and local blood flow on parathyroid injury. It is used to comprehensively assess the risk of parathyroid injury, establish an automatic decision-making mechanism based on the comprehensive risk assessment indicators, determine whether the parathyroid gland needs protection, and if the comprehensive risk assessment indicators exceed the preset threshold, it will provide protection recommendations to the surgical team.

[0170] The early warning unit is used to compare real-time blood flow changes and electrical impedance changes during surgery with existing standard models. It triggers an alarm by setting a threshold. When the monitored data exceeds the set threshold, an alarm is triggered to remind the surgical team of the potential risk of damage to the parathyroid gland area. At the same time, it automatically activates protective measures based on the parathyroid function risk assessment model.

[0171] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0172] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0173] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-modal fluorescence imaging system for application in thyroid surgery, characterized in that, The system comprises: A multi-modal image acquisition unit for simultaneously acquiring image data in the infrared, near-infrared and visible light bands using a multi-wavelength endoscope system, pre-processing the image data, preliminarily synthesizing the pre-processed images to generate a multi-spectral image set, and optimizing the multi-spectral image set to output an optimized image set; An image enhancement unit for performing Fourier transform on the optimized image set to obtain a frequency domain representation, then designing a weighting strategy for the low-frequency and high-frequency parts of each band respectively to perform frequency domain image fusion, introducing a deep convolutional neural network model based on the fused frequency domain images to perform target detection and positioning, and after positioning is completed, performing fine enhancement processing on the target region based on a local contrast adjustment enhancement technique to further improve the contrast and clarity of the target region, to obtain an enhanced image; A thyroid surgery real-time data monitoring unit for monitoring the blood flow state of the parathyroid gland in real time, constructing a parathyroid function protection and blood flow change evaluation model, evaluating blood flow changes according to the blood flow change metric, and evaluating parathyroid function state according to the electrical impedance change rate, constructing a multi-spectral image and fluid dynamics model based on the blood flow change and parathyroid function state, calculating the local blood flow velocity field and local blood flow in real time, and establishing a parathyroid function risk evaluation model based on the blood flow change metric, electrical impedance change rate and local blood flow to evaluate the parathyroid function risk in real time; A thyroid surgery risk assessment unit for designing a comprehensive risk assessment index based on the parathyroid function risk evaluation model, combining the influence of parathyroid injury on the parathyroid blood flow change metric, electrical impedance change rate and local blood flow, to comprehensively evaluate the parathyroid injury risk, establishing an automatic decision-making mechanism based on the comprehensive risk assessment index to determine whether the parathyroid needs protection, and feeding back protection suggestions to the surgical team if the comprehensive risk assessment index exceeds the preset threshold; An early warning unit for comparing the real-time blood flow change and electrical impedance change during surgery with the existing standard model, triggering an alarm through a set threshold, and reminding the surgical team of the possible parathyroid injury risk when the monitoring data exceeds the set threshold, and automatically activating protection measures according to the parathyroid function risk evaluation model; The blood flow change measure is quantified by a blood flow change indicator The calculation is as follows: ； wherein, and is the electrical impedance value of the parathyroid region before and after surgery; is the weighted coefficient of each blood vessel region; is the velocity component of the blood flow velocity field in a specific region; is the microelement area of the blood vessel surface; is the number of blood vessel regions; The rate of change of electrical impedance , calculated as follows: ； wherein, and Zmax and Zmin are the maximum and minimum electrical impedance values during the monitoring period, respectively; The multi-spectral image and fluid dynamics model is based on the blood flow change metric and the electrical impedance change rate, and the parathyroid function risk evaluation model is represented as: ； wherein, is the local blood flow, is the local blood flow velocity field; is the microelement area on the vessel cross-section; is the volume of blood vessels within the parathyroid region; Simultaneously introducing a blood flow weighting function , adjusting blood flow signals in the image according to blood vessel display characteristics of different wavebands: ； wherein, is the local contrast for each band image; is the number of bands. The comprehensive risk assessment index is represented as: ； wherein, is a risk score, is a weighting factor; is a blood flow; is a blood flow change indicator; is a rate of change of electrical impedance; The automatic activation of protection measures based on the decision support model includes: ； wherein, are weighting coefficients, indicating the importance of different input data to the overall risk assessment; Combining blood flow and electrical impedance data to dynamically adjust the surgical process through feedback control, and the specific control formula is: The multi-modal fluorescence imaging system also performs the following functions: ； wherein, is a control signal, indicating the surgical strategy that needs to be adjusted; is an error function, indicating the deviation between the monitoring data and the set standard; are proportional, derivative and integral gain coefficients, reflecting the adjustment strength of the feedback system.

2. The multi-modal fluorescence imaging system for thyroid surgery according to claim 1, wherein, According to the real-time data and surgical results obtained during multiple surgeries of the same person, the decision support system is optimized using machine learning algorithms, and customized surgical plans are generated for different patients; An adaptive adjustment mechanism is designed to dynamically adjust the parathyroid function risk evaluation model based on real-time monitoring data and feedback information during the surgical process. ​ 3.The multi-modal fluorescence imaging system applied to thyroid surgery of claim 1, wherein, The image preprocessing includes denoising processing by wavelet transform to ensure the smoothness and details of the image, and then wavelength normalization is performed on the denoised image to ensure the comparability of data between different wavebands; The optimization processing further includes: An adaptive spectrum selection mechanism based on a deep learning model is introduced, the optimal wavelength is calculated in real time by automatically learning the image features to maximize the imaging effect of the target region, and a regularization term is introduced in the adaptive spectrum selection mechanism, which reduces the overfitting risk of waveband selection and ensures the stability and robustness of waveband selection, and finally the output wavelength of the endoscopic laser source is adjusted based on the optimal wavelength through a mapping function, so that the image has the best contrast and resolution at this waveband. The mapping function is a mapping function optimized according to the spectral characteristics of the target tissue. 4.The multi-modal fluorescence imaging system applied to thyroid surgery of claim 1, wherein, The optimized image set is subjected to Fourier transform to obtain a frequency domain representation, and then a weighting strategy is designed for the low-frequency part and the high-frequency part of each waveband for frequency domain image fusion, which is specifically represented as: ； wherein, denotes the fused frequency domain image, and denotes the frequency transform of the low and high frequency parts, respectively, and are weighting coefficients for the low and high frequency parts, N is the total number of wavebands, and i is the i-th waveband. Among them, based on the spatial frequency distribution of each wave band image in a certain area, the weighting coefficients for the low frequency and high frequency parts are designed and , is expressed as: ； wherein and respectively represent the waveband local contrast and information entropy of the image, which are used to measure the feature intensity of the image region.

5. The multi-modal fluorescence imaging system for thyroid surgery of claim 1, wherein, A deep convolutional neural network model is introduced based on the fused frequency domain image to perform target detection and positioning on the fused image, which specifically includes: The fused image is input into a pre-trained convolutional neural network, wherein the pre-trained convolutional neural network is composed of multiple convolutional layers and pooling layers; the output of the pre-trained convolutional neural network is the boundary box coordinates of the target region, i.e. the accurate position and size of the region of interest; At the same time, a spatial consistency regularization term is introduced in the convolutional neural network training process to encourage the network to maintain consistent positioning results when processing multiple waveband images, thereby avoiding the problem of inaccurate positioning due to differences between different wavebands.

6. The multi-modal fluorescence imaging system for thyroid surgery of claim 5, wherein, After image positioning, an adaptive local histogram equalization technique is used to perform fine enhancement processing on the target region, which specifically includes: ； wherein, represents the enhanced image, is a local histogram equalization operation computed according to the target region ​ 7.The multi-modal fluorescence imaging system applied to thyroid surgery of claim 2, wherein, The customized surgical plan automatically recommends a surgical strategy based on the specific characteristics of the patient through a deep learning model.

Images