Multi-mode fluorescence imaging method and system applied to thyroid surgery

Abstract

The invention provides a multi-mode fluorescence imaging method and system applied to thyroid surgery, and the method comprises the steps: collecting image data through a multi-wavelength endoscope system, and generating an image set; performing Fourier transform on the image set, and performing refined enhancement processing on the target area based on an enhancement technology of local contrast adjustment to obtain an enhanced image; monitoring the blood flow state of the parathyroid gland in real time, and constructing a parathyroid gland function protection and blood flow change evaluation model to evaluate the parathyroid gland function risk in real time; real-time blood flow changes and electrical impedance changes in an operation are compared with an existing standard model, when monitoring data exceed a set threshold value, an alarm is triggered, an operation team is reminded of the possible damage risk of the parathyroid gland area, and meanwhile protection measures are automatically activated according to a parathyroid gland function risk assessment model. Through spectral navigation, tumor positioning and protection of surrounding important tissues in the operation process are powerfully guaranteed, and the problems of missed cutting and accidental injury are avoided.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal fluorescence imaging, and particularly relates to a multimodal fluorescence imaging method and system applied to thyroid surgery. Background Art

[0002] Currently, the commonly used imaging assistance techniques in thyroid surgery include traditional endoscopic imaging, fluorescence imaging, frozen section technique, etc. However, these methods have limitations to varying degrees. First of all, the traditional endoscopic system can only provide conventional images of the organs and tissues in the surgical field. Surgeons usually rely on visual judgment of the morphology, color, and anatomical structure of the tissues, but the ability to identify small tumors and metastatic lymph nodes is limited. In particular, early cancerous changes or small metastatic lymph nodes are often missed, resulting in incomplete resection. Secondly, the current fluorescence imaging technology mainly relies on exogenous fluorescence contrast agents. Although these contrast agents can enhance the image contrast to a certain extent, they may have problems such as allergic reactions, toxicity, retention, and contamination of the surgical field. In addition, the effect of exogenous fluorescence contrast agents has a low identification ability in some tissues (such as parathyroid glands, small tumors), especially the early recognition of small tumors is not sensitive enough. Moreover, although the frozen section technique can analyze tissue samples in real time, it requires a certain operation time and may cause interference during the operation, and cannot provide sufficiently accurate immediate feedback. Although some advanced techniques (such as nano-carbon tracer) provide color contrast of tissues during the operation, there are still great controversies about its safety, operation standardization, and contamination problems, which are also limited in clinical applications. Therefore, the existing endoscopic systems have poor adaptability to tasks such as the protection of small thyroid tumors, metastatic lymph nodes, parathyroid glands, and blood vessel monitoring, and are prone to serious problems such as accidental injury to normal tissues and omission of metastatic lymph nodes during the operation.

[0003] In addition, although spectral imaging techniques (such as narrow-band spectral imaging, multispectral imaging, etc.) have been applied in thyroid surgery in recent years, most of these techniques rely on a single imaging method and lack the ability to flexibly respond to different tissue characteristics. The imaging techniques of the existing systems are mostly fixed, difficult to be adaptively adjusted according to the spectral characteristics of different tissues during the operation, and lack the multi-dimensional joint recognition and real-time evaluation functions for multiple tissue types (such as tumors, parathyroid glands, adipose tissues, etc.). This limitation leads to the deficiencies of the existing surgical endoscopic systems in intraoperative positioning, precise resection, and functional protection. Therefore, it is necessary to solve this problem through a more intelligent, multimodal image fusion and navigation system. Summary of the Invention

[0004] The objective of the present invention is to propose a multimodal fluorescence imaging method and system for thyroid surgery, which has achieved a breakthrough in precisely locating parathyroid function by integrating adaptive intelligent spectral imaging, multispectral collaborative navigation, and real-time functional evaluation.

[0005] To achieve the above objective, the present invention provides a multimodal fluorescence imaging method for thyroid surgery, and the method includes:

[0006] S1. Use a multi-wavelength endoscope system to simultaneously collect image data in the infrared, near-infrared, and visible light bands, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multispectral image set, and perform optimization processing on the multispectral image set to output an optimized image set;

[0007] S2. Perform Fourier transform on the optimized image set to obtain a frequency-domain representation, then design a weighting strategy for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion, introduce a deep convolutional neural network model based on the fused frequency-domain image to perform target detection and localization, and after completing the localization, perform refined enhancement processing on the target area based on an enhancement technique of local contrast adjustment to further improve the contrast and clarity of the target area to obtain an enhanced image;

[0008] S3. Real-time monitor the blood flow status of the parathyroid gland, construct a parathyroid function protection and blood flow change evaluation model to evaluate blood flow changes according to the blood flow change metric and evaluate the parathyroid function status according to the impedance change rate, construct a multispectral image and hydrodynamic model based on the blood flow changes and the parathyroid function status, calculate the local blood flow velocity field and local blood flow in real time, and establish a parathyroid function risk assessment model according to the blood flow change metric, the impedance change rate, and the local blood flow to real-time evaluate the parathyroid function risk;

[0009] S4. According to the parathyroid function risk assessment model, combine the effects of parathyroid blood flow change metric, impedance change rate, and local blood flow on parathyroid injury, design a comprehensive risk assessment index to comprehensively evaluate the parathyroid injury risk, establish an automatic decision-making mechanism based on the comprehensive risk assessment index to judge whether the parathyroid gland needs protection, and if the comprehensive risk assessment index exceeds a preset threshold, feedback a protection suggestion to the surgical team;

[0010] S5. Compare the real-time blood flow changes and impedance changes during the surgery with a standard model, trigger an alarm through a set threshold, and when the monitored data exceeds the set threshold, trigger an alarm to remind the surgical team of the possible injury risk in the parathyroid area, and at the same time automatically activate protection measures according to the parathyroid function risk assessment model.

[0011] Further, the multimodal fluorescence imaging method further includes:

[0012] S6. Optimize the decision support system using machine learning algorithms based on real-time data and surgical outcomes obtained during multiple surgeries on the same person, and generate customized surgical plans for different patients;

[0013] S7. Design an adaptive adjustment mechanism to dynamically adjust the parathyroid function risk assessment model based on real-time monitoring data and feedback information during the surgery.

[0014] Further, the image preprocessing includes denoising using wavelet transform to ensure the smoothness and details of the image, and then performing wavelength normalization on the denoised image to ensure data comparability between different bands;

[0015] Among them, the optimization process further includes:

[0016] Introduce an adaptive spectral selection mechanism based on a deep learning model. Through automatic learning of image features, the laser band is adjusted in real time to calculate the optimal wavelength to maximize the imaging effect of the target area. At the same time, a regularization term is introduced into the adaptive spectral selection mechanism. The regularization term ensures the stability and robustness of the band selection by reducing the risk of overfitting in band selection. Finally, based on the optimal wavelength, the output wavelength of the endoscopic laser source is adjusted through a mapping function, so that the image has the best contrast and resolution in this band;

[0017] Among them, the mapping function is a mapping function optimized according to the spectral characteristics of the target tissue.

[0018] Further, perform Fourier transform on the optimized image set to obtain a frequency domain representation, and then design a weighting strategy for the low-frequency and high-frequency parts of each band respectively for frequency domain image fusion, specifically expressed as:

[0019]

[0020] Among them, F fused represents the fused frequency domain image, F low (λ i ) and F high (λ i ) represent the frequency transforms of the low-frequency and high-frequency parts respectively, w low (λ i ) and w high (λ i ) are the weighting coefficients for the low-frequency and high-frequency parts, N is the total number of bands, and i is the i-th band;

[0021] Among them, based on the spatial frequency distribution of each band image in a specific area, weighting coefficients w low (λ i ) and w high (λ i ) for the low-frequency and high-frequency parts are designed and expressed as:

[0022]

[0023] Among them, and respectively represent the local contrast and information entropy of the image of band i, and are used to measure the feature intensity of the image area.

[0024] Furthermore, the depth convolutional neural network model is introduced according to the fused frequency-domain image to perform object detection and localization on the fused image, specifically:

[0025] The fused image is input into a pre-trained convolutional neural network, where the pre-trained convolutional neural network consists of multiple convolutional layers and pooling layers; the output of the pre-trained convolutional neural network is the bounding box coordinates of the target area, that is, the exact position and size of the region of interest;

[0026] At the same time, a spatial consistency regularization term is introduced during the training process of the convolutional neural network to encourage the network to maintain consistent localization results when processing multiple band images, thereby avoiding the problem of inaccurate localization caused by differences between different bands.

[0027] Furthermore, after image localization, through the adaptive local histogram equalization technique, the target area is refined and enhanced, specifically:

[0028] I enhanced (P target ) = I fused (P target ) · ALHE(P target ),

[0029] where I enhanced (P target ) represents the enhanced image, and ALHE(P target ) is the local histogram equalization operation calculated according to the target area P target .

[0030] Furthermore, the blood flow change measure is quantitatively measured by a blood flow change index, and the blood flow change index F bio is calculated as follows:

[0031]

[0032] Among them, Z pre and Zpost is the impedance value of the parathyroid region before and after surgery; w i is the weighting coefficient of each vascular region; v i is the velocity component of the blood flow velocity field in a specific region; dA is the infinitesimal area on the vessel surface; N is the number of vascular regions;

[0033] The impedance change rate ΔZ bio , is calculated as follows:

[0034]

[0035] where Z max and Z min are the maximum and minimum impedance values during the monitoring period, respectively;

[0036] The multi - spectral image and hydrodynamic model,

[0037]

[0038] where Q local is the local blood flow, v local (x) is the local blood flow velocity field; dA is the infinitesimal area on the vessel cross - section; V is the vascular volume within the parathyroid region;

[0039] At the same time, a blood flow weighting function w flow (λ) is introduced to adjust the blood flow signal in the image according to the vascular display characteristics of different bands:

[0040]

[0041] where Contrast λ is the local contrast of each band image; λ is the band number;

[0042] The parathyroid function risk assessment model is expressed as:

[0043] R risk =γ1·F bio +γ2·Q flow +γ3·ΔZ bio

[0044] where R risk is the risk score, and γ1, γ2, γ3 are weighting coefficients.

[0045] Furthermore, the automatic activation of protection measures according to the decision - making support model is as follows:

[0046] Dynamically adjust the surgical process through feedback control by combining blood flow and impedance data. The specific control formula is:

[0047]

[0048] Among them, u(t) is the control signal, representing the surgical strategy that needs to be adjusted; e(t) is the error function, representing the deviation between the monitored data and the set standard; K p , K d , K i are the proportional, derivative, and integral gain coefficients, reflecting the adjustment strength of the feedback system.

[0049] Furthermore, the customized surgical plan automatically recommends a surgical strategy according to the specific characteristics of the patient through a deep learning model.

[0050] In the second aspect of the present invention, a multimodal fluorescence imaging system for thyroid surgery is provided. The system includes:

[0051] A multimodal image acquisition unit, configured to simultaneously acquire image data in the infrared, near-infrared, and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multi-spectral image set, and perform optimization processing on the multi-spectral image set to output an optimized image set;

[0052] An image enhancement unit, configured to perform Fourier transform on the optimized image set to obtain a frequency-domain representation, then design a weighting strategy for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion, introduce a deep convolutional neural network model for target detection and localization on the fused frequency-domain image, and perform refined enhancement processing on the target area based on the local contrast adjustment enhancement technology after positioning to further improve the contrast and clarity of the target area and obtain an enhanced image;

[0053] A thyroid surgery real-time data monitoring unit, configured to monitor the blood flow state of the parathyroid gland in real time, construct a parathyroid function protection and blood flow change evaluation model to evaluate blood flow changes according to the blood flow change metric and evaluate the parathyroid function state according to the impedance change rate, construct a multi-spectral image and hydrodynamic model based on the blood flow changes and the parathyroid function state, calculate the local blood flow velocity field and local blood flow in real time, and establish a parathyroid function risk assessment model according to the blood flow change metric, the impedance change rate, and the local blood flow to evaluate the parathyroid function risk in real time;

[0054] A thyroid surgery risk assessment unit, which is used to design comprehensive risk assessment indicators for comprehensively assessing the risk of parathyroid injury according to a parathyroid function risk assessment model, combining the influence of parathyroid blood flow change measurement, impedance change rate and local blood flow on parathyroid injury, establish an automatic decision-making mechanism based on the comprehensive risk assessment indicators to judge whether the parathyroid needs to be protected, and if the comprehensive risk assessment indicator exceeds a preset threshold, feedback a protection suggestion to the surgical team;

[0055] An early warning unit, which is used to compare the real-time blood flow change and impedance change during the operation with an existing standard model, trigger an alarm through a set threshold, and when the monitored data exceeds the set threshold, trigger an alarm to remind the surgical team of the possible injury risk in the parathyroid area, and at the same time automatically activate protection measures according to the parathyroid function risk assessment model.

[0056] The beneficial technical effects of the present invention are at least as follows:

[0057] By integrating adaptive intelligent spectral imaging, multispectral collaborative navigation and real-time function evaluation, the present invention has made breakthrough progress in accurately locating tumors, metastatic lymph nodes and protecting parathyroid function. The core innovation of the system lies in its ability to intelligently select and adjust the best spectral excitation band according to the images and tissue characteristics collected in real time during the operation, so as to provide high-quality multispectral imaging for different tissue types, greatly improving the resolution and contrast between tumors and surrounding normal tissues. In addition, the system combines near-infrared imaging and photoacoustic imaging technologies, can monitor the blood flow conditions of the thyroid and parathyroid in real time, and evaluate their functional status through dynamic imaging, providing real-time decision-making support for doctors and reducing parathyroid injury and postoperative complications.

[0058] The adaptive spectral imaging technology of the present invention dynamically selects the excitation light source by analyzing the spectral responses of different tissues in real time, significantly improving the distinguishability between tumors and normal tissues, and is particularly suitable for the accurate identification of micro-lesions. By comparing with existing fluorescence imaging technologies, it can obtain clearer imaging results through endogenous fluorescence and autologous spectral characteristics without relying on exogenous contrast agents, thus overcoming the problems of the safety of traditional contrast agents, standardized operation and contamination.

[0059] In addition, the multispectral collaborative navigation function in the system accurately locates the positions of the thyroid and parathyroid by integrating near-infrared and photoacoustic imaging technologies, and monitors the blood flow status of the parathyroid in real time, so as to ensure the effective protection of the parathyroid function during tumor resection and reduce the risk of hypoparathyroidism. Through this multispectral navigation system, the tumor localization during the operation and the protection of surrounding important tissues are strongly guaranteed, avoiding the problems of missed resection and accidental injury. Description of the Drawings

[0060] The present invention will be further described with reference to the accompanying drawings. However, the embodiments shown in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the following drawings without creative work.

[0061] Figure 1 It is a flowchart of a multimodal fluorescence imaging method applied to thyroid surgery according to the present invention. Detailed implementation manners

[0062] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for explaining the present invention and should not be construed as a limitation to the present invention.

[0063] As Figure 1 shown, a multimodal fluorescence imaging method applied to thyroid surgery provided by an embodiment of the present invention includes the following:

[0064] S1. Use a multi-wavelength endoscope system to simultaneously collect image data in the infrared, near-infrared, and visible light bands, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multispectral image set, and perform optimization processing on the multispectral image set to output an optimized image set.

[0065] Specifically, the input of this step is the real-time image data captured by the endoscope system during the operation. We first use a multi-wavelength endoscope system to simultaneously collect image data in the infrared, near-infrared, and visible light bands, which are respectively denoted as I(λ i ), where λ i is the wavelength of different bands, and i represents the index of the wavelength (for example, λ1 represents the infrared band, λ2 represents the near-infrared band, and λ3 represents the visible light band). The images of each band are preliminarily synthesized through an image fusion algorithm to generate a multispectral image set and perform the following image preprocessing:

[0066] Denoising processing: Use wavelet transform to remove the high-frequency noise in the image and ensure the smoothness and details of the image. At this time, the image is processed by wavelet transform through the following formula:

[0067] I′(λ i ) = Wavelet(I(λ i ))

[0068] where I′(λ i ) represents the denoised image, and Wavelet(·) is the wavelet transform function.

[0069] Normalization processing: Perform wavelength normalization on the denoised image to ensure data comparability between different bands. The normalization processing is carried out through the following formula:

[0070]

[0071] where μ λi is the mean value of band λ i , is the standard deviation of band λ i , and I′(λ i ) is the normalized image.

[0072] The purpose of this preprocessing step is to improve data quality and lay a foundation for subsequent image optimization and tissue recognition.

[0073] Furthermore, during thyroid cancer surgery, different tissues (such as tumors, parathyroid glands, lymph nodes, etc.) have significant differences in the reflection or emission spectral characteristics of different spectral bands. Existing imaging methods usually rely on fixed spectral bands, which may lead to poor recognition of some key tissues (such as small tumors, lymph nodes), or even omission.

[0074] To solve this problem, we introduce an adaptive spectral selection mechanism. This mechanism is based on a deep learning model and can automatically adjust the laser band in real time by automatically learning the image features to maximize the imaging effect of the target area. We define a deep neural network model M(·), which outputs an optimal wavelength λ opt for each image band according to the spectral characteristics:

[0075] λ opt = M(I′(λ i ))

[0076] where M(·) is a trained deep learning model that can extract the features of the target tissues (such as tumors, parathyroid glands, etc.) from the input image data and calculate the most suitable laser band λ opt .

[0077] To further optimize the effect of the laser, we introduce a regularization term in the band selection process, taking into account the similarity of different tissues and the influence of data noise. This regularization term ensures the stability and robustness of the band selection by reducing the risk of overfitting in the band selection. The formula is expressed as:

[0078]

[0079] where I target is the ideal image of the target tissue, is a regularization term, and its specific design is as follows:

[0080]

[0081] Among them, represents the wavelength band λ i the gradient of the image, and α is the regularization strength parameter. The introduction of this regularization term can effectively reduce noise, enhance the contrast between the real target and the background, and thus improve the recognition accuracy of the target tissue.

[0082] Furthermore, based on the optimal wavelength band λ opt calculated according to the above steps, we further adjust the output wavelength of the endoscopic laser source so that the image has the best contrast and resolution in this wavelength band. This optimization step is achieved through the following mapping function:

[0083] I opt (λ opt ) = f(I′(λ opt ))

[0084] Among them, I opt (λ opt ) is the optimized image, and f(·) is a mapping function optimized according to the spectral characteristics of the target tissue (such as tumors, thyroid glands, etc.).

[0085] Through this optimization process, we can obtain high-quality images optimized for different tissue types in real time during the operation, especially in the identification of small tumors, metastatic lymph nodes, and parathyroid glands, improving the precision and safety of the operation.

[0086] S2. Perform Fourier transform on the optimized image set to obtain the frequency domain representation, then design weighted strategies for the low-frequency part and the high-frequency part of each wavelength band respectively for frequency domain image fusion. Based on the fused frequency domain image, introduce a deep convolutional neural network model to perform target detection and localization on the fused image. After completing the localization, perform fine enhancement processing on the target area based on the local contrast adjustment enhancement technology to further improve the contrast and clarity of the target area, and obtain the enhanced image.

[0087] Specifically, multispectral image fusion is the core link of this step. The purpose is to improve the overall visual effect and information content of the image by fusing information from different spectral bands, especially enhancing the features of the target area. To achieve this goal, we adopt a method called "Weighted Frequency Domain Optimized Fusion" (WFD-OF).

[0088] Traditional image fusion methods usually rely on only one optimization method in the spatial domain or frequency domain, but these methods often cannot well balance the fusion accuracy of information in different bands. In the proposed scheme, the fusion process takes into account the local correlation of spectral information and the information complementarity between different bands. Therefore, we weight the low-frequency and high-frequency parts of each band image separately in the frequency domain.

[0089] Specifically, first, perform Fourier transform on each band image I opt (λ i ) to obtain the frequency domain representation F opt (λ i ). Then, design weighted strategies for the low-frequency and high-frequency parts of each band respectively, and comprehensively consider the spatial information and spectral information of each band image to obtain the weighted frequency domain expression. The mathematical formula for fusion is expressed as:

[0090]

[0091] where F fused represents the fused frequency domain image, F low (λ i ) and F high (λ i ) represent the frequency transforms of the low-frequency and high-frequency parts respectively, and w low (λ i ) and w high (λ i ) are the weighting coefficients for the low-frequency and high-frequency parts.

[0092] The design of the weighting coefficients w low (λ i ) and w high (λ i ) takes into account the spatial frequency distribution of each band image in a specific region. We introduce the local contrast and the local information entropy Entropy λi of the band image to calculate the weighting coefficients, which can effectively emphasize the high-frequency information in the information-dense regions of the image while maintaining the smooth effect in the low-frequency regions:

[0093]

[0094] where and represent the local contrast and information entropy of the image of band i respectively, and are used to measure the feature intensity of the image region.

[0095] Through these weighting schemes, we not only ensure the accuracy of image fusion, but also effectively avoid information loss and enhance the feature expression of target regions (such as tumors, thyroids, etc.).

[0096] Further, after completing the image fusion, the next step is to automatically locate the target region in the fused image. For this purpose, we introduce a deep convolutional neural network (CNN) model L loc (·), and use this model to detect and locate the target in the fused image. This model is specially designed and trained to automatically identify and mark the key regions in the image, especially in target regions with complex structures and morphologies, such as tumors or thyroids.

[0097] First, we input the fused image I fused into a pre-trained convolutional neural network, which consists of multiple convolutional layers and pooling layers. The output of the network is the bounding box coordinates P target of the target region, that is, the exact position and size of the region of interest (ROI).

[0098] The specific localization process can be expressed as:

[0099] P target = L loc (I fused )

[0100] where P target = (x min , y min , x max , y max ) are the four coordinates of the output target region bounding box, and L loc (·) is the deep learning model for locating the target region.

[0101] To improve the localization accuracy and stability, we introduce a spatial consistency regularization term during the training process. This regularization term encourages the network to maintain consistent localization results when processing multi-band images, thereby avoiding inaccurate localization caused by differences between different bands. This regularization term can be expressed as:

[0102]

[0103] where λ is the regularization strength, is the gradient of the target region bounding box coordinates. The role of this regularization term is to improve the localization accuracy by constraining the smoothness of the target region boundary.

[0104] Furthermore, after image localization, we will perform a refined enhancement process on the target area to further improve the contrast and clarity of the target area. This step adopts an enhancement technique based on local contrast adjustment, making the important target areas in the image more prominent while reducing the interference of background noise. Specifically, we use the "Adaptive Local Histogram Equalization" (ALHE) technique, which can automatically adjust the contrast according to the spatial characteristics of the target area.

[0105] This enhancement process is achieved through the histogram equalization algorithm in the local area and can be expressed as:

[0106] I enhanced (P target ) = I fused (P target ) · ALHE(P target )

[0107] Where, I enhanced (P target ) represents the enhanced image, and ALHE(P target ) is the local histogram equalization operation calculated according to the target area P target . This step can greatly improve the visualization effect of the target area and provide clearer information for the doctor's decision-making during the operation.

[0108] S3. Real-time monitor the blood flow status of the parathyroid gland, construct an evaluation model for parathyroid function protection and blood flow change, evaluate the blood flow change according to the blood flow change metric and evaluate the parathyroid function status according to the rate of impedance change, construct a model based on multi-spectral images and hydrodynamic models according to the blood flow change and parathyroid function status, calculate the local blood flow velocity field and local blood flow in real time, and establish a parathyroid function risk assessment model based on the blood flow change metric, rate of impedance change and local blood flow, and evaluate the parathyroid function risk in real time.

[0109] Specifically, in the previous steps, through image fusion and precise localization, we have obtained the multi-spectral image and spatial information of the parathyroid gland and accurately located the target area. Next, our task is to real-time monitor the blood flow status of the parathyroid gland and evaluate its functional changes in order to provide timely decision support during the operation and prevent the loss of parathyroid function or insufficient blood perfusion.

[0110] Furthermore, the evaluation model for parathyroid function protection and blood flow change:

[0111] Input: The multi-spectral image data and impedance monitoring data output from the previous step.

[0112] Output: Blood flow change metrics and impedance change indicators in the parathyroid region.

[0113] Specifically, the blood flow change is evaluated by combining the impedance value with the blood flow velocity field. The blood flow change indicator F is defined through the vascular features extracted from the multispectral image and the real-time blood flow velocity field. bio :

[0114]

[0115] Among them, Z pre and Z post are the impedance values of the parathyroid region before and after surgery. w i is the weighting coefficient for each vascular region. v i is the velocity component of the blood flow velocity field in a specific region. dA is the differential area on the vascular surface. N is the number of vascular regions. This formula combines the vascular features in the image with the impedance data and can accurately evaluate the blood flow change in the parathyroid gland.

[0116] Furthermore, the impedance change rate ΔZ bio is used to evaluate the functional status of the parathyroid gland:

[0117]

[0118] Among them, Z max and Z min are the maximum and minimum impedance values during the monitoring period, respectively. The change in impedance is directly related to the blood flow and tissue status of the parathyroid gland and can reflect its health status in real time.

[0119] Furthermore, real-time blood flow monitoring and feedback mechanism:

[0120] Input: Blood flow change data and impedance change data from the previous step.

[0121] Output: Real-time blood flow monitoring results and feedback.

[0122] Specifically, based on the multispectral image and the hydrodynamic model, the local blood flow velocity field v local and the flow rate Q local are calculated in real time and defined as follows:

[0123]

[0124] Among them, v local (x) is the local blood flow velocity field; dA is the differential area on the vascular cross-section; V is the vascular volume in the parathyroid region. Using the hydrodynamic equation, we can accurately describe the dynamic changes of blood flow, especially the blood flow status in the parathyroid region.

[0125] Further, introduce a blood flow weighting function w flow (λ). According to the blood vessel display characteristics of different bands, adjust the blood flow signal in the image:

[0126]

[0127] where Contrast λ is the local contrast of each band image; λ is the band number. This weighting function helps us accurately detect the blood flow signal and enhance the visualization effect of blood flow information.

[0128] Further, based on the blood flow change data and impedance change data from the previous two steps, output the parathyroid function protection decision and real-time feedback.

[0129] Specifically, based on the blood flow change metric, impedance change metric, and real-time blood flow data, establish a parathyroid function risk assessment model R risk :

[0130] R risk = γ1·F bio + γ2·Q flow + γ3·ΔZ bio

[0131] where γ1, γ2, γ3 are weighting coefficients; F bio is the comprehensive index of impedance and blood flow velocity field; Q flow is the blood flow volume; ΔZ bio is the impedance change rate. Through the analysis of real-time data, R risk can real-time evaluate the parathyroid function risk and provide decision support for doctors on protection measures.

[0132] Further, if R risk exceeds the preset threshold, the system will automatically give real-time feedback to prompt the doctor whether protection measures are needed. The system provides efficient real-time monitoring and decision support for the operation by combining blood flow data, impedance change, and image information.

[0133] S4. According to the parathyroid function risk assessment model, combined with the effects of parathyroid blood flow change metric, impedance change rate, and local blood flow volume on parathyroid injury, design a comprehensive risk assessment index to comprehensively evaluate the parathyroid injury risk. Based on the comprehensive risk assessment index, establish an automatic decision-making mechanism to judge whether the parathyroid needs protection. If the comprehensive risk assessment index exceeds the preset threshold, then feedback the protection suggestion to the operation team.

[0134] Specifically, based on the blood flow and impedance change data obtained in the previous stage and in combination with the protection requirements of the parathyroid gland, a comprehensive risk assessment model was designed. This model considered the effects of blood flow changes, impedance changes, and local blood flow in the parathyroid region on parathyroid gland injury.

[0135] We define a comprehensive risk assessment index R total , which is in the form of a weighted sum:

[0136] R total = α1·F bio + α2·ΔZ bio + α3·Q local + α4·w flow

[0137] where α1, α2, α3, α4 are weighting coefficients, representing the importance of different input data to the total risk assessment; F bio is the measure of blood flow change in the parathyroid region; ΔZ bio is the measure of impedance change in the parathyroid region; Q local is the local blood flow in the parathyroid region; w flow is the blood flow weighting coefficient, reflecting the overall importance of blood flow. This risk assessment model comprehensively evaluates the risk of parathyroid gland injury by weighting the importance of different data sources. This comprehensive assessment index provides a quantitative basis for the subsequent decision support system.

[0138] Furthermore, based on the comprehensive risk assessment index R total , we established an automatic decision-making mechanism to determine whether the parathyroid gland needs protection. If R total exceeds the preset threshold R threshold , the system will feedback a protection suggestion to the surgical team:

[0139]

[0140] where R total is the comprehensively risk assessment value calculated in real time; R threshold is the threshold determined through historical surgical data and clinical experience, representing the risk threshold of parathyroid gland injury. This decision support system can determine whether parathyroid gland protection measures need to be taken through real-time calculation and can provide scientific decision support for the surgical team based on real-time data.

[0141] S5. Compare the real-time blood flow change and impedance change during the operation with the existing standard model, trigger an alarm through the set threshold, and when the monitored data exceeds the set threshold, trigger an alarm to remind the surgical team of the possible injury risk in the parathyroid region, and at the same time automatically activate the protection measures according to the parathyroid function risk assessment model.

[0142] Specifically, based on the decision support result Decision in the previous stage and the real-time data during the operation (such as impedance and blood flow change data), calculate the real-time feedback to ensure that the protective measures for the parathyroid gland during the operation are implemented.

[0143] Furthermore, compare the real-time blood flow change and impedance change with the existing standard model, and trigger an alarm through the set threshold:

[0144]

[0145] Among them, F ref is the standard blood flow change value and serves as a reference value; ΔZ ref is the standard impedance change value and serves as a reference value; ∈ bio and ∈ Z are the set tolerance error thresholds, representing the normal fluctuation ranges of blood flow and impedance.

[0146] When the monitored data exceeds the set threshold, the system triggers an alarm to remind the surgical team of the possible damage risk in the parathyroid gland area. This mechanism provides precise feedback for the real-time monitoring during the operation.

[0147] Furthermore, if the alarm is triggered, the system will automatically activate the protective measures according to the decision support model. For example, if the impedance value is too high or the blood flow change is abnormal, the system can reduce the damage risk to the parathyroid gland through real-time control of surgical instruments or drug regulation.

[0148] In order to quickly respond and automatically execute the protective measures, the system dynamically adjusts the operation process through feedback control by combining blood flow and impedance data. The specific control formula is:

[0149]

[0150] Among them, u(t) is the control signal, representing the surgical strategy that needs to be adjusted (such as adjusting the current, drug delivery volume, etc.); e(t) is the error function, representing the deviation between the monitored data and the set standard; K p , K d , K i are the proportional, derivative, and integral gain coefficients, reflecting the adjustment strength of the feedback system.

[0151] S6. According to the real-time data and surgical results obtained during multiple surgeries of the same person, use machine learning algorithms to optimize the decision support system and generate customized surgical plans for different patients.

[0152] Specifically, according to the real-time data obtained during the operation and the operation results, machine learning algorithms (such as deep neural networks or reinforcement learning) are used to optimize the decision support system. By learning from historical data, the weighting coefficients in the system are adjusted to better predict the risk of parathyroid injury.

[0153] We update the weighting coefficients through the following optimization formula:

[0154]

[0155] where α i is the weighting coefficient used to adjust the weights of different data sources; η is the learning rate; is the loss function, which is calculated based on the error between the predicted result and the actual result; is the gradient of the loss function with respect to the weighting coefficient.

[0156] Through continuous optimization, the weighting coefficients will be adjusted according to the surgical feedback of different patients, thereby improving the prediction accuracy of the system.

[0157] Furthermore, the situations of each patient are different. Therefore, the decision support system needs to generate customized surgical plans based on the personalized data of the patient (such as parathyroid position, blood flow pattern, etc.). We use a deep learning model to automatically recommend surgical strategies according to the specific characteristics of the patient. The personalized recommendation can be modeled by the following formula:

[0158]

[0159] where is the surgical plan for personalized recommendation; x patient is the personalized characteristics of the patient (such as parathyroid position, blood flow pattern, impedance value, etc.); θ is the parameter of the model, which is trained based on historical surgical data. The personalized recommendation can not only improve the efficiency of parathyroid protection, but also optimize the surgical treatment plan for each patient.

[0160] S7. Design an adaptive adjustment mechanism to dynamically adjust the parathyroid function risk assessment model according to the real-time monitoring data and the feedback information during the operation.

[0161] To cope with emergencies in different surgical scenarios, the system adopts an adaptive adjustment mechanism to dynamically adjust the decision support model according to the real-time monitoring data and the feedback information during the operation. This mechanism ensures that in a changing surgical environment, the system can respond quickly and optimize the operation.

[0162] The adaptive adjustment is implemented using the following feedback control formula:

[0163]

[0164] Among them, u(t) is the adjusted control signal, indicating the surgical protection strategy to be adopted; e(t) is the current error, that is, the gap between the real-time data and the expected target; K p , K d , K i are the proportional, derivative, and integral gains, which determine the response speed and sensitivity of the controller.

[0165] In another embodiment of the present invention, a multimodal fluorescence imaging system applied to thyroid surgery is provided. The system includes:

[0166] A multimodal image acquisition unit, which is used to simultaneously acquire image data in the infrared, near-infrared, and visible light bands using a multi-wavelength endoscope system, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multispectral image set, and perform optimization processing on the multispectral image set to output an optimized image set;

[0167] An image enhancement unit, which is used to perform Fourier transform on the optimized image set to obtain a frequency-domain representation, then design a weighting strategy for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion, introduce a deep convolutional neural network model for target detection and localization of the fused image based on the fused frequency-domain image, and after completing the localization, perform refined enhancement processing on the target area based on the enhancement technology of local contrast adjustment to further improve the contrast and clarity of the target area to obtain an enhanced image;

[0168] A thyroid surgery real-time data monitoring unit, which is used to monitor the blood flow status of the parathyroid gland in real time, construct a parathyroid function protection and blood flow change evaluation model to evaluate blood flow changes according to the blood flow change metric and evaluate the parathyroid function status according to the impedance change rate, construct a multi-spectral image and hydrodynamic model based on the blood flow changes and parathyroid function status, calculate the local blood flow velocity field and local blood flow in real time, and establish a parathyroid function risk assessment model according to the blood flow change metric, impedance change rate, and local blood flow to evaluate the parathyroid function risk in real time;

[0169] A thyroid surgery risk assessment unit, which is used to design a comprehensive risk assessment index based on the parathyroid function risk assessment model, combined with the impact of parathyroid blood flow change metric, impedance change rate, and local blood flow on parathyroid injury, to comprehensively evaluate the parathyroid injury risk, establish an automatic decision-making mechanism based on the comprehensive risk assessment index to judge whether the parathyroid gland needs protection, and if the comprehensive risk assessment index exceeds the preset threshold, feedback a protection suggestion to the surgical team;

[0170] The early warning unit is used to compare the real-time blood flow changes and impedance changes during the operation with the existing standard model, trigger an alarm through a set threshold. When the monitored data exceeds the set threshold, the alarm is triggered to remind the surgical team of the possible injury risk in the parathyroid area. At the same time, the protection measures are automatically activated according to the parathyroid function risk assessment model.

[0171] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0172] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0173] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0174] Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A multimodal fluorescence imaging method applied to thyroid surgery, characterized in that, The method includes: S1. Use a multi-wavelength endoscope system to simultaneously collect image data in the infrared, near-infrared, and visible light bands, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multi-spectral image set, and perform optimization processing on the multi-spectral image set to output an optimized image set; S2. Perform Fourier transform on the optimized image set to obtain a frequency-domain representation, then design a weighting strategy for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion. Based on the fused frequency-domain image, introduce a deep convolutional neural network model to perform target detection and localization on the fused image. After completing the localization, perform a refined enhancement process on the target area based on an enhancement technique of local contrast adjustment to improve the contrast and clarity of the target area, and obtain an enhanced image; S3. Real-time monitor the blood flow state of the parathyroid gland, construct a parathyroid function protection and blood flow change evaluation model to evaluate blood flow changes according to the blood flow change metric and evaluate the parathyroid function state according to the impedance change rate. Based on the blood flow changes and the parathyroid function state, construct a model based on multi-spectral images and hydrodynamic models to calculate the local blood flow velocity field and local blood flow in real time, and establish a parathyroid function risk assessment model according to the blood flow change metric, the impedance change rate, and the local blood flow to real-time evaluate the parathyroid function risk; S4. According to the parathyroid function risk assessment model, combine the effects of the parathyroid blood flow change metric, the impedance change rate, and the local blood flow on parathyroid injury, design a comprehensive risk assessment index to comprehensively evaluate the parathyroid injury risk, establish an automatic decision-making mechanism based on the comprehensive risk assessment index to determine whether the parathyroid gland needs protection. If the comprehensive risk assessment index exceeds a preset threshold, feedback a protection recommendation to the surgical team; S5. Compare the real-time blood flow changes and impedance changes during the operation with the existing standard model, trigger an alarm through a set threshold. When the monitored data exceeds the set threshold, trigger an alarm to remind the surgical team of the possible injury risk in the parathyroid area, and at the same time automatically activate protection measures according to the parathyroid function risk assessment model.

2. The multimodal fluorescence imaging method applied to thyroid surgery according to claim 1, wherein The multi-modal fluorescence imaging method further includes: S6. According to the real-time data and surgical results obtained during multiple surgeries of the same person, use machine learning algorithms to optimize the decision support system and generate customized surgical plans for different patients; S7. Design an adaptive adjustment mechanism to dynamically adjust the parathyroid function risk assessment model according to the real-time monitoring data and feedback information during the operation.

3. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 1, wherein, The image preprocessing includes performing denoising processing using wavelet transform to ensure the smoothness and details of the image, and then performing wavelength normalization on the denoised image to ensure the data comparability between different bands; Among them, the optimization processing further includes: An adaptive spectral selection mechanism based on a deep learning model is introduced. Through the automatic learning of image features, the laser band is adjusted in real time to calculate the optimal wavelength, so as to maximize the imaging effect of the target area. At the same time, a regularization term is introduced into the adaptive spectral selection mechanism. The regularization term ensures the stability and robustness of band selection by reducing the overfitting risk of band selection. Finally, based on the optimal wavelength, the output wavelength of the endoscopic laser source is adjusted through a mapping function, so that the image has the best contrast and resolution in this band; Among them, the mapping function is a mapping function optimized according to the spectral characteristics of the target tissue.

4. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 1, characterized in that, The optimized image set is subjected to Fourier transform to obtain a frequency-domain representation, and then weighted strategies are designed for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion, which is specifically expressed as: Among them, F fused represents the fused frequency-domain image, F low (λ i ) and F high (λ i ) respectively represent the frequency transforms of the low-frequency and high-frequency parts, w low (λ i ) and w high (λ i ) are the weighting coefficients for the low-frequency and high-frequency parts, N is the total number of bands, and i is the i-th band; Among them, based on the spatial frequency distribution of each band image in a specific region, weighting coefficients w low (λ i ) and w high (λ i ) for the low-frequency and high-frequency parts are designed and expressed as: Among them, and respectively represent the local contrast and information entropy of the band-i image, and are used to measure the feature intensity of the image region.

5. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 1, wherein, Based on the fused frequency-domain image, a deep convolutional neural network model is introduced to perform target detection and localization on the fused image, specifically including: The fused image is input into a pre-trained convolutional neural network, where the pre-trained convolutional neural network consists of multiple convolutional layers and pooling layers; the output of the pre-trained convolutional neural network is the bounding box coordinates of the target area, that is, the exact position and size of the region of interest; At the same time, a spatial consistency regularization term is introduced during the training process of the convolutional neural network to encourage the network to maintain consistent localization results when processing multi-band images, thus avoiding the problem of inaccurate localization caused by differences between different bands.

6. The multimodal fluorescence imaging method applied to thyroid surgery according to claim 5, wherein After image localization, through the adaptive local histogram equalization technique, fine enhancement processing is performed on the target area, specifically as follows: I enhanced (P target ) = I fused (P target )·ALHE(P target ), Among them, I enhanced (P target ) represents the enhanced image, and ALHE(P target ) is the local histogram equalization operation calculated according to the target region P target .

7. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 1, wherein The blood flow change measure is quantitatively measured by a blood flow change index, and the blood flow change index F bio is calculated as follows: Among them, Z pre and Z post are the impedance values of the parathyroid region before and after surgery; w i is the weighting coefficient of each vascular region; v i is the velocity component of the blood flow velocity field in a specific region; dA is the infinitesimal area of the vessel surface; N is the number of vascular regions; The rate of change in impedance ΔZ bio , is calculated as follows: Among them, Z max and Z min are the maximum and minimum impedance values during the monitoring period, respectively; Based on the multi-spectral image and the hydrodynamic model, where Q local is the local blood flow, v local (x) is the local blood flow velocity field; dA is the differential area on the blood vessel cross-section; V is the blood vessel volume within the parathyroid region; At the same time, introduce the blood flow weighting function w flow (λ), and adjust the blood flow signal in the image according to the blood vessel display characteristics of different bands: Among them, Contrast λ is the local contrast of each band image; λ is the band number; The parathyroid function risk assessment model is expressed as: R risk = γ1·F bio + γ2·Q flow + γ3·ΔZ bio Among them, R risk is the risk score, and γ1, γ2, γ3 are the weighting coefficients.

8. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 7, characterized in that, According to the decision support model, protection measures are automatically activated, specifically including: Combining blood flow and impedance data to dynamically adjust the surgical process through feedback control, and the specific control formula is: Among them, u(t) is the control signal, representing the surgical strategy that needs to be adjusted; e(t) is the error function, representing the deviation between the monitored data and the set standard; K p , K d , K i are the proportional, derivative, and integral gain coefficients, reflecting the adjustment strength of the feedback system.

9. A multimodal fluorescence imaging method applied to thyroid surgery according to claim 2, characterized in that, The customized surgical plan automatically recommends surgical strategies according to the specific characteristics of the patient through a deep learning model.

10. A multimodal fluorescence imaging system applied to thyroid surgery, characterized in that, The system includes: A multi-modal image acquisition unit, which is used to simultaneously acquire image data in the infrared, near-infrared, and visible light bands using a multi-wavelength endoscopic system, perform image preprocessing on the image data, preliminarily synthesize the preprocessed images to generate a multi-spectral image set, and perform optimization processing on the multi-spectral image set to output an optimized image set; An image enhancement unit, which is used to perform Fourier transform on the optimized image set to obtain a frequency-domain representation, then design weighted strategies for the low-frequency part and the high-frequency part of each band respectively for frequency-domain image fusion, introduce a deep convolutional neural network model based on the fused frequency-domain image to perform target detection and localization on the fused image, and after completing the localization, perform fine enhancement processing on the target area based on the enhancement technique of local contrast adjustment to further improve the contrast and clarity of the target area, and obtain an enhanced image; A real-time data monitoring unit for thyroid surgery, which is used to monitor the blood flow status of the parathyroid gland in real time, construct an evaluation model for parathyroid function protection and blood flow change assessment to evaluate blood flow changes according to blood flow change metrics and evaluate the parathyroid function status according to the impedance change rate, construct a multi-spectral image and hydrodynamic model based on blood flow changes and parathyroid function status, calculate the local blood flow velocity field and local blood flow volume in real time, and establish a parathyroid function risk assessment model according to blood flow change metrics, impedance change rate and local blood flow volume to evaluate the parathyroid function risk in real time; A thyroid surgery risk assessment unit, which is used to design a comprehensive risk assessment index for comprehensively evaluating the risk of parathyroid injury according to the parathyroid function risk assessment model, combined with the influence of parathyroid blood flow change metrics, impedance change rate and local blood flow volume on parathyroid injury, establish an automatic decision-making mechanism based on the comprehensive risk assessment index to judge whether the parathyroid gland needs protection, and if the comprehensive risk assessment index exceeds the preset threshold, feedback a protection suggestion to the surgical team; An early warning unit, which is used to compare the real-time blood flow changes and impedance changes during the operation with the existing standard model, trigger an alarm through the set threshold, trigger an alarm when the monitoring data exceeds the set threshold, remind the surgical team of the possible injury risk in the parathyroid area, and automatically activate the protection measures according to the parathyroid function risk assessment model.

Images