A multi-modal facial blood perfusion image acquisition and analysis system for combined infrared thermal imaging and fundus microcirculation images

Abstract

This invention discloses a multimodal facial blood perfusion image acquisition device and intelligent analysis system for combined infrared thermal imaging and fundus microcirculation imaging, comprising the following steps: S1, filling out a data acquisition form and collecting relevant data; S2, conducting a personalized breath-holding test; S3, acquiring infrared images of the user's face and acquiring color fundus images at different times after pupil dilation of the patient's left and right eyes; S4, converting the acquired color fundus images into fluorescein angiography (FFA) images using an improved Real-ESRGAN model; S5, extracting features from the facial infrared images and the generated FFA images respectively; S6, fusing the obtained infrared features and FFA features; S7, outputting the feature results. This invention acquires hemodynamic information from FFA non-invasively and combines it with facial infrared features for fusion analysis, enabling efficient and accurate acquisition and analysis of facial blood perfusion features, providing data support for related physiological research.

Description

Technical Field

[0001] This application relates to the field of biomedical image processing, and in particular to a multimodal facial blood flow perfusion image acquisition and analysis system for combined infrared thermal imaging and fundus microcirculation images. Background Technology

[0002] The carotid artery is primarily responsible for supplying blood to the head and face. Its blood supply status directly affects the blood perfusion of the head and face. Accurately collecting physiological data related to blood perfusion in the face and head has significant technical and application value.

[0003] Currently, examination of the carotid artery supplying the craniofacial region typically relies on ultrasound equipment. Doppler ultrasound allows for real-time monitoring of blood flow velocity and direction, assessing the hemodynamic characteristics of the carotid artery. Ultrasound technology offers advantages such as being non-invasive, real-time, and portable, but it requires highly skilled operators and is limited by factors such as patient obesity and gas interference. Magnetic resonance imaging (MR angiography, MRA) and phase-contrast magnetic resonance imaging (PC-MRI) are another non-invasive method that provides detailed anatomical and hemodynamic information of the carotid artery. This technique performs well in assessing blood flow velocity, blood volume, and the degree of arterial stenosis, but the equipment is expensive and the examination time is relatively long. CT angiography (CTA) combines high-resolution CT images with hemodynamic analysis, effectively assessing the anatomical morphology and blood flow status of the carotid artery. However, CT examinations involve radiation and are sensitive to contrast agents, potentially making them unsuitable for all patients.

[0004] The forehead is supplied by branches of the internal carotid artery, while the cheek is supplied by the external carotid artery. Both vessels are superficial and closely related to intracranial blood supply; therefore, their perfusion data are of significant reference value for assessing the overall microcirculation status. Infrared imaging technology can non-invasively capture local blood flow characteristics, indirectly reflecting the distribution and regulation of subcutaneous blood flow by observing dynamic changes in blood flow in this area. Fundus microcirculation, as a "window" into the overall microcirculation, directly reflects the functional status of the entire vascular system. Combining these two approaches provides more comprehensive support for the acquisition of microcirculatory physiological data.

[0005] In recent years, non-invasive functional imaging techniques have provided a new perspective for blood flow assessment. Infrared thermography, by detecting subtle changes in body surface temperature, can indirectly reflect the distribution and regulation of subcutaneous blood flow, offering safety and intuitive visualization. On the other hand, the retinal vessels in the fundus serve as a directly observable window into microcirculation, and their condition is closely related to the overall health of the body's blood vessels. Fluorescein fundus angiography (FFA) is the standard method for assessing retinal hemodynamics, providing important functional parameters such as perfusion time and vascular leakage. However, this examination is invasive, requiring intravenous injection of contrast agents, making the procedure complex and carrying certain risks, thus limiting its use for routine screening. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal facial blood flow perfusion image acquisition device and intelligent analysis system for combined infrared thermal imaging and fundus microcirculation imaging, in order to solve the defects or problems mentioned in the background art.

[0007] To achieve the above-mentioned objectives, the present invention aims to provide a multimodal facial blood flow perfusion image acquisition and analysis method that combines infrared thermal imaging and fundus microcirculation imaging, the steps of which are as follows:

[0008] S1: Information Acquisition: Acquire basic information, facial infrared images, and high-definition color fundus images of the left and right eyes;

[0009] S2: An improved Real-ESRGAN model is used to convert the acquired high-resolution color fundus images into fluorescein angiography (FFA) images.

[0010] S3: Feature Extraction: The generated FFA image and the acquired facial infrared image are preprocessed and time-aligned, key structures and phase points are calibrated, and the region of interest is defined. Then, the temporal dynamics, morphological perfusion features and permeability features of the FFA image are extracted, and the thermodynamics, morphological temperature distribution features and thermal conduction features of the facial infrared image are extracted.

[0011] S4: Feature Fusion: Features extracted from facial infrared images are used as query vectors, and the microcirculation-related features of the fundus microcirculation of the left and right eyes are used as two independent key vectors and value vectors, respectively. They are fused through a cross-attention mechanism, and after being spliced ​​to form a comprehensive feature map, the core features are further extracted through a fully connected layer.

[0012] S5: Output Results: The Softmax function outputs blood perfusion characteristic parameters and analyzes the differences in fundus microcirculation characteristics between the left and right eyes.

[0013] Preferably, the improved Real-ESRGAN structure in S2 is as follows: The Real-ESRGAN model is based on the RRDB module, and the size is enlarged after upsampling. When it is used for image generation, there is no need to enlarge the resolution. The improved Real-ESRGAN model removes the nearest neighbor sampling layer when scale=1; reduces the layer depth to four layers; and greatly simplifies the redundant network structure, thereby significantly improving the inference efficiency of the original model while maintaining feature fidelity.

[0014] Preferably, the improved Real-ESRGAN structure in S2 is as follows:

[0015] S2-1-1 Shallow Feature Extraction: First, a convolutional layer is used to extract shallow features from the low-resolution input image; this convolutional layer has 64 output channels, a kernel size of 3×3, a stride of 1, padding of 1, and is followed by a LeakyReLU activation function.

[0016] S2-1-2, Deep Feature Extraction: The shallow features are then processed through four consecutive RRDB blocks; each RRDB block contains multiple dense connections and residual connections.

[0017] S2-1-3, Reconstruction Module: After deep features pass through the RRDB block, a convolutional layer is used for feature fusion; after removing the upsampling module, another convolutional layer is used to generate a high-quality image;

[0018] S2-1-4, Discriminator Architecture: The discriminator architecture adopts the U-Net discriminator, which has an encoder-decoder structure;

[0019] S2-1-5 Loss Function: To train the image generation model, a composite loss function is used, which consists of three parts: pixel-level L1 loss, adversarial loss, and improved perceptual loss.

[0020] By introducing an innovative design of the loss function, combining clDice topological loss with VGG-19 improved perceptual loss, we can specifically solve the problem of distortion in the topological connectivity of fundus blood vessels, significantly improve the accuracy of blood vessel branch structure and spatial morphology in the generated image, and accurately adapt to the needs of physiological data feature extraction.

[0021] The total loss function is defined as follows:

[0022]

[0023] Where λ is a hyperparameter that balances the weights of each loss term.

[0024] Preferably, the impairment function includes pixel loss, adversarial loss, and improved perceptual loss.

[0025] The L1 loss (mean absolute error) is used as the pixel loss, and the formula is as follows:

[0026]

[0027] Where H, W, and C represent the height, width, and number of channels of the image, respectively;

[0028] The formula for adversarial loss is:

[0029]

[0030] D is the discriminator, where D(Y, X) represents the probability that the discriminator considers the real image to be more realistic than the generated image; and D(X,Y) represents the probability that the generated image is more realistic than the real image.

[0031] The improved formula for perceptual loss is:

[0032]

[0033] In the above formula, This represents the feature map of the l-th layer in the VGG-19 network. This is the size of the feature map.

[0034] Preferably, based on the topological characteristics of fundus images, an index clDice that can evaluate fundus structures is added to guide the balance between the realism of structures such as blood vessels in the generated images and the feature learning of diseases, ensuring the topological connectivity of blood vessels and the accuracy of macroscopic structures.

[0035]

[0036]

[0037]

[0038]

[0039] in Representing the truth graph, This indicates the predicted segmentation result. and These represent the skeletons shown in the two figures respectively.

[0040] Preferably, the feature extraction in S3 includes extracting features from the facial infrared image, as follows:

[0041] S3-1-1. Preprocess the acquired infrared thermal imaging sequences of the face, cheeks and forehead to ensure spatial temperature consistency of the time series.

[0042] S3-1-2, Identify key anatomical structures and temperature-time phase points;

[0043] S3-1-3. Define the region of interest (ROI) on the image for feature extraction, and use face-specific ROIs to delineate independent analysis regions for feature extraction.

[0044] S3-1-4. Extract temperature dynamics features, including baseline temperature, peak temperature and time to peak, and area under the temperature change curve.

[0045] S3-1-5. Extract morphological temperature distribution features, including temperature anomaly detection and quantification, and temperature uniformity calculation;

[0046] S3-1-6. Extract heat conduction characteristics, including heat diffusion rate and temperature difference in symmetrical regions.

[0047] Preferably, the feature extraction in S3 includes feature analysis of the generated FFA image, with the following steps:

[0048] S3-2-1. Preprocess the generated FFA image sequence to align it with the time series and ensure the spatial consistency of the time series.

[0049] S3-2-2, Identify key anatomical structures and time phase points;

[0050] S3-2-3. Define the region of interest (ROI) on the image for feature extraction, and use the blood vessel-specific ROI to extract features from the segmented analysis region defined along a specific arteriovenous trunk.

[0051] S3-2-4. Extract time-dynamic features, including local filling time, peak time and intensity, and area under the curve:

[0052] S3-2-5. Extracting morphological perfusion features, including detection and quantification of non-perfusion areas and calculation of perfusion density:

[0053] S3-2-6. Extract permeability characteristics, including leakage index and leakage area diffusion ratio.

[0054] Preferably, the specific steps of S4 are as follows:

[0055] S6-1, Feature preparation;

[0056] S6-1-1. Obtain the processed feature representations of the facial infrared thermal imaging image and the fundus microcirculation images of both eyes respectively.

[0057] S6-1-2. Use facial infrared features as Q (Query) and fundus microcirculation features of the left and right eyes as two independent sets of K (Key) and V (Value), respectively.

[0058] S6-2. Fusion is achieved through a cross-attention mechanism, formula:

[0059]

[0060] in It is the dimension of the key;

[0061] S6-3. The processed feature maps are stitched together to form a comprehensive feature map;

[0062] S6-4. The fully connected layer performs deep processing on the spliced ​​comprehensive feature map to further extract core features and suppress redundant information. Compared with the traditional single splicing and fusion algorithm, it improves the feature extraction accuracy while taking into account the fusion effect and computational efficiency.

[0063] This application also provides a multimodal facial blood flow perfusion image acquisition and analysis device that combines infrared thermal imaging and fundus microcirculation imaging, including an information acquisition device and an intelligent analysis device.

[0064] Preferably, the information acquisition device includes an infrared thermal imager, a fundus image acquisition component, and a data transmission component. The infrared thermal imager is used to acquire infrared images of the user's face, extract the temperature of key points on the cheeks and forehead, monitor facial temperature changes to reflect blood perfusion status, and simultaneously perform thermal noise suppression, temperature contrast enhancement, and image registration. The fundus image acquisition component is used to acquire high-definition color fundus images of the user's left and right eyes in a time-division process, and the user's pupils must be dilated in one eye before acquisition. The data transmission component is used to transmit the data acquired by the infrared thermal imager and the fundus image acquisition component to an intelligent analysis device. The intelligent analysis device is used to run an improved Real-ESRGAN model, feature extraction algorithm, cross-attention fusion algorithm, and fully connected layer processing algorithm to realize image conversion, feature extraction, feature fusion, and result output functions.

[0065] Compared with the prior art, this application has at least the following beneficial effects:

[0066] (1) The multimodal facial blood flow perfusion image acquisition device prepared for carotid artery stenosis of the present invention has the advantage of combining facial infrared and left and right eye fundus microcirculation features for analysis, compared with intelligent analysis system and separate facial infrared or fundus microcirculation. It adopts integrated acquisition technology, video analysis technology, deep learning technology and exercise load test, which greatly improves the accuracy of physiological data measurement, while optimizing the portability of the device. It can efficiently capture facial blood flow perfusion features and provide technical support for the accurate acquisition of blood flow related physiological data.

[0067] (2) The present invention adopts a stimulation scheme that combines binocular time-sharing acquisition with standardized breath-holding test. This design not only ensures the clarity and consistency of image acquisition for each eye, but also effectively reveals potential abnormalities in vascular regulation function and bilateral asymmetry through physiological load, providing objective and repeatable technical basis for dynamic observation of blood perfusion data and improving the ability to identify data differences.

[0068] (3) This device integrates infrared thermal imaging image acquisition and left and right eye fundus microcirculation image acquisition functions, enabling the acquisition of multiple physiological information in a single operation, avoiding the inconvenience and additional burden on patients caused by multiple examinations. Considering the user experience, this device adopts a lightweight design, making it easy to carry and operate. In addition, the operation process of the device is simple and clear, suitable for home self-examination or use in health service centers. When using this device for examination, the patient only needs to place their face and eyes in the acquisition area of ​​the device according to the instructions. The device will automatically perform simultaneous acquisition of infrared thermal imaging and left and right eye fundus images. The whole process is fast and efficient, usually completed within 10 minutes, significantly improving data acquisition efficiency, reducing the user threshold and waiting cost, and enhancing the convenience of non-invasive acquisition.

[0069] (4) This invention effectively solves the problems of invasiveness and complex operation in traditional FFA examinations. Through the improved generative model, it is possible to obtain hemodynamically significant image information from conventional non-invasive color fundus images, thereby achieving FFA assessment without the need for intravenous contrast agents. This significantly reduces operational risks and user concerns, broadens the applicable population for blood perfusion data collection, and can be applied to large-scale routine physiological data screening, highlighting the universal value of non-invasive data collection.

[0070] (5) This invention innovatively applies an improved Real-ESRGAN model specifically designed for high-quality, high-fidelity cross-modal generation of images from color fundus images (CF) to fluorescein angiography (FFA) style images. This model is made more suitable for image style transfer tasks at the same resolution by removing the upsampling module from the original super-resolution task and optimizing the network depth. Crucially, during model training, this application introduces the clDice loss function, which focuses on preserving vascular topology, as part of the improved perceptual loss. This improvement ensures that the generated images not only closely resemble real FFA images in appearance but also accurately maintain the physiological connectivity and branching structure of the fundus vascular network, guaranteeing the accuracy and reliability of the extracted hemodynamic parameters (such as non-perfusion areas, leakage, filling time, etc.), providing a high-quality data foundation for subsequent feature analysis.

[0071] (6) This invention employs advanced deep learning algorithms to fuse and analyze multimodal images, enriching the extraction of blood perfusion features and significantly improving the accuracy and reliability of image analysis. The Swin Transformer effectively fuses information from different modalities, extracting richer features. The multi-head attention mechanism captures information from different levels and angles through parallel computation of multiple attention heads. This feature makes the extracted blood perfusion features more comprehensive, providing technical support for in-depth analysis of physiological data and further improving the accuracy of parameter extraction.

[0072] (7) This invention integrates multiple technologies such as integrated acquisition, video analysis, deep learning and exercise load testing to achieve rapid non-invasive acquisition and accurate analysis of blood flow perfusion data. Through multimodal feature fusion, it efficiently extracts core blood flow parameters from infrared and FFA images, greatly improving the efficiency, non-invasiveness and accuracy of data acquisition, forming a comprehensive technical advantage, and providing an efficient technical solution for the research and application of blood flow-related physiological data. Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating a multimodal facial blood flow perfusion image acquisition and analysis system based on combined infrared thermal imaging and fundus microcirculation imaging, according to one embodiment of this application.

[0074] Figure 2 This is a schematic diagram of the hardware acquisition device in a multimodal facial blood flow perfusion image acquisition and analysis system for combined infrared thermal imaging and fundus microcirculation imaging, according to one embodiment of this application.

[0075] Figure 3 This is a schematic diagram illustrating feature fusion of a multimodal facial blood flow perfusion image acquisition and analysis system based on combined infrared thermal imaging and fundus microcirculation images, according to one embodiment of this application.

[0076] Figure 4 This is a result image of the Real-ESRGAN model, a multimodal facial blood flow perfusion image acquisition and analysis system based on combined infrared thermal imaging and fundus microcirculation images, according to one embodiment of this application. Detailed Implementation

[0077] Please see Figure 1 This application provides a multimodal facial blood flow perfusion image acquisition and analysis system for combined infrared thermal imaging and fundus microcirculation imaging, including an information acquisition device and an intelligent analysis system. The information acquisition device is used to collect information from the user, and the intelligent analysis system is used to receive and analyze the collected information.

[0078] In one implementation, please refer to Figure 2The information acquisition device includes an infrared thermal imager, a fundus image acquisition component, and a data transmission component. The infrared thermal imager is used to acquire infrared images of the user's face. Specifically, the infrared thermal imager is used to extract the facial temperature at key points on the cheeks and forehead, and monitor facial temperature changes in real time during the acquisition process to reflect the blood perfusion status. At the same time, thermal noise suppression, temperature contrast enhancement, and image registration based on facial feature points are performed simultaneously during the acquisition process to eliminate environmental thermal radiation interference and inter-frame offset caused by head micro-movements, ensuring spatial temperature consistency.

[0079] The fundus image acquisition component is used to acquire high-definition color fundus images of the user's left and right eyes. The acquisition process must be carried out in a time-sharing manner: first, the pupil of one eye of the user is dilated. After the pupil dilation is completed, the component acquires a high-definition color fundus image of that eye. After the acquisition of one eye is completed, the other eye is dilated and a high-definition color fundus image is acquired in the same manner to ensure the clarity and consistency of the images of both eyes.

[0080] The data transmission device is used to synchronously transmit infrared image data acquired by the infrared thermal imager and high-definition color fundus image data acquired by the fundus image acquisition component to the intelligent analysis system, and the intelligent analysis system analyzes the data.

[0081] In one embodiment, the intelligent analysis system uses an improved Real-ESRGAN model specifically designed to achieve high-quality, high-fidelity cross-modal generation from color fundus images (CF) to fluorescein angiography (FFA) style images.

[0082] Furthermore, this application also provides the steps of the aforementioned multimodal facial blood flow perfusion image acquisition and analysis system for combined infrared thermal imaging and fundus microcirculation imaging:

[0083] S1. Design a pre-examination information collection form based on medical knowledge and collect relevant data;

[0084] An information collection form was designed based on relevant medical background knowledge. Relevant medical data was collected; based on the information collection results, a personalized exercise load plan was developed.

[0085] The information collected includes: 1) the patient's gender, age, height, and weight; 2) whether the patient has a history of unstable angina, uncontrolled hypertension, or severe arrhythmia; 3) whether the patient has a cold or fever; and 4) whether the patient has any skin diseases.

[0086] S2. Conduct a personalized breath-holding test;

[0087] In one implementation, the examiner can conduct a breath-holding test according to a personalized protocol provided, i.e., the breath-holding time.

[0088] Personalized plan: 20-40 years old, hold your breath for 40 seconds; 40-60 years old, hold your breath for 30 seconds; 60-80 years old, hold your breath for 20 seconds; 80 years and older, hold your breath for 15 seconds.

[0089] S3. Infrared images of the user's face are captured, and color fundus images of the patient's left and right eyes are captured at different times after pupil dilation.

[0090] S3-1. Acquiring multimodal data

[0091] S3-1-1 extracts facial temperature from key points on the cheeks and forehead, and monitors facial temperature changes in real time during movement to reflect blood perfusion status.

[0092] S3-1-2 First, dilate one eye of the patient and acquire a high-resolution color fundus image of that eye. Then, repeat the same procedure to dilate the other eye and acquire an image.

[0093] S4. Use the improved Real-ESRGAN model to convert the acquired color fundus images into fluorescein angiography images;

[0094] S4-1, Improved Real-ESRGAN Structure: The Real-ESRGAN model is based on the RRDB module, which is upsampled and enlarged in size. When used for image generation, there is no need to enlarge the resolution. The improved Real-ESRGAN model removes the nearest neighbor sampling layer when scale=1. The layer depth is reduced to four layers. The redundant network structure is greatly simplified, and the inference efficiency of the original model is significantly improved while maintaining feature fidelity.

[0095] S4-1-1 Shallow Feature Extraction: First, a convolutional layer is used to extract shallow features from the low-resolution input image. This convolutional layer has 64 output channels, a kernel size of 3×3, a stride of 1, padding of 1, followed by a LeakyReLU activation function (with a negative slope of 0.2).

[0096] S4-1-2, Deep Feature Extraction: Shallow features are then processed through four consecutive RRDB blocks. Each RRDB block contains multiple dense connections and residual connections. Within each RRDB block, Leaky ReLU (with a negative slope of 0.2) is used as the activation function, and a residual scaling factor β=0.2 is employed for stable training.

[0097] S4-1-3, Reconstruction Module: After the deep features pass through the RRDB block, this application uses a convolutional layer (64 channels, 3×3 convolutional kernel) for feature fusion. After removing the upsampling module, another convolutional layer (3 output channels, 3×3 convolutional kernel) is used to generate a high-quality image.

[0098] S4-1-4, Discriminator Architecture: The discriminator architecture adopts the U-Net discriminator, featuring an encoder-decoder structure. The encoder consists of multiple convolutional layers, each followed by a Leaky ReLU activation function and downsampling with a stride of 2, progressively extracting features and reducing spatial resolution. The decoder restores spatial resolution through upsampling and convolutional layers, ultimately outputting a discrimination result map of the same size as the input image, with each pixel corresponding to a true / false probability. This discriminator can capture image features at multiple scales, helping the generator recover more detailed textures.

[0099] S4-1-5, Loss Function: To train the image generation model, a composite loss function is adopted, which guides the generator training from three aspects: pixel accuracy, feature similarity, and visual realism. This function consists of three parts: pixel-level L1 loss, adversarial loss, and improved perceptual loss. An innovative loss function design is introduced, combining clDice topological loss with VGG-19 improved perceptual loss to specifically address the problem of retinal vascular topological connectivity distortion, significantly improving the accuracy of vascular branch structures and spatial morphology in generated images, and precisely adapting to the needs of physiological data feature extraction.

[0100] The total loss function is defined as follows:

[0101]

[0102] Where λ is a hyperparameter that balances the weights of each loss term.

[0103] 1. Pixel loss

[0104] L1 loss (mean absolute error) is used as the pixel loss to measure the difference between the generated image (X) and the real image (Y) at the pixel level, ensuring the consistency of the generated and original images in overall structure and color, and providing basic supervision signals for the model. Compared to L2 loss (mean squared error), L1 loss is less sensitive to outliers and produces sharper image edges. Its calculation formula is as follows:

[0105]

[0106] Where H, W, and C represent the height, width, and number of channels of the image, respectively.

[0107] 2. Combating Losses

[0108] The model used is also based on the Generative Adversarial Network (GAN) framework. To improve the visual realism of the generated images, it specifically uses the adversarial loss from Relativistic Average GAN (RaGAN). Compared to previous methods of judging whether an image is real or generated, RaGAN focuses more on determining whether one image is more realistic than another. The adversarial loss of the generator is defined as:

[0109]

[0110] Here, D is the discriminator, and D(Y, X) represents the probability that the discriminator considers the real image to be more realistic than the generated image. Similarly, D(X, Y) represents the probability that the generated image is more realistic than the real image. The discriminator D is stably trained using a Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP).

[0111] 3. Improve perceived loss

[0112] Perceptual loss aims to compare the similarity between generated and real images in a high-level feature space, rather than at the pixel level. This guides the generator to learn more structural and textural details of the image, improving the realism between generated and real images. Therefore, features from the previous five convolutional layers activated in each residual block of the model are fed into a VGG-19 network pre-trained on ImageNet to compute the perceptual loss, and L1 loss between feature maps is also calculated.

[0113]

[0114] in, This represents the feature map of the l-th layer in the VGG-19 network. This is the size of the feature map.

[0115] Specifically, considering the topological characteristics of fundus images, an index called clDice, which can evaluate fundus structures, was added to guide the balance between the realism of structures such as blood vessels in the generated images and the learning of disease features, ensuring the topological connectivity of blood vessels and the accuracy of macroscopic structures.

[0116]

[0117]

[0118]

[0119]

[0120] in Representing the truth graph, This indicates the predicted segmentation result. and These represent the skeletons shown in the two figures respectively.

[0121] S5. Extract features from the facial infrared image and the generated FFA image, respectively;

[0122] S5-1. Perform feature analysis on the generated FFA image.

[0123] S5-1-1. Preprocess the generated FFA image sequence to align it with the time sequence, including image denoising, contrast enhancement, and image registration based on vascular features, to eliminate inter-frame displacement caused by the patient's slight eye movements and ensure the spatial consistency of the time sequence.

[0124] S5-1-2, Identify key anatomical structures and time phase points;

[0125] S5-1-3. Based on clinical diagnostic standards, define regions of interest (ROIs) on the image for feature extraction, and use vascular-specific ROIs to extract features from segmented analysis regions delineated along specific arteriovenous trunks.

[0126] S5-1-4. Extract time-dynamic features, including local filling time, peak time and intensity, and area under the curve:

[0127] Filling time: Calculate the time point at which fluorescence first appears within the ROI ( ) and the time point at which the fluorescence intensity within the ROI reaches a stable plateau ( ), their difference This refers to the local filling time of the ROI.

[0128] Peak time and peak intensity: Extract the maximum intensity value from the average fluorescence intensity time curve within this ROI ( ) and their corresponding time points ( ).

[0129] Area under the curve: The area under the fluorescence intensity curve from the appearance of fluorescence to a preset late time point (e.g., T=300 seconds) is calculated as a quantitative indicator of the total fluorescence load of the ROI.

[0130] S5-1-5. Extracting morphological perfusion features, including detection and quantification of non-perfusion areas and calculation of perfusion density:

[0131] Detection and quantification of non-perfusion regions: Using threshold segmentation and region growing algorithms, continuous regions within a Region of Interest (ROI) with fluorescence intensity more than two standard deviations below the background mean are identified and defined as non-perfusion regions. Their area, perimeter, and distance from the fovea are calculated.

[0132] Perfusion density calculation: For the microcirculation ROI, the filled capillary network is first extracted using a vessel segmentation algorithm, and then the perfusion density is calculated: ;

[0133] S5-1-6. Extract permeability characteristics, including leakage index and leakage area diffusion ratio:

[0134] Leakage Index: Calculation ,in and These represent the average intensity of late and early images, respectively.

[0135] Leakage area diffusion ratio: The ratio of the area of ​​fluorescence in a late image that extends beyond the vessel boundary in an early image to the original vessel area of ​​that ROI.

[0136] S5-2, Feature analysis of infrared thermal imaging

[0137] S5-2-1. Preprocess the acquired infrared thermal imaging sequences of the face, cheeks and forehead to align them with the time sequence, including thermal noise suppression, temperature contrast enhancement and image registration based on facial feature points, in order to eliminate inter-frame shifts caused by patient head movements and environmental thermal radiation interference, and ensure the temperature spatial consistency of the time sequence.

[0138] S5-2-2, Calibrate key anatomical structures and temperature-time phase points.

[0139] S5-2-3. Based on clinical assessment guidelines, define regions of interest (ROIs) on the image for feature extraction. Using facial region-specific ROIs, delineate independent analysis regions for feature extraction in the entire forehead, left cheek, and right cheek.

[0140] S5-2-4. Extract temperature kinetic characteristics, including baseline temperature, peak temperature and time to peak, and area under the temperature change curve:

[0141] Baseline temperature: The average temperature value (Tbase) within the ROI during the initial stabilization period (first 20 seconds) of the thermal imaging sequence is calculated as a reference for the physiological temperature of that facial region.

[0142] Peak temperature and time to peak: Extract the maximum temperature value (Tmax) and its corresponding time point (Tpeak) from the temperature-time curve within the ROI.

[0143] Area under the temperature change curve: Calculate the area under the temperature-time curve from the start of observation to the preset late time point (e.g., T=300 seconds), as a quantitative indicator of the total heat metabolic load of the ROI;

[0144] S5-2-5. Extract morphological temperature distribution features, including temperature anomaly detection and quantification, and temperature uniformity calculation:

[0145] Temperature Anomaly Detection and Quantification: Using adaptive threshold segmentation and region growing algorithms, continuous regions within the ROI with temperatures 1.0℃ higher than or lower than the normal physiological threshold for the face (36.0℃-37.2℃) are identified, and their area, perimeter, and straight-line distance from the midpoint of the nasal root are calculated.

[0146] Temperature uniformity calculation: For each facial ROI, the effective temperature distribution range is first extracted using a temperature gradient clustering algorithm, and then the following calculation is made: Temperature uniformity = (1 - standard deviation of temperature within ROI / baseline temperature) × 100%;

[0147] S5-2-6. Extract heat conduction characteristics, including heat diffusion rate and temperature difference in symmetrical regions:

[0148] Thermal diffusion rate: Calculate the thermal diffusion rate = (Tmax - Tbase) / (Tpeak - Tstart), where Tstart is the observation start time point, reflecting the conduction efficiency of the temperature rising from the baseline to the peak within the ROI.

[0149] Temperature difference in symmetrical regions: The average temperature difference between the left and right cheek regions (ROI) during the same period (Tsym = Tright - Tlef) was measured. The average temperature difference between the left and right symmetrical areas of the forehead during the same period is used as a quantitative indicator of facial temperature symmetry.

[0150] S6. Perform feature fusion on the obtained infrared and FFA features; such as... Figure 3 As shown, the specific steps of S6 are as follows:

[0151] S6-1, Feature preparation;

[0152] S6-1-1. Obtain the processed feature representations of the facial infrared thermal imaging image and the fundus microcirculation images of both eyes respectively.

[0153] S6-1-2. Use facial infrared features as Q (Query) and fundus microcirculation features of the left and right eyes as two independent sets of K (Key) and V (Value).

[0154] S6-2. Fusion is achieved through a cross-attention mechanism, formula:

[0155]

[0156] in It is the dimension of the key.

[0157] S6-3. The processed feature maps are spliced ​​together to form a comprehensive feature map.

[0158] S6-4. The fully connected layer performs deep processing on the spliced ​​comprehensive feature map to further extract core features and suppress redundant information. Compared with the traditional single splicing and fusion algorithm, it improves the feature extraction accuracy while taking into account the fusion effect and computational efficiency.

[0159] S7. Output feature results.

[0160] The blood perfusion characteristic parameters are output through the Softmax function, and combined with the difference analysis of the fundus microcirculation characteristics of the left and right eyes, the output results are as follows: Figure 4 As shown, this is used to determine the blood flow status.

[0161] This invention integrates multiple technologies such as one-piece acquisition, video analysis, deep learning, and exercise stress testing to achieve rapid, non-invasive acquisition and accurate analysis of blood flow perfusion data. Through multimodal feature fusion, it efficiently extracts core blood flow parameters from infrared and FFA images, significantly improving the efficiency, non-invasiveness, and accuracy of data acquisition, forming a comprehensive technical advantage and providing an efficient technical solution for the research and application of blood flow-related physiological data.

Claims

1. A method for acquiring and analyzing multimodal facial blood flow perfusion images by combining infrared thermal imaging and fundus microcirculation imaging, characterized in that: The steps are as follows: S1: Information Acquisition: Acquire basic information, facial infrared images, and high-definition color fundus images of the left and right eyes; S2: An improved Real-ESRGAN model is used to convert the acquired high-resolution color fundus images into fluorescein angiography (FFA) images. S3: Feature Extraction: The generated FFA image and the acquired facial infrared image are preprocessed and time-aligned, key structures and phase points are calibrated, and the region of interest is defined. Then, the temporal dynamics, morphological perfusion features and permeability features of the FFA image are extracted, and the thermodynamics, morphological temperature distribution features and thermal conduction features of the facial infrared image are extracted. S4: Feature Fusion: Features extracted from facial infrared images are used as query vectors, and the microcirculation-related features of the fundus of the left and right eyes are used as two independent key vectors and value vectors, respectively. They are fused through a cross-attention mechanism, and after being spliced ​​to form a comprehensive feature map, the core features are further extracted through a fully connected layer. S5: Output Results: The Softmax function outputs blood perfusion characteristic parameters and analyzes the differences in fundus microcirculation characteristics between the left and right eyes.

2. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 1, characterized in that: The improved Real-ESRGAN structure in S2 is as follows: The Real-ESRGAN model is based on the RRDB module, and the size is enlarged after upsampling. When it is used for image generation, there is no need to enlarge the resolution. The improved Real-ESRGAN model removes the nearest neighbor sampling layer when scale=1; and reduces the layer depth to four layers. The redundant network structure is greatly simplified, and the inference efficiency of the original model is significantly improved while maintaining feature fidelity.

3. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 1, characterized in that: The specific structure of the improved Real-ESRGAN in S2 is as follows: S2-1-1 Shallow Feature Extraction: First, a convolutional layer is used to extract shallow features from the low-resolution input image; this convolutional layer has 64 output channels, a kernel size of 3×3, a stride of 1, padding of 1, followed by a Leaky ReLU activation function; S2-1-2, Deep Feature Extraction: The shallow features are then processed through four consecutive RRDB blocks; each RRDB block contains multiple dense connections and residual connections. S2-1-3, Reconstruction Module: After deep features pass through the RRDB block, a convolutional layer is used for feature fusion; after removing the upsampling module, another convolutional layer is used to generate a high-quality image; S2-1-4, Discriminator Architecture: The discriminator architecture adopts the U-Net discriminator, which has an encoder-decoder structure; S2-1-5 Loss Function: To train the image generation model, a composite loss function is used, which consists of three parts: pixel-level L1 loss, adversarial loss, and improved perceptual loss. By introducing an innovative design of the loss function, combining clDice topological loss with VGG-19 improved perceptual loss, we can specifically solve the problem of distortion in the topological connectivity of fundus blood vessels, significantly improve the accuracy of blood vessel branch structure and spatial morphology in the generated image, and accurately adapt to the needs of physiological data feature extraction. The total loss function is defined as follows: Where λ is a hyperparameter that balances the weights of each loss term.

4. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 3, characterized in that: The impairment function includes pixel loss, adversarial loss, and improved perceptual loss. The L1 loss (mean absolute error) is used as the pixel loss, and the formula is as follows: Where H, W, and C represent the height, width, and number of channels of the image, respectively; The formula for adversarial loss is: D is the discriminator, where D(Y, X) represents the probability that the discriminator considers the real image to be more realistic than the generated image; and D(X, Y) represents the probability that the generated image is more realistic than the real image. The improved formula for perceptual loss is: In the above formula, This represents the feature map of the l-th layer in the VGG-19 network. This is the size of the feature map.

5. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 4, characterized in that: Based on the topological characteristics of fundus images, an index clDice, which can evaluate fundus structures, was added to guide the balance between the realism of structures such as blood vessels in the generated images and the learning of disease features, ensuring the topological connectivity of blood vessels and the accuracy of macroscopic structures. in Representing the truth graph, This indicates the predicted segmentation result. and These represent the skeletons shown in the two figures respectively.

6. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 1, characterized in that: The feature extraction in S3 includes extracting features from the facial infrared image, and the steps are as follows: S3-1-1. Preprocess the acquired infrared thermal imaging sequences of the face, cheeks and forehead to ensure spatial temperature consistency of the time series. S3-1-2, Identify key anatomical structures and temperature-time phase points; S3-1-3. Define the region of interest (ROI) on the image for feature extraction, and use face-specific ROIs to delineate independent analysis regions for feature extraction. S3-1-4. Extract temperature dynamics features, including baseline temperature, peak temperature and time to peak, and area under the temperature change curve. S3-1-5. Extract morphological temperature distribution features, including temperature anomaly detection and quantification, and temperature uniformity calculation; S3-1-6. Extract heat conduction characteristics, including heat diffusion rate and temperature difference in symmetrical regions.

7. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 1, characterized in that: The feature extraction in S3 includes feature analysis of the generated FFA image, and the steps are as follows: S3-2-1. Preprocess the generated FFA image sequence to align it with the time series and ensure the spatial consistency of the time series. S3-2-2, Identify key anatomical structures and time phase points; S3-2-3. Define the region of interest (ROI) on the image for feature extraction, and use the blood vessel-specific ROI to extract features from the segmented analysis region defined along a specific arteriovenous trunk. S3-2-4. Extract time-dynamic features, including local filling time, peak time and intensity, and area under the curve: S3-2-5. Extracting morphological perfusion features, including detection and quantification of non-perfusion areas and calculation of perfusion density: S3-2-6. Extract permeability characteristics, including leakage index and leakage area diffusion ratio.

8. The method for acquiring and analyzing multimodal facial blood flow perfusion images combining infrared thermal imaging and fundus microcirculation imaging according to claim 1, characterized in that: The specific steps of S4 are as follows: S6-1, Feature preparation; S6-1-1. Obtain the processed feature representations of the facial infrared thermal imaging image and the fundus microcirculation images of both eyes respectively. S6-1-2. Use facial infrared features as Q (Query) and fundus microcirculation features of the left and right eyes as two independent sets of K (Key) and V (Value), respectively. S6-2. Fusion is achieved through a cross-attention mechanism, formula: in It is the dimension of the key; S6-3. The processed feature maps are stitched together to form a comprehensive feature map; S6-4. The fully connected layer performs deep processing on the spliced ​​comprehensive feature map to further extract core features and suppress redundant information. Compared with the traditional single splicing and fusion algorithm, it improves the feature extraction accuracy while taking into account the fusion effect and computational efficiency.

9. A multimodal facial blood flow perfusion image acquisition and analysis device combining infrared thermal imaging and fundus microcirculation imaging, characterized in that: This includes information collection devices and intelligent analysis equipment.

10. The multimodal facial blood flow perfusion image acquisition and analysis device combining infrared thermal imaging and fundus microcirculation imaging according to claim 9, characterized in that: The information acquisition device includes an infrared thermal imager, a fundus image acquisition component, and a data transmission component. The infrared thermal imager is used to acquire infrared images of the user's face, extract the temperature of key points on the cheeks and forehead, monitor facial temperature changes to reflect blood perfusion status, and simultaneously perform thermal noise suppression, temperature contrast enhancement, and image registration. The fundus image acquisition component is used to acquire high-definition color fundus images of the user's left and right eyes in a time-division process, requiring pupil dilation of each eye before acquisition. The data transmission component is used to transmit the data acquired by the infrared thermal imager and the fundus image acquisition component to an intelligent analysis device. The intelligent analysis device is used to run an improved Real-ESRGAN model, feature extraction algorithm, cross-attention fusion algorithm, and fully connected layer processing algorithm to achieve image conversion, feature extraction, feature fusion, and result output functions.

Images