Macular pigment density analysis method and system based on multispectral imaging and deep learning, and storage medium

Abstract

The application discloses a macular pigment optical density analysis method and system based on multispectral imaging and deep learning and a storage medium. The method comprises the following steps: collecting multispectral fundus images of a human eye retina under narrow-band wavelengths and performing registration; automatically segmenting a macular region and a foveal region by using a pre-trained deep learning segmentation model; extracting multispectral features from the segmented region; inputting the features into a deep regression model with physical information constraints to output the macular pigment optical density values of each position; and finally generating an MPOD spatial distribution map and performing visualization. The application innovatively introduces a physical information constraint neural network, integrates the Beer-Lambert law as a physical loss term into model training, significantly improves the accuracy, robustness, interpretability and device independence of MPOD quantification, overcomes the technical defects of existing subjective methods and traditional optical imaging methods, and can provide a powerful tool for ophthalmic disease screening and monitoring.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of ophthalmic medical image analysis, and particularly relates to a macular pigment density analysis method and system based on multi-spectral imaging and deep learning and a storage medium. BACKGROUND

[0002] Macular pigment (MP) is mainly composed of lutein and zeaxanthin, and is selectively deposited in the macular region of the human retina. It plays a crucial role in protecting retinal photoreceptor cells and delaying photooxidative damage by absorbing high-energy short-wave blue light and exerting antioxidant effects. Macular pigment optical density (MPOD) is a key biomarker for quantifying the content of macular pigment. Numerous studies have shown that a low level of MPOD is significantly associated with the risk of age-related macular degeneration (AMD) and other ocular fundus diseases. Therefore, accurate, objective and quantitative measurement of MPOD has great clinical significance for early screening, diagnosis, disease monitoring and nutritional intervention effect evaluation of ophthalmic diseases.

[0003] Currently, the measurement techniques of MPOD can be mainly divided into subjective psychophysical methods and objective optical imaging methods.

[0004] The typical representative of the subjective psychophysical method is the heterochromatic flicker photometry (HFP) method, which was once regarded as the "gold standard" for MPOD measurement. HFP relies on the subject's subjective judgment of the fusion point of blue-green flicker light, and the accuracy and repeatability of the measurement results are greatly affected by the subject's cooperation, understanding ability and visual function state, and the applicability is poor for patient groups with cognitive impairment or poor vision. In addition, HFP can only provide MPOD values for a few discrete points in the fovea, and cannot obtain two-dimensional spatial distribution information reflecting the overall health status of the macular region.

[0005] To overcome the limitations of subjective methods, objective optical imaging methods have emerged. For example, Chinese Patent Application No. CN202010257198.7 discloses a macular pigment optical density measurement method and device based on fundus images, which estimates MPOD by analyzing the intensity ratio of blue and green channels of the image, and automatically locates the macula and fovea using a neural network. Although this method realizes the application of conventional fundus cameras and has a lower cost, the RGB image only contains three wide-band spectral information, and the spectral resolution is seriously insufficient, which makes it difficult to accurately distinguish the absorption characteristics of macular pigment and other retinal pigments (such as melanin), and there is an inherent bottleneck in measurement accuracy. In addition, this method relies on pre-collected large amounts of data for specific devices for empirical normalization, which is cumbersome to operate in actual application, and has poor adaptability to parameter drift caused by new models or device aging.

[0006] Other objective methods such as dual-wavelength autofluorescence imaging can provide a two-dimensional distribution map of MPOD, but the signal is easily disturbed by various factors such as the health status of retinal pigment epithelial cells and lipofuscin concentration, which may introduce measurement errors.

[0007] In recent years, multispectral and hyperspectral imaging technology has brought new opportunities for MPOD measurement, which can collect images in multiple or continuous narrow bands, providing spectral dimension information far beyond that of RGB images. However, existing solutions based on multispectral technology rely on simplified physical models (such as the Beer-Lambert law) and linear unmixing algorithms, which cannot fully describe the nonlinear effects of light propagation in the complex multi-layered retina. At the same time, how to robustly extract MPOD features from high-dimensional spectral data and effectively overcome the influence of individual differences, other pigment interference and device differences is still a great challenge for current technology.

[0008] In summary, existing MPOD measurement technologies have different degrees of deficiencies in measurement accuracy, objectivity, spatial resolution, robustness, and compatibility with device differences. There is an urgent need in the field for a new method of MPOD quantitative analysis that can fully utilize advanced imaging technology and intelligent data analysis algorithms to achieve high precision, high robustness, provide detailed spatial distribution information, and be easily promoted in clinical practice. SUMMARY

[0009] The present application aims to solve the above technical problems in the prior art and provides a macular pigment density analysis method, system and storage medium based on multispectral imaging and deep learning, which combines multispectral fundus imaging technology with advanced deep learning models, especially introduces a neural network with physical information constraints, to realize accurate, objective, robust and good generalization ability of quantitative analysis of macular pigment, and provide detailed MPOD spatial distribution information.

[0010] To solve the above technical problems, the present application adopts the following technical solutions:

[0011] The macular pigment density analysis method based on multispectral imaging and deep learning includes the following steps:

[0012] Step one: collect multispectral fundus images of human eye retina under multiple narrowband wavelengths, and perform registration processing on the multispectral fundus images;

[0013] Step two: input the registered multispectral fundus images into a pre-trained deep learning segmentation model to segment the macular region and foveal region;

[0014] Step three: extract multispectral features from the segmented macular region and foveal region;

[0015] Step four: input the multispectral features into a deep regression model with physical information constraints, and output the macular pigment optical density value at each position in the macular region from the deep regression model;

[0016] Step five: generate an MPOD spatial distribution map of the macular region based on the macular pigment optical density value.

[0017] Further, in step one, the multispectral fundus images are collected by a multispectral fundus camera.

[0018] Further, in step one, the narrowband wavelengths include a macular pigment absorption band and at least one reference band.

[0019] Further, the macular pigment absorption band includes at least one wavelength in the range of 400nm-460nm, and the reference band includes at least one wavelength in the range of 500nm-540nm and / or 570nm-660nm.

[0020] Further, in step four, the deep regression model with physical information constraints is a physical information constraint neural network.

[0021] Further, in the training process of the physical information constraint neural network, the total loss function includes a data loss term and a physical loss term;

[0022] The data loss term is used to measure the difference between the predicted macular pigment optical density value and the true value;

[0023] The physical loss term is constructed based on the Beer-Lambert law, and is used to measure the difference between the theoretical reflected light intensity calculated based on the predicted macular pigment optical density value and the actually measured reflected light intensity.

[0024] Further, in step three, the multispectral features include at least one of the following:

[0025] Average pixel intensity: average pixel intensity of each wavelength channel within the macular region and foveal region;

[0026] Pixel intensity histogram: pixel intensity histogram of each wavelength channel within the macular region and foveal region;

[0027] Spectral ratio: pixel intensity ratio between different wavelength channels;

[0028] Texture feature: texture feature within the macular region and foveal region.

[0029] Further, the method further comprises a longitudinal tracking and correlation analysis step: storing and managing the macular pigment optical density measurement data of the same patient at different time points, generating the trend graph of macular pigment optical density over time, and conducting correlation analysis combined with the patient's clinical information, for evaluating disease progression or intervention effect.

[0030] Further, in step two, the deep learning segmentation model is a fully convolutional neural network of U-Net architecture, DeepLabV3+ architecture, AttentionU-Net architecture or ResU-Net architecture.

[0031] Further, in step five, the MPOD spatial distribution map is visualized in the form of a pseudo-color map, a heat map or a three-dimensional surface map.

[0032] Further, the visualization is interactive visualization, supporting user query of MPOD value at a specific location in the MPOD spatial distribution map, and / or generating MPOD value cross-sectional distribution curve.

[0033] A system for implementing the macular pigment density analysis method based on multi-spectral imaging and deep learning as described above, comprising a multi-spectral fundus image acquisition module, an image preprocessing module, a deep learning segmentation module, a multi-spectral feature extraction module, a physically information constrained deep regression quantization module and an MPOD spatial distribution map generation and visualization module.

[0034] The multi-spectral fundus image acquisition module is used to acquire multi-spectral fundus images of human eye retina;

[0035] The image preprocessing module is used to perform registration processing on the acquired multi-spectral fundus images;

[0036] The deep learning segmentation module has a pre-trained deep learning segmentation model built in, which is used to receive the preprocessed multi-spectral images and output the segmentation mask of the macular region and foveal region;

[0037] The multi-spectral feature extraction module is used to extract multi-spectral features from the multi-spectral images according to the segmentation mask;

[0038] The physical information constrained deep regression quantification module is internally provided with a physical information constrained deep regression model, and is used for quantifying the macular pigment optical density value according to the extracted multi-spectral feature;

[0039] The MPOD spatial distribution map generation and visualization module is used for generating a spatial distribution map according to the quantified macular pigment optical density value and providing an interactive visualization interface, wherein the MPOD spatial distribution map is displayed in the form of a pseudo-color map, a heat map or a three-dimensional surface map, and the user is supported to perform interactive query.

[0040] Further, the system further comprises a data management and longitudinal tracking module, which is used for storing patient data, measurement results and clinical information, and providing a longitudinal tracking and trend analysis function.

[0041] A computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the macular pigment density analysis method based on multi-spectral imaging and deep learning.

[0042] The present application has the following beneficial effects due to the adoption of the above technical solutions:

[0043] The present application has high measurement accuracy and strong specificity. By using a multi-spectral fundus camera to collect images at multiple narrow-band wavelengths, rich spectral information far exceeding traditional RGB three-color imaging is obtained. This enables the system to more accurately capture the unique spectral absorption curve of macular pigment, effectively distinguishing macular pigment from other retinal pigments (such as melanin), and fundamentally solving the problem of low measurement accuracy caused by insufficient spectral information in methods based on RGB images.

[0044] The present application has strong objectivity and good repeatability. The entire process from image acquisition, region segmentation to MPOD quantification is highly automated, completely avoiding the dependence of subjective psychophysical methods such as heterochromatic flicker photometry on subject cooperation, reducing manual intervention and subjective judgment, making the measurement results more objective and reliable, and having excellent repeatability.

[0045] The present application has accurate and adaptive region segmentation. Advanced deep learning segmentation models (such as U-Net) are used to automatically identify the macula and fovea regions, which can accurately delineate the boundaries at the pixel level according to the image content, overcoming the limitations of traditional methods that rely on fixed geometric shapes or anatomical landmark distances, better adapting to anatomical differences and pathological changes between different individuals, and ensuring the accuracy of subsequent quantification regions.

[0046] The present application has excellent robustness and generalization ability. The most core innovation of the present application is to introduce a physically information constrained neural network for MPOD quantification. During the training process, the model not only learns the characteristics of the data itself, but also incorporates the Beer-Lambert law, a core physical law, as a constraint through the loss function. This enables the model to learn complex nonlinear relationships while ensuring that its output must comply with the basic principles of light and biological tissue interaction, thereby significantly enhancing the robustness to different imaging devices, different lighting conditions, and individual physiological differences, greatly improving the generalization ability and cross-device applicability of the model, and effectively solving the device compatibility and calibration problem.

[0047] The present application provides comprehensive spatial distribution information: the present method can generate an MPOD spatial distribution map of the entire macular region, not just the average of a few discrete points in the fovea. This provides clinicians with intuitive visual information on the two-dimensional distribution and density gradient of macular pigments, which helps to detect local MPOD abnormalities early and provides unprecedented detailed basis for precise intervention and treatment.

[0048] The present application has great clinical application value. By integrating longitudinal tracking and clinical correlation analysis functions, the system can long-term monitor the dynamic changes of MPOD in patients and conduct trend analysis and effect evaluation combined with clinical data. This provides a powerful quantitative tool for early diagnosis, progression monitoring, and treatment effect evaluation of age-related macular degeneration and other ophthalmic diseases, and has high clinical translation value and broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0049] The present application will be further described below in conjunction with the accompanying drawings:

[0050] Figure 1 The flowchart of the present application based on multispectral imaging and deep learning macular pigment density analysis method;

[0051] Figure 2 The architecture diagram of the present application macular pigment density analysis system;

[0052] Figure 3 The working principle diagram of the present application physical information constrained neural network (PINN). DETAILED DESCRIPTION

[0053] Example 1: Macular pigment density analysis method based on multispectral imaging and deep learning

[0054] As Figure 1 and Figure 3 The present application provides a macular pigment density analysis method based on multispectral imaging and deep learning, comprising the following steps:

[0055] Step 1: Multispectral fundus image acquisition

[0056] The multispectral fundus camera is used to collect multispectral fundus images of human eye retina at multiple narrow-band wavelengths, and the multispectral fundus images are subjected to registration processing. Different from the traditional RGB three-color imaging, the multispectral imaging captures images at multiple narrow-band wavelengths, thereby providing richer spectral dimension information.

[0057] Macular pigment (MP) has significant absorption characteristics for blue light, and the narrow-band wavelength should cover the absorption band of macular pigment (MP) and at least one reference band. Preferably, the multispectral fundus camera can collect images in the following typical wavelength ranges:

[0058] Absorption band: wavelength selected from the range of 400nm-460nm (blue light region). This band is the region with the strongest absorption of macular pigment, and is crucial for the quantification of MPOD; for example, 460nm can be selected as the absorption wavelength.

[0059] Reference band: wavelength selected from the range of 500nm-540nm (green light region) and / or 570nm-660nm (red light region). The absorption of macular pigment is weak in the green light region, and the absorption of macular pigment is almost negligible in the red light region. These bands can be used as reference benchmarks. By comparing the image information of the absorption band and the reference band, the optical characteristics of MP can be effectively separated. For example, 540nm and 630nm can be selected as the reference wavelengths.

[0060] The multispectral fundus camera of the present application can be implemented using technologies such as sequential imaging based on a filter wheel, imaging based on a tunable light source, or synchronous imaging based on a beam splitter / diffraction grating. Sequential imaging based on a filter wheel: by rapidly rotating a filter wheel with different narrow-band filters, images at different wavelengths are sequentially collected in a short time. To avoid eye movement artifacts, a high-frame-rate camera and precise synchronization control are required. Imaging based on a tunable light source: a tunable laser or LED light source is used to obtain images at different wavelengths by changing the wavelength of the light source. Synchronous imaging based on a beam splitter / diffraction grating: the fundus reflected light is decomposed into different wavelength beams by a beam splitting element, and is synchronously received by multiple detector arrays, thereby achieving multiple wavelength images in one exposure.

[0061] The images collected at different wavelengths need to be subjected to precise image registration processing to eliminate the image displacement or deformation caused by eye micro-movement, and to ensure that the different wavelength images are spatially aligned at the pixel level. The reason why the images of different spectra must be accurately registered is mainly based on the following key reasons:

[0062] First, eliminate eye micro-motion artifacts and achieve pixel-level alignment. Multispectral fundus images are usually acquired sequentially at different narrow-band wavelengths. Despite the fast acquisition speed, the patient's eyeball inevitably undergoes slight, involuntary physiological tremors (micro-motions) in the interval between sequential acquisitions. Such micro-motions can cause the images at different wavelengths to shift or deform spatially at a sub-pixel level or even a pixel level. If not registered, directly comparing the pixels at the same coordinate position in different wavelength images is actually comparing different anatomical positions on the retina, which will introduce serious alignment errors.

[0063] Second, ensure the accuracy of subsequent multispectral analysis and MPOD quantification. The core of the present application lies in analyzing the spectral absorption characteristics of the same retinal position at different wavelengths. Both the subsequent "multispectral feature extraction" (step three) and "MPOD quantification with physical information constraints" (step four) strictly depend on a basic premise: each feature vector extracted and each pixel / region used for regression must have its corresponding multispectral intensity value derived from the exact same physical location on the retina. Precise registration ensures that the images at different wavelengths are aligned to the same spatial coordinate system, forming a standardized three-dimensional data cube (two spatial dimensions X, Y; one spectral dimension - wavelength). Only in this way can the spectral curves extracted from the data cube truly reflect the absorption characteristics of the macular pigment, thereby providing accurate and reliable input for the deep learning model and ultimately obtaining a high-precision MPOD spatial distribution map.

[0064] In short, image registration is a key preprocessing step that ensures "homologous comparison" and "positional consistency", and is the cornerstone for the accurate implementation of all subsequent advanced analysis techniques.

[0065] Feature point-based (such as SIFT, SURF) or image intensity-based (such as mutual information method) registration algorithms can be used. Aligning images at different wavelengths to the same spatial coordinate system, the registered images form a three-dimensional data cube (two spatial dimensions X, Y coordinates; one spectral dimension wavelength), ensuring pixel-level accuracy for subsequent analysis.

[0066] It should be noted that the "three-dimensional" in the "three-dimensional data cube" here refers to the dimensional structure of the data, not the number of wavelengths acquired. Specifically:

[0067] • First dimension (X-axis): represents the width direction spatial coordinate of the image

[0068] • Second dimension (Y-axis): represents the height direction spatial coordinate of the image

[0069] • Third dimension (λ-axis): represents the spectral dimension, i.e., different imaging wavelengths

[0070] This means that for each spatial position (X, Y) on the retina, there is a complete spectral vector corresponding to it, containing the reflection intensity information of the position at all collected wavelengths. The multi-spectral imaging adopted by the present application collects images at multiple narrow-band wavelengths, far exceeding the three wide bands of traditional RGB imaging, thereby providing more rich spectral dimension information. For example, in actual implementation, a spectrum set including multiple wavelengths such as 460 nm, 540 nm, 630 nm, etc. can be selected to obtain complete spectral characterization of macular pigment absorption characteristics.

[0071] The organization structure of this three-dimensional data cube is the key technical basis of the present application, which provides complete spectral-spatial data support for subsequent multi-spectral feature extraction and physical information constrained deep learning analysis.

[0072] The two steps of image registration and region segmentation in the present application have a clear sequence and one-way dependence:

[0073] Registration, as an image preprocessing step, is completely independent of the segmentation step and must be completed before segmentation. The registration process only depends on the original multi-spectral image itself, and the spatial alignment of images of different wavelengths is realized through a feature point or image intensity based registration algorithm to form a standardized three-dimensional data cube. This process does not require any prior information about the macular position, and its core goal is to eliminate the image displacement or deformation caused by eyeball micro-movement.

[0074] Segmentation, as the core analysis step, must use the registered image as input. Only by ensuring that the images of different wavelengths are accurately aligned in space, can the deep learning segmentation model accurately learn the spectral features and morphological features of the macular and foveal regions, and thus output reliable segmentation results. If the unregistered image is directly segmented, the feature extraction will be based on different anatomical positions on the retina, which will seriously affect the accuracy of subsequent MPOD quantification.

[0075] This one-way dependence design of "registration first, segmentation later" ensures the spatial consistency and reliability of the entire analysis process, and is an important architectural foundation for the high-precision MPOD measurement of the present method.

[0076] Step 2: Accurate segmentation of the retinal region

[0077] The registered multi-spectral fundus image is input into a pre-trained deep learning segmentation model to automatically and accurately segment the macular region and foveal region.

[0078] The model preferably adopts a fully convolutional neural network with an encoder-decoder structure, such as a U-Net architecture, a DeepLabV3+ architecture, an Attention U-Net architecture, or a ResU-Net architecture.

[0079] In this step, the specific way in which the deep learning segmentation model processes the three-dimensional data cube is as follows: the registered multi-spectral fundus image, i.e., the three-dimensional data cube, is regarded as a two-dimensional image with N channels, where N is equal to the number of wavelengths collected. The model processes the spatial dimension (X, Y) and the spectral dimension (wavelength λ) simultaneously through its convolutional layers, extracting comprehensive information from the multi-spectral image, including spectral features such as intensity distribution at different wavelengths and morphological features such as regional shape, vascular structure, etc. This processing method enables the segmentation model to fully utilize the rich information of multi-spectral data, achieving adaptive and accurate segmentation of the macular region and foveal region.

[0080] Through learning the spectral and morphological features in the multi-spectral image, the model outputs a pixel-level binary segmentation mask of the macular region and foveal region, providing an accurate spatial range for subsequent feature extraction and MPOD quantification.

[0081] The training of this model is based on a large multi-spectral fundus image dataset accurately labeled by ophthalmic experts. During the training process, the model learns to extract comprehensive information from multi-spectral images, including spectral features (such as intensity distribution at different wavelengths) and morphological features (such as regional shape, vascular structure), enabling it to be robust to individual anatomical differences and image quality fluctuations.

[0082] The core purpose of setting the retinal region accurate segmentation step in this invention is:

[0083] First, accurately define the target region for quantitative analysis. The deep learning segmentation model automatically and accurately identifies and segments the pixel-level boundaries of the macular region and foveal region, providing a clear and reliable spatial range definition for subsequent multi-spectral feature extraction and MPOD quantification.

[0084] Second, ensure the positioning accuracy of feature extraction. The accurate segmentation mask output by the segmentation model ensures that the extraction of multi-spectral features in step three is strictly limited within the correct anatomical region. Only by extracting spectral features within the accurate macular region and foveal region can the accuracy of subsequent MPOD quantification be guaranteed, avoiding measurement deviations caused by unclear region definition.

[0085] Third, realize individualized adaptive analysis. This method overcomes the limitations of traditional methods relying on fixed geometric shapes or distances from anatomical landmarks, and can adaptively define boundaries according to the specific retinal anatomy of each patient, better adapting to anatomical differences and pathological changes among different individuals.

[0086] Fourth, the reliability of the overall measurement is improved. By precise pixel-level segmentation, the problem of ambiguous or inaccurate region definition in traditional methods is avoided, providing a reliable spatial basis for the entire MPOD quantitative analysis process, thereby significantly improving the accuracy and repeatability of the final measurement results.

[0087] It should be emphasized that since all images of different wavelengths have been accurately registered to the same spatial coordinate system in step one, forming a spatially aligned three-dimensional data cube, the deep learning segmentation model outputs a single set of macular and foveal region segmentation masks based on this data cube. This unified mask will be applied to all wavelength channels (i.e., all spectra) of the image for subsequent feature extraction. This "single mask, multi-spectrum sharing" mechanism ensures that the features extracted from different wavelength images correspond completely to the same anatomical location in space, which is the basis for subsequent high-precision MPOD quantification.

[0088] Step three: multi-spectral feature extraction

[0089] According to the segmentation mask obtained in step two, multi-spectral features for quantifying MPOD are extracted from the segmented macular and foveal regions; feature extraction is strictly limited within the target anatomical structure identified by the deep learning segmentation model, rather than processing the entire image area.

[0090] The types of features extracted include but are not limited to at least one of the following:

[0091] Average pixel intensity: calculate the average pixel intensity value of each wavelength channel in the macular and foveal regions;

[0092] Pixel intensity histogram: statistics of pixel intensity distribution of each wavelength channel in the macular and foveal regions;

[0093] Spectral ratio: calculate the pixel intensity ratio between different wavelength channels, such as the intensity ratio of 460nm and 540nm;

[0094] Texture features: texture features in the macular and foveal regions, such as gray level co-occurrence matrix (GLCM) algorithm to extract texture information in the target region. The extracted features are combined into a high-dimensional feature vector, which is used as the input of the subsequent deep regression model.

[0095] It is to be noted that the features extracted from the macular region and foveal region (including scalar, vector and matrix form features such as average pixel intensity, spectral ratio, texture feature matrix, histogram, etc.) will go through a feature vectorization process when combined into the final feature vector. Specifically, all non-1D features (such as 2D texture feature matrix, 1D histogram array, etc.) will be flattened and spliced with other scalar features to form a unified 1D high-dimensional feature vector. This vectorization process ensures that the complex multi-dimensional information extracted from the image can be regularly input into the subsequent deep regression model.

[0096] Step four: MPOD quantification with physical information constraint

[0097] The multispectral features are input into a specially trained deep regression model with physical information constraint, and the macular pigment optical density values of each position (pixel or sub-region) in the macular region are output by the deep regression model.

[0098] It is to be noted that the key to generating the MPOD spatial distribution map by this method lies in its implementation of dense, position-corresponding local quantification. Specifically: for each pixel or pre-defined sub-region (spatial coordinates x, y) in the macular region, the system will extract an independent multispectral feature vector Vx,y. This feature vector is always strictly bound to its spatial coordinates (x, y) when input into the regression model for quantification. The model outputs a corresponding MPOD value Cx,y for each Vx,y. Finally, by remapping all position Cx,y values to a two-dimensional matrix according to their original coordinates (x, y), the MPOD spatial distribution map is generated. Therefore, the feature vectorization process does not discard spatial information, but achieves the conversion from local quantification to global distribution by maintaining the strict correspondence between "position-feature-predicted value".

[0099] The model is preferably a physics-informed neural network (PINN), and its basic architecture can be a multi-layer perceptron (MLP) or a convolutional neural network (CNN).

[0100] In this step, the specific way in which the deep regression model with physical information constraint processes the three-dimensional data cube is as follows: first, according to the segmentation mask obtained in step two, multispectral features are extracted from the macular region and foveal region of the three-dimensional data cube to form a high-dimensional feature vector. This feature extraction process is essentially a dimension reduction and feature encoding of the three-dimensional data. Then, the feature vector is input into the physics-informed neural network, and the network learns to map the feature vector to the target output - the MPOD value of each pixel or pre-defined sub-region in the macular region. In the training process, the physical loss term is constructed directly using the actually measured reflectance intensity I_measured (λ), ensuring that the MPOD value predicted by the model conforms to the physical constraints of the Beer-Lambert law.

[0101] The core innovation of this model lies in its training method:

[0102] Its total loss function (L_ total It consists of two parts:

[0103] Data loss term (L_ data The mean squared error (MSE) function is used to measure the difference between the MPOD value predicted by the model and the actual MPOD value obtained by the gold standard method (such as high performance liquid chromatography, HPLC).

[0104] It should be noted that the actual MPOD value (MPOD) used to calculate the data loss item is... true The form of the ground truth depends on how the training dataset is constructed; it can be a single scalar value for the entire region (e.g., from HPLC) or a pixel-level spatial distribution map (e.g., from high-precision imaging methods). The training framework of this invention is compatible with both of these ground truth forms. When the true MPOD value is a single scalar, the MPOD spatial distribution map predicted by the model needs to be averaged over the corresponding region before calculating the data loss.

[0105] Physical loss term (L_ physics (This is based on the Beer-Lambert law. This law describes the attenuation of light in an absorbing medium: I = I0 × exp(-α × c × L);)

[0106] Where: I is the intensity of reflected light; I0 ​​is the intensity of incident light; α is the MP absorption coefficient (wavelength-dependent); c is the MP concentration (i.e., MPOD); and L is the optical path length.

[0107] The MPOD value predicted by the physical loss term penalty model is inconsistent with the laws of physics. Specifically, for each wavelength λ, using the model-predicted MPOD value c, the known MP absorption coefficient α(λ), the estimated optical path length L, and the incident light intensity I0(λ), the theoretical reflected light intensity I_ can be calculated. theory (λ). L_ physics Then calculate I_ theory (λ) and the actual measured intensity of reflected light I_ measured The mean square error between (λ).

[0108] Physical loss term L_ physicsThe calculations cover all acquired narrowband wavelengths. If N wavelengths are acquired (e.g., N=10), the theoretical reflected light intensity at each of these N wavelengths must be calculated using the MPOD value c predicted by the model. The total difference (e.g., mean square error) between these N theoretical values ​​and N actual measured values ​​is then calculated. This means that the predicted single MPOD value c must simultaneously satisfy the physical constraints of all N spectral channels, forcing the model to learn more fundamental spectral absorption characteristics and significantly improving the model's accuracy and robustness.

[0109] It should be noted that during the training and inference process of the Physical Information Constrained Neural Network (PINN) of this invention, the model outputs only one comprehensive MPOD prediction value for each pixel or predefined sub-region within the macula. This prediction value is the unique result obtained by deep regression based on the input multispectral feature vector (which integrates information from all available bands).

[0110] During the training phase:

[0111] 1. Calculation of Data Loss: For each training sample, the model's output of a single MPOD prediction is compared with a single true MPOD value obtained through the gold standard method, and the data loss (L_) is calculated. data This ensures that the model's predictions are consistent with authoritative measurements on a macroscopic level.

[0112] 2. Calculation of Physical Loss and Utilization of Multispectral Information: The core role of multispectral information is reflected in physical loss (L_ physics The calculation involves substituting the same MPOD prediction value from the model output into the Beer-Lambert law formula for each wavelength λ, and combining this with the known absorption coefficient α(λ) of the macular pigment at that wavelength to calculate the theoretical reflected light intensity I_ at that wavelength. theory (λ). Then, iterate through all the narrowband wavelengths collected, and calculate a series of I_ theory (λ) and the measured intensity I_ corresponding to wavelength λ in the actual acquired multispectral image. measured (λ) is compared and the physical loss is calculated in a comprehensive manner.

[0113] Although the input to the Physically Constrained Neural Network (PINN) is a multispectral feature vector, the implementation of physical constraints does not depend on the forward propagation process within the network, but is ensured through the calculation of the loss function during training. The specific mechanism is as follows:

[0114] During the model training phase, each training sample contains two key components: first, a feature vector extracted from the multispectral image, which serves as the input to PINN; and second, the original multispectral intensity data I_ measured(λ), as the basis of physical verification.

[0115] The forward propagation process of PINN only uses the feature vector as input, and outputs the predicted MPOD value c. While calculating the total loss function, the system calls the stored original multi-spectral intensity data: first, the theoretical reflectance intensity I_ theory (λ) at each wavelength is calculated using the predicted MPOD value c combined with the Beer-Lambert law; then I_ theory (λ) is compared with the actual I_ measured (λ), and the physical loss term L_ physics is calculated.

[0116] This design realizes the perfect combination of data-driven and physical constraints: the feature vector drives the network to learn complex data patterns, while the original spectral data is physically verified in the loss function, ensuring that the network output conforms to the basic physical laws. Even if the network only processes the feature vector internally, its output is still forced to be consistent with all the original spectral observation data under the physical law.

[0117] This design realizes the deep fusion of multi-spectral information and the constraint of physical consistency: the model does not independently predict an MPOD value for each wavelength, but is trained to find an optimal, single MPOD value that can best explain the reflectance intensity data measured at all observed wavelengths, i.e., the MPOD prediction value is compatible with all multi-spectral observation data under the physical law. This mechanism is the key to the high precision, strong robustness, and excellent generalization ability of the PINN model of the present application.

[0118] The total loss function is: L_ total = L_ data +λ_ physics × L_ physics ; where λ_ physics is a hyperparameter used to balance the weights of the two loss terms.

[0119] By minimizing this total loss function, the model is forced to follow the physical law while learning data-driven features, thereby significantly enhancing the accuracy, robustness, interpretability, and generalization ability (device independence) of MPOD quantification, which is manifested in:

[0120] Enhanced robustness: even with limited training data or in the presence of noise, the physical constraint can guide the model to learn a more reasonable solution, reducing overfitting;

[0121] Improved generalization ability: the model can better generalize to unseen imaging devices or different individuals, as the relationship it learns is based on universal physical principles rather than just the statistical properties of the training data;

[0122] Interpretability: Because the model incorporates physical knowledge, its decision-making process is more interpretable, which helps to understand the intrinsic relationship between MPOD and spectral data;

[0123] Device independence: By incorporating device-dependent optical characteristics (such as light source spectrum and detector response) into the physical model, PINN can effectively eliminate or compensate for differences between different imaging devices, thereby achieving MPOD quantization that is closer to "device independence".

[0124] Step 5: MPOD Spatial Distribution Map Generation and Visualization

[0125] Based on the macular pigment optical density values, a spatial distribution map of MPOD across the entire macular region is generated. This distribution map is presented in an intuitive visualization, clearly showing the two-dimensional density distribution and gradient changes of MPOD on the retina. The visualization format can be as follows:

[0126] Pseudocolor mapping: This maps different MPOD values ​​to different predefined color spectra. For example, high MPOD values ​​are represented by warm colors (such as red and yellow), and low MPOD values ​​are represented by cool colors (such as blue and green). This makes the distribution pattern of MPOD immediately apparent.

[0127] Heatmap: Similar to pseudocolor map, but with a greater emphasis on high-density areas, usually using color gradients to highlight density changes.

[0128] 3D surface map: Using MPOD values ​​as the Z-axis height, a 3D topographic map is constructed on the 2D fundus image to visually display the peak and valley values ​​of MPOD.

[0129] The visualization is interactive, allowing users to:

[0130] To query the MPOD value at a specific location on the MPOD spatial distribution map: Click on any point on the map with the mouse to display the precise MPOD value for that point;

[0131] Generate MPOD value cross-sectional distribution curves; plot MPOD value profiles along user-defined straight lines or curves to analyze the radial or specific directional distribution variations of MPOD, etc.

[0132] Overlay other images: Overlay the MPOD spatial distribution map with the original multispectral fundus image, angiography or retinal tomography image (such as OCT image) to enable comprehensive analysis of multimodal images.

[0133] Step Six: Vertical Tracking and Correlation Analysis

[0134] Store and manage macular pigment optical density measurement data of the same patient at different time points, generate macular pigment optical density trend over time, and analyze the correlation with the patient's clinical information to assess disease progression or intervention effect.

[0135] In order to realize long-term monitoring of the disease in the present application:

[0136] Data storage and management: Establish a patient database to store and manage MPOD data of each measurement (including original images, segmentation masks, MPOD spatial distribution maps and quantitative values) and corresponding clinical information (such as patient ID, age, diagnosis, medication history, lifestyle, etc.);

[0137] Trend analysis: automatically analyze the MPOD data of the same patient over time to generate a trend graph of MPOD over time (for example, the central foveal peak MPOD or the average MPOD value of the macular area can be plotted to generate a curve, etc.);

[0138] Correlation analysis: correlate the change trend of MPOD with the clinical progress of the patient (such as AMD staging) or intervention measures (such as lutein supplementation), evaluate the effectiveness of MPOD as a biomarker, and generate personalized reports to assist clinical decision-making.

[0139] Embodiment two: macular pigment density analysis system

[0140] As shown in Figure 2 The present application also provides an analysis system for implementing the above analysis method, which comprises the following modules:

[0141] A multispectral fundus image acquisition module for acquiring multispectral fundus images of the human eye retina; the module includes multispectral fundus camera hardware and corresponding image acquisition control software.

[0142] An image preprocessing module for performing registration processing on the acquired multispectral fundus images, and performing registration, denoising, illumination correction, etc. on the images through preprocessing to provide high-quality input data for subsequent analysis;

[0143] A deep learning segmentation module with a pre-trained deep learning segmentation model for receiving preprocessed multispectral images and outputting accurate segmentation masks of the macular region and the foveal region;

[0144] A multispectral feature extraction module for extracting multispectral features from the multispectral images according to the segmentation masks;

[0145] A physical information constrained deep regression quantification module with a physical information constrained deep regression model, which is the core computing unit of the system, for quantifying macular pigment optical density values according to the extracted multispectral features;

[0146] The MPOD spatial distribution map generation and visualization module is configured to generate a spatial distribution map of MPOD according to the quantified MPOD values and provide an interactive visualization interface, wherein the MPOD spatial distribution map is displayed in the form of a pseudo-color map, a heat map or a three-dimensional surface map, and the user is allowed to query the MPOD value at a specific position in the MPOD spatial distribution map and / or generate a cross-sectional distribution curve of the MPOD value;

[0147] The data management and longitudinal tracking module is configured to store, manage and retrieve patient data, measurement results and clinical information, and provide longitudinal tracking and trend analysis functions.

[0148] The user interface module provides a unified graphical user interface (GUI) that integrates the functions of various modules, facilitating user operation, result viewing and report generation.

[0149] These modules can be integrated on a single high-performance computing device (such as a workstation), or can adopt a distributed architecture, with some modules deployed on a cloud server and some deployed on a local device, to flexibly adapt to different application scenarios and computing requirements, achieving efficient data processing and storage.

[0150] Embodiment three: computer-readable storage medium

[0151] The present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method for analyzing macular pigment density based on multi-spectral imaging and deep learning is realized.

[0152] The computer-readable storage medium can be, but is not limited to, a non-transitory storage medium such as a hard disk, a solid state disk, an optical disk, a USB flash drive, RAM, ROM, etc. The processor can be a general-purpose processor, a special-purpose processor, a microcontroller, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit), etc.

[0153] The present application realizes high measurement accuracy, strong robustness and generalization ability, accurate and adaptive regional segmentation, objective and automatic process, detailed spatial distribution information and strong clinical transformation value through the innovative combination of multi-spectral imaging and deep learning of physical information, effectively overcoming the many bottlenecks of existing subjective methods and traditional objective imaging methods.

[0154] The above is only a specific embodiment of the present application, but the technical features of the present application are not limited thereto. Any simple change, equivalent replacement or modification made on the basis of the present application to solve the same technical problem and achieve the same technical effect is also covered by the protection scope of the present application.

Claims

1. A method for analyzing macular pigment density based on multispectral imaging and deep learning, characterized in that, The steps include the following: Step 1: Acquire multispectral fundus images of the human retina at narrow wavelengths and perform registration processing on the multispectral fundus images; Step 2: Input the registered multispectral fundus image into the pre-trained deep learning segmentation model to segment the macular region and the foveal region; Step 3: Extract multispectral features from the segmented macular and foveal regions; Step 4: Input the multispectral features into the physically constrained deep regression model, and the deep regression model outputs the macular pigment optical density value of each pixel or predefined sub-region within the macular region; The deep regression model for physical information constraints is a physical information constraint neural network. During the training process, the total loss function of the physical information constraint neural network includes a data loss term and a physical loss term. The data loss term is used to measure the difference between the model-predicted macular pigment optical density value and the true value; The physical loss term is constructed based on the Beer-Lambert law and is used to measure the difference between the theoretical reflected light intensity at multiple wavelengths calculated based on the macular pigment optical density value predicted by the model and the reflected light intensity actually measured at the corresponding multiple wavelengths. Step 5: Based on the macular pigment optical density values, generate a spatial distribution map of MPOD in the macular region.

2. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: In step one, the multispectral fundus images are acquired by a multispectral fundus camera.

3. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: In step one, the narrowband wavelength includes the macular pigment absorption band and at least one reference band.

4. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 3, characterized in that: The macular pigment absorption band includes at least one wavelength in the range of 400nm to 460nm, and the reference band includes at least one wavelength in the range of 500nm to 540nm and / or 570nm to 660nm.

5. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: In step three, the multispectral features include at least one of the following: Average pixel intensity: The average pixel intensity of each wavelength channel in the macula and fovea regions; Pixel intensity histogram: The pixel intensity histogram for each wavelength channel in the macula and fovea regions; Spectral ratio: The ratio of pixel intensity between channels of different wavelengths; Texture features: Texture features in the macular region and the central concave region.

6. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: The method also includes longitudinal tracking and correlation analysis steps: storing and managing macular pigment optical density measurement data of the same patient at different time points, generating a trend graph of macular pigment optical density changes over time, and performing correlation analysis in conjunction with the patient's clinical information to assess disease progression or intervention effects.

7. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: In step two, the deep learning segmentation model is a fully convolutional neural network with U-Net architecture, DeepLabV3+ architecture, Attention U-Net architecture, or ResU-Net architecture.

8. The method for macular pigment density analysis based on multispectral imaging and deep learning according to claim 1, characterized in that: In step five, the MPOD spatial distribution map is visualized using a pseudo-color map, heat map, or three-dimensional surface map.

9. The method for analyzing macular pigment density based on multispectral imaging and deep learning according to claim 8, characterized in that: The visualization is interactive, allowing users to query MPOD values ​​at specific locations in the MPOD spatial distribution map and / or generate cross-sectional distribution curves of MPOD values.

10. An analytical system for implementing the macular pigment density analysis method based on multispectral imaging and deep learning as described in any one of claims 1 to 9, characterized in that, include: Multispectral fundus image acquisition module, used to acquire multispectral fundus images of the human retina; The image preprocessing module is used to perform registration processing on the acquired multispectral fundus images; The deep learning segmentation module has a built-in pre-trained deep learning segmentation model, which is used to receive pre-processed multispectral images and output segmentation masks for the macular region and the fovea region. A multispectral feature extraction module is used to extract multispectral features from a multispectral image based on a segmentation mask. The physical information-constrained deep regression quantization module has a built-in physical information-constrained deep regression model, which is used to quantify the optical density value of macular pigment based on the extracted multispectral features. The MPOD spatial distribution map generation and visualization module is used to generate spatial distribution maps based on the quantified macular pigment optical density values ​​and provide an interactive visualization interface.

11. The analysis system for the macular pigment density analysis method based on multispectral imaging and deep learning according to claim 10, characterized in that: The MPOD spatial distribution map is displayed in the form of a pseudo-color map, heat map, or three-dimensional surface map, and supports users to query the MPOD value at a specific location in the MPOD spatial distribution map, and / or generate MPOD value cross-sectional distribution curves.

12. The analysis system for the macular pigment density analysis method based on multispectral imaging and deep learning according to claim 10, characterized in that: It also includes a data management and longitudinal tracking module for storing patient data, measurement results and clinical information, and providing longitudinal tracking and trend analysis functions.

13. A computer-readable storage medium storing a computer program thereon, characterized in that: When the computer program is executed by the processor, it implements the macular pigment density analysis method based on multispectral imaging and deep learning as described in any one of claims 1 to 9.

Images