A method for simultaneously detecting bilirubin and hemoglobin concentrations based on eye single-frame image
By combining a fixed light source and a standard color chart, and using YOLOv8 and a twin Swing-Transformer network for eye image processing, high-precision simultaneous detection of bilirubin and hemoglobin concentrations was achieved. This solves the problems of unstable color correction and insufficient model robustness in existing technologies, and provides a reliable non-invasive detection method.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2026-04-02
- Publication Date
- 2026-07-10
AI Technical Summary
Current technologies cannot achieve simultaneous non-invasive detection of bilirubin and hemoglobin concentrations, and traditional methods suffer from unstable color correction and insufficient model robustness.
Color correction was performed using a fixed light source and a standard color chart. The sclera and lower eyelid regions were extracted using the YOLOv8 segmentation network, and detection was performed using a twin-transformer network with shared weights and left-right eye feature consistency loss.
It achieves high-precision simultaneous non-invasive detection of bilirubin and hemoglobin concentrations, improving the accuracy and robustness of the detection, and providing a reliable non-invasive detection method for jaundice screening and anemia monitoring.
Smart Images
Figure CN122367902A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing and intelligent diagnostic technology, and more specifically, to a method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single frame image of the eye. Background Technology
[0002] Bilirubin and hemoglobin concentrations are important biochemical indicators for the clinical diagnosis of jaundice, anemia, hemolytic diseases, and abnormal liver function. Traditional detection methods mainly rely on invasive venous blood collection or percutaneous bilirubinometers. The former has problems such as pain, infection risk, and long processing time, while the latter is easily affected by skin thickness, melanin content, and ambient light, resulting in limited accuracy and repeatability.
[0003] In recent years, non-invasive detection technologies based on eye images have gradually attracted attention. Some researchers have attempted to use mobile phones or ordinary cameras to photograph the sclera or lower eyelid area and estimate bilirubin or hemoglobin through color analysis or simple machine learning models. However, these methods usually lack systematic color correction techniques, resulting in insufficient accuracy and repeatability of color information, making it difficult to provide a reliable basis for subsequent quantitative analysis.
[0004] Some researchers have used only single-modal information from the sclera or lower eyelid for single-parameter prediction, neglecting the importance of scleral yellowing as a standard feature of bilirubin and the value of lower eyelid congestion in hemoglobin assessment. At the same time, they have not made full use of the prior knowledge of the physiological symmetry of the left and right eyes, resulting in insufficient model robustness and the inability to detect hemoglobin and bilirubin simultaneously.
[0005] Therefore, how to design a non-invasive detection method that can simultaneously detect bilirubin and hemoglobin concentrations, achieve stable color correction through a fixed light source combined with a standard color chart, and fully utilize the multimodal information of the sclera and lower eyelid as well as the a priori symmetry between the left and right eyes has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] In view of this, the present invention provides a method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single frame image of the eye. The method acquires the original image of the subject's eye containing a standard color chart under a fixed light source, integrates the standard color chart for precise color correction, extracts the dual-modal regions of the sclera and lower eyelid using an image segmentation network, and introduces left and right eye feature consistency constraints based on a twin neural network architecture, thereby achieving high-precision simultaneous non-invasive detection of bilirubin and hemoglobin concentrations.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single-frame image of the eye, characterized by comprising the following steps: S1. Under a fixed light source, acquire the original image containing the subject's eye area and the standard color chart, and at the same time, obtain the measured hemoglobin concentration value through blood detection and the measured bilirubin concentration value through a transcutaneous bilirubin meter. S2. Perform manual white balance correction on the original image based on the neutral gray blocks of the standard color chart to obtain a white balance image; S3. Construct a least-squares optimization objective function based on the standard color chart, solve the color correction matrix CCM, multiply the white balance image by the CCM and convert it to the CIELAB color space to obtain a color-normalized image; S4. Input the color-normalized image into the YOLOv8 segmentation network to extract the sclera region and the lower eyelid region respectively; S5. The scleral region and lower eyelid region of the left and right eyes are cropped and paired and input into a twin Swin-Transformer feature extractor with shared weights. The feature extractor includes a Patch Embedding layer and multiple Stages of Swin-Transformer Blocks. S6. The feature maps output by the left and right eyes are respectively fed into the Neck module for multi-scale feature fusion, and then the preliminary prediction values are obtained through generalized mean pooling and regression head. S7. During the training phase, introduce left-right eye feature consistency loss, and output the final bilirubin concentration and hemoglobin concentration after averaging the predicted values of the left and right eyes.
[0008] Furthermore, S2 includes: S21. Identify the neutral gray area of the standard color chart in the original image; S22. Calculate the average value of the neutral gray patch region in the R, G, and B channels; S23. Calculate the gain coefficients of the R, G, and B channels respectively, and perform independent multiplication adjustment on the entire image to obtain the white balance image.
[0009] Furthermore, S3 includes: S31. Use the standard CIELAB values of each color patch provided by the manufacturer under D65 standard light source, which are included with the standard color card. S32. Extract the white balance RGB values of the corresponding color blocks from the actual acquired image; S33. Construct an overdetermined system of equations using all color blocks, and solve the 3×3 color correction matrix CCM using the least squares method. S34. Multiply each pixel of the white balance image with the CCM and convert it to the CIELAB color space to obtain a color-normalized image.
[0010] Furthermore, S4 includes: S41. Input the color-normalized image into the YOLOv8 segmentation network; S42. The YOLOv8 segmentation network uses weights pre-trained on the publicly available Periorbital dataset and constructs a self-made dataset to achieve further fine-tuning. S43. Output segmentation images of the sclera and lower eyelid simultaneously using the instance segmentation head; S44. After performing morphological processing and connected component analysis on the segmentation map, extract the final region of interest.
[0011] Furthermore, S5 includes: S51. Perform uniform size cutting and normalization on the sclera and lower eyelid areas of the left and right eyes respectively; S52. Pair the cropped left and right eye images and input them into two Swin-Transformer branches with identical structures and shared weights. S53, each branch includes a Patch Embedding layer, a Swin-Transformer Block for multiple stages, and a Shifted-Window multi-head self-attention mechanism.
[0012] Furthermore, S6 includes: S61. Input the feature maps output from multiple Stages of the Swin-Transformer into the Neck module; S62. The Neck module achieves multi-scale feature fusion through top-down and bottom-up paths; S63. Perform generalized mean pooling on the fused left and right eye feature maps to obtain fixed-dimensional feature vectors. S64. Input the feature vectors into the linear regression head to predict the concentrations of bilirubin and hemoglobin, respectively.
[0013] Furthermore, S7 includes: S71. During the training phase, calculate the cosine similarity of the left and right eye Neck output feature vectors and construct the feature consistency loss. in, and These are the left and right eye feature vectors of the i-th sample extracted from the network backbone, respectively. Represents the vector dot product. Describes the L2 norm of a vector; S72. Optimize the total loss function by weighted summation of left and right eye regression loss and feature consistency loss; and These are the network's predictions for the left and right eyes of the i-th sample. and Here, N is the corresponding real label, N is the batch size, and SmoothL1Loss is defined as: ; and The regression losses for the left and right eyes are respectively. It is feature consistency loss. It is a hyperparameter used to balance the weights between the regression task and the consistency constraint; S73. In the inference stage, the average of the prediction results of the left and right eyes is taken as the final output.
[0014] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects: 1. For the first time, it achieves simultaneous non-invasive detection of bilirubin and hemoglobin concentrations based on eye images, overcoming the limitation of existing technologies that can only detect single parameters.
[0015] 2. By using fixed light source reflection illumination and combining it with a standard color chart to perform manual white balance and least squares color correction matrix (CCM) correction in sequence, highly stable color constancy was achieved, ensuring accurate color information was obtained under fixed acquisition conditions, and providing high-quality standardized input for subsequent deep learning models.
[0016] 3. The YOLOv8 instance segmentation network is used to extract the dual-modal regions of the sclera and lower eyelid simultaneously, making full use of the scleral yellowing and lower eyelid congestion features, which is significantly better than the existing technology that only uses a single modality.
[0017] 4. We designed a twin-Swin-Transformer network with shared weights and introduced left-right eye feature consistency loss. By utilizing the physiological symmetry prior of the human body, we effectively suppressed individual differences and random noise, further improving the robustness of detection.
[0018] 5. This invention provides an integrated portable hardware implementation solution that can be applied to jaundice screening, anemia monitoring, and community health management, and has significant clinical application value. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0020] Figure 1 A flowchart of a method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single frame image of the eye, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of acquiring eye video data of a subject, as provided in an embodiment of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] like Figure 1 As shown, this embodiment provides a method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single-frame image of the eye, including the following steps: S1. Under a fixed light source, acquire the original image containing the subject's eye area and the standard color chart, and at the same time, obtain the measured hemoglobin concentration value through blood biochemistry detection and the measured bilirubin concentration value through a percutaneous bilirubin meter. S2. Perform manual white balance correction on the original image based on the neutral gray blocks of the standard color chart to obtain a white balance image; S3. Construct a least-squares optimization objective function based on the standard color chart, solve the color correction matrix CCM, multiply the white balance image by the CCM and convert it to the CIELAB color space to obtain a color-normalized image; S4. Input the color-normalized image into the YOLOv8 segmentation network to extract the sclera region and the lower eyelid region respectively; S5. Crops the scleral region and lower eyelid region of the left and right eyes respectively and inputs them into a twin-Swin-Transformer feature extractor with shared weights. S6. The feature maps output by the left and right eyes are respectively fed into the Neck module for multi-scale feature fusion, and then the preliminary prediction values are obtained through generalized mean pooling and regression head. S7. During the training phase, introduce left-right eye feature consistency loss, and output the final bilirubin concentration and hemoglobin concentration after averaging the predicted values of the left and right eyes.
[0023] This method not only achieves highly stable color constancy by using a fixed light source and standard color chart, but also makes full use of the dual-modal characteristics of scleral icterus and lower eyelid congestion, as well as the physiological symmetry prior of the left and right eyes, which greatly improves the accuracy and robustness of bilirubin and hemoglobin concentration prediction, providing a reliable non-invasive detection method for scenarios such as jaundice screening and anemia monitoring.
[0024] The following provides a further detailed explanation of each step in the above technical solution: In this embodiment S1, a dedicated acquisition device is used to capture the original image containing the subject's eye area and a standard color chart under a fixed light source; like Figure 2 As shown, in the specific process of acquiring eye images, a fixed light source 3 first illuminates the subject's eye region 1, and the light is uniformly diffused onto the eye region through the reflective hemisphere 4 inside the integrating sphere. The camera 5 is located on the central axis of the hemisphere, enabling it to capture the eye image uniformly illuminated by the reflective hemisphere 4, with the camera's optical axis aligned with the center of the pupil. This setup ensures that the fixed light source illumination and standard color chart are acquired synchronously without causing glare, providing high-quality, standardized, and consistent original images for subsequent color correction and region segmentation.
[0025] In this embodiment S2, manual white balance correction is performed on the original image based on the neutral gray blocks of the standard color chart, including: S21. Identify the neutral gray area of the standard color chart in the original image; S22. Calculate the average value of the neutral gray patch region in the R, G, and B channels; S23. Calculate the gain coefficients of the R, G, and B channels respectively, and perform independent multiplication adjustment on the entire image to obtain the white balance image.
[0026] In this embodiment S3, a least-squares optimization objective function is constructed based on a standard color chart, and the color correction matrix CCM is solved, including: S31. Use the standard CIELAB values of each color patch provided by the manufacturer under D65 standard light source, which are included with the standard color card. S32. Extract the white balance RGB values of the corresponding color blocks from the actual acquired image; S33. Construct an overdetermined system of equations using all color blocks, and solve the 3×3 color correction matrix CCM using the least squares method. S34. Multiply each pixel of the white balance image with the CCM and convert it to the CIELAB color space to obtain a color-normalized image.
[0027] In this embodiment S4, the color-normalized image is input into the YOLOv8 segmentation network to extract the scleral region and the lower eyelid region, including: S41. Input the color-normalized image into the YOLOv8 segmentation network; S42. The YOLOv8 segmentation network is initialized using weights pre-trained on the publicly available Periorbital dataset; S43. Output segmentation images of the sclera and lower eyelid simultaneously using the instance segmentation head; S44. After performing morphological processing such as dilation and erosion on the segmentation map and filtering for the largest connected component, the final scleral region and lower eyelid region are extracted.
[0028] In this embodiment S5, the scleral regions and lower eyelid regions of the left and right eyes are cropped and paired, and then input into a twin-Swin-Transformer feature extractor with shared weights, including: S51. Perform uniform size cropping and normalization on the left and right eye areas respectively; S52. Pair the cropped left and right eye images and input them into two Swin-Transformer branches with identical structures and shared weights. S53, each branch includes a Patch Embedding layer, a multi-stage Swin-Transformer Block, a Shifted-Window multi-head self-attention mechanism, and a Feed-Forward Network.
[0029] In this embodiment S6, the feature maps output by the left and right eyes are respectively fed into the Neck module for multi-scale feature fusion. After generalized mean pooling and regression head, preliminary predicted values are obtained, including: S61. Input the feature maps output from multiple Stages of the Swin-Transformer into the Neck module; S62. The Neck module achieves multi-scale feature fusion through top-down and bottom-up paths; S63. Perform generalized mean pooling on the fused left and right eye feature maps to obtain fixed-dimensional feature vectors. S64. Input the feature vectors into a three-layer fully connected regression head to predict bilirubin and hemoglobin concentrations respectively.
[0030] In this embodiment S7, a left-right eye feature consistency loss is introduced during the training phase. The final result is output after averaging the prediction values of the left and right eyes, including: S71. During the training phase, calculate the cosine similarity of the left and right eye Neck output feature vectors and construct the feature consistency loss. in, and These are the left and right eye feature vectors of the i-th sample extracted from the network backbone, respectively. Represents the vector dot product. Describes the L2 norm of a vector; S72. Optimize the total loss function by weighted summation of left and right eye regression loss and feature consistency loss; and These are the network's predictions for the left and right eyes of the i-th sample. and Here, N is the corresponding real label, N is the batch size, and SmoothL1Loss is defined as: ; and The regression losses for the left and right eyes are respectively. It is feature consistency loss. It is a hyperparameter used to balance the weights between the regression task and the consistency constraint; S73. In the inference stage, the average of the prediction results of the left and right eyes is taken as the final output.
[0031] This embodiment provides a method for simultaneous detection of bilirubin and hemoglobin concentrations based on single-frame images of the eye. It acquires an eye image containing a standard color chart under a fixed light source. The standard color chart is used to perform manual white balance and least-squares color correction matrix (CCM) correction to obtain a color-standardized image. A YOLOv8 instance segmentation network is used to simultaneously extract the sclera and lower eyelid regions. A shared-weights twin-transformer is used to extract features, and a left-right eye feature consistency loss is introduced. Finally, high-precision, simultaneous, and non-invasive detection of bilirubin and hemoglobin concentrations is achieved. This method effectively compensates for color deviations under different imaging devices and lighting conditions through a dual correction process using the standard color chart. It obtains accurate and repeatable color information under fixed acquisition conditions while retaining the technical potential for expansion to other light sources and camera scenarios. It fully utilizes the dual-modal features of scleral icterus and lower eyelid congestion, as well as the physiological symmetry prior of the left and right eyes, significantly improving the accuracy and robustness of the detection. This provides a reliable clinical auxiliary means for jaundice screening, anemia monitoring, and community health management.
[0032] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0033] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for simultaneous detection of bilirubin and hemoglobin concentrations based on a single-frame image of the eye, characterized in that, Includes the following steps: S1. Under a fixed light source, acquire the original image containing the subject's eye area and a standard color chart, obtain the measured hemoglobin concentration value through blood testing, and obtain the measured bilirubin concentration value through a transcutaneous bilirubin analyzer. S2. Perform manual white balance correction on the original image based on the neutral gray blocks of the standard color chart to obtain a white balance image; S3. Construct a least-squares optimization objective function based on the standard color chart, solve the color correction matrix CCM, multiply the white balance image with the CCM, and then convert it to the CIELAB color space to obtain a color-accurate standardized image; S4. Input the standardized image into the YOLOv8 segmentation network to extract the sclera region and the lower eyelid region respectively; S5. The scleral region and lower eyelid region of the left and right eyes are cropped and paired and input into a twin Swin-Transformer feature extractor with shared weights. The feature extractor includes a Patch Embedding layer, a Linear Embedding layer and Stage1, Stage2, Stage3 and Stage4 connected in sequence, wherein each Stage contains multiple Swin-Transformer Blocks. S6. The left and right eye feature maps output by the Twin-Transformer are respectively sent to the Neck module for multi-scale feature fusion. After generalized mean pooling and linear regression head, the preliminary predicted values of bilirubin and hemoglobin for the left and right eyes are obtained. S7. During the training phase, introduce left-right eye feature consistency loss, and output the final bilirubin concentration and hemoglobin concentration after averaging the prediction values of the left and right eyes.
2. The method according to claim 1, characterized in that, S2 includes: S21. Identify the neutral gray area in the standard color chart; S22. Calculate the average value of the neutral gray patch region in the R, G, and B channels; S23. Calculate the gain coefficients of the R, G, and B channels respectively according to the Von Kries diagonal model, and perform channel-independent multiplication adjustment on the entire original image to obtain the white balance image.
3. The method according to claim 1 or 2, characterized in that, S3 includes: S31. Use the standard CIELAB values of each color patch provided by the manufacturer under a D65 standard light source, which are included with the standard color chart; S32. Extract the RGB values of the corresponding color blocks after white balance from the actual acquired image; S33. Construct an overdetermined system of equations using all color patches as samples, and solve for the optimal 3×3 color correction matrix CCM using the least squares method; S34. Multiply each pixel of the white balance image with the CCM and convert it to the CIELAB color space to obtain a color-normalized image.
4. The method according to any one of claims 1-3, characterized in that, S4 includes: S41. Input the standardized CIELAB image into the YOLOv8 segmentation network; S42. The YOLOv8 segmentation network uses weights pre-trained on the publicly available Periorbital dataset and fine-tuned on a self-made dataset; S43. Output scleral segmentation map and lower eyelid segmentation map simultaneously through the instance segmentation head; S44. After performing connected component analysis and morphological processing on the segmentation graph, extract the final region of interest.
5. The method according to any one of claims 1-4, characterized in that, S5 includes: S51. The sclera and lower eyelid regions of the left and right eyes are cropped and normalized to a uniform size; S52. Pair the cropped left and right eye images and input them into two Swing-Transformer branches with identical structures and shared weights; S53. The Swin-Transformer includes Patch Embedding, Window-based Multi-head Self-Attention, Shifted Window Multi-head Self-Attention and Feed-ForwardNetwork.
6. The method according to any one of claims 1-5, characterized in that, S6 includes: S61. Input the four-scale feature maps output from Stage1 to Stage4 into the Neck module; S62. The Neck module achieves multi-scale feature fusion through a bidirectional path of top-down and bottom-up; S63. Perform generalized mean pooling on the fused left and right eye feature maps respectively to obtain fixed-dimensional feature vectors; S64. Input the feature vector into the linear regression head and output the preliminary predicted values of bilirubin and hemoglobin respectively.
7. The method according to any one of claims 1-6, characterized in that, S7 includes: S71. During the training phase, calculate the cosine similarity of the left and right eye Neck output feature vectors and construct the feature consistency loss. in, and These are the left and right eye feature vectors of the i-th sample extracted from the network backbone, respectively. Represents the vector dot product. Describes the L2 norm of a vector; S72. Optimize the total loss function by weighted summation of left and right eye regression loss and feature consistency loss; and These are the network's predictions for the left and right eyes of the i-th sample. and Here, N is the corresponding real label, N is the batch size, and SmoothL1Loss is defined as: ; and The regression losses for the left and right eyes are respectively. It is feature consistency loss. It is a hyperparameter used to balance the weights between the regression task and the consistency constraint; S73. During the inference phase, the average of the prediction results for the left and right eyes is taken as the final output.