Semi-supervised non-contact neonatal jaundice home intelligent early warning method based on image calibration

Abstract

The invention discloses a semi-supervised non-contact neonatal jaundice home intelligent early warning method based on image calibration, and the method comprises the steps: building a camera response function database, and building a color constancy depth model and a skin region segmentation network; designing a strong and weak enhancement strategy by using a large amount of label-free home data, constructing a time and context comparison learning module, and performing self-supervised training on a feature encoder; a small amount of labeled hospital data and a large amount of unlabeled family data are combined, and semi-supervised fine tuning is realized through pseudo label generation and supervised comparative learning; image sequences continuously uploaded by a user are input into the model, multi-day prediction of the bilirubin level is achieved, prediction uncertainty is quantified in combination with a Bayesian method, the risk boundary is dynamically adjusted, and personalized early warning is provided. According to the method, a user does not need to use a physical colorimetric card, color measurement errors caused by model differences of mobile equipment and variability of household illumination are inhibited from a data source, and the accuracy and robustness of subsequent jaundice assessment are improved; through a training framework combining self-supervised pre-training and semi-supervised fine tuning, the understanding ability and generalization performance of the model for the sequential characteristics of jaundice are improved.

Description

Technical Field

[0001] This invention relates to the field of medical and health monitoring technology, specifically to a non-contact method for dynamic monitoring and early warning of neonatal jaundice based on artificial intelligence, which is particularly suitable for applications in a home environment. Background Technology

[0002] Neonatal hyperbilirubinemia, clinically manifesting as neonatal jaundice, is one of the most common clinical problems in the neonatal period. Statistics show that approximately 50%-60% of full-term infants and 80% of premature infants will exhibit jaundice to varying degrees. Most cases of neonatal jaundice are physiological and resolve spontaneously; however, if bilirubin levels are abnormally high, it can develop into acute bilirubin encephalopathy, leading to irreversible neurological damage. Therefore, accurate and continuous monitoring of neonatal jaundice is a crucial aspect of neonatal health management.

[0003] Currently, clinical monitoring methods for neonatal jaundice mainly include total serum bilirubin (TSB) testing and transcutaneous bilirubin (TcB) measurement. TSB testing is the gold standard for diagnosis, but it is an invasive procedure and not suitable for frequent monitoring in home settings. While TcB measurement is non-invasive and convenient, the equipment is expensive, making it difficult to popularize in home settings, resulting in a lack of effective continuous monitoring methods for newborns after discharge. Typically, newborns are discharged around 72 hours after birth, while the peak of jaundice often occurs on the 4th to 5th day after birth. This temporal discrepancy makes static assessments at discharge difficult to predict subsequent disease progression, leading to the risk of missed diagnosis and delayed treatment for some high-risk infants.

[0004] In recent years, technologies have emerged that utilize smartphones to capture images of newborn skin and combine them with artificial intelligence algorithms for non-invasive jaundice assessment. Existing solutions mostly employ a "standard colorimetric card + application (App)" model, requiring users to place the colorimetric card next to the newborn's skin for simultaneous image capture, aiming to calibrate the image color using the colorimetric card. However, this method faces several technical bottlenecks in practical applications: the operation process is relatively cumbersome, and the calibration effect is highly susceptible to interference from various factors such as ambient lighting conditions, shooting angle, differences in mobile phone models, and improper user operation (e.g., camera beauty mode not being turned off), which may lead to systematic errors in color measurement, affecting the accuracy and repeatability of the assessment results.

[0005] Furthermore, existing methods also have limitations at the algorithmic level. First, color deviations caused by different camera equipment and lighting environments directly affect the accuracy of quantifying the degree of jaundice in skin color, resulting in insufficient generalization ability of the model in complex home environments. Second, existing assessments are mostly static, single-point detections, only reflecting the bilirubin level at the current moment, lacking the ability to predict the dynamic development trend of jaundice, and unable to quantify the uncertainty of the model's prediction results, making risk stratification insufficiently accurate and failing to meet the clinical need for early dynamic warning.

[0006] In summary, there is an urgent need for a new intelligent method for monitoring neonatal jaundice that can break free from dependence on physical color cards, operate stably across multiple devices and environments, effectively integrate labeled data from within hospitals with unlabeled data from outside hospitals, and provide dynamic trend prediction and quantitative risk assessment. Summary of the Invention

[0007] This invention aims to overcome the aforementioned shortcomings of existing technologies by providing a semi-supervised, non-contact, home-based intelligent early warning method for neonatal jaundice based on image calibration. This addresses the problems in existing technologies, such as inaccurate color measurements due to equipment differences and varying lighting conditions, as well as insufficient model generalization ability and the inability to perform dynamic trend prediction and reliability assessment due to sparse clinical annotation data.

[0008] The technical solution adopted by this invention to solve its technical problem is:

[0009] A semi-supervised, non-contact, home-based intelligent early warning method for neonatal jaundice based on image calibration includes the following steps:

[0010] (1) A camera response function database is established, and a color constancy depth model and a skin region segmentation network are constructed. The color constancy depth model includes an input layer, convolutional units 1-4, a global adaptive average pooling layer, a fully connected regression head, and an output layer. The convolutional units are used to extract multi-scale color and texture features from the input image; the global adaptive average pooling layer is used to compress spatial features into a fixed-dimensional vector; the fully connected regression head is used to output the normalized RGB color vector of the scene light source; and the output layer is used to output the light source color vector. The skin region segmentation network adopts a U-Net structure, which includes an encoder downsampling section, a bottleneck layer, a decoder upsampling section, and an output layer. The encoder downsampling section is used to extract features step-by-step and reduce resolution; the bottleneck layer is used to fuse contextual information; the decoder upsampling section is used to gradually restore resolution and fuse encoded features; and the output layer is used to generate a binary skin mask.

[0011] (2) Utilizing a large amount of unlabeled home data, a strong and weak enhancement strategy was designed, and a time and context contrast learning module was constructed to perform self-supervised training on the feature encoder. The time and context contrast learning module sequentially includes a position encoder, a time encoder, a context contrast projection head, a time prediction head, and a contrast loss calculation unit. The time encoder is used to extract temporal features; the context contrast projection head is used to map global features and calculate contrast loss; the time prediction head is used to predict future features based on historical features; and the contrast loss calculation unit is used to calculate and weight the joint loss. The feature encoder sequentially includes an input layer, an initial convolutional block, a residual or convolutional downsampling block, an adaptive average pooling layer, and a feature projection head. The initial convolutional block is used to extract preliminary features of the input image; the residual or convolutional downsampling block is used to extract deep features and reduce dimensionality; the adaptive average pooling layer is used to convert the feature map into a fixed vector; and the feature projection head is used to map and normalize the feature vector.

[0012] (3) By combining a small amount of labeled inpatient data with a large amount of unlabeled family data, semi-supervised fine-tuning is achieved through pseudo-label generation and supervised comparative learning;

[0013] (4) Input the image sequences continuously uploaded by users into the model to achieve multi-day prediction of bilirubin levels, and combine the Bayesian method to quantify the prediction uncertainty, dynamically adjust the risk boundary, and provide personalized early warning.

[0014] In one implementation, in step (1), the structure of the color constancy depth model is as follows: starting from the input layer, convolutional unit 1, convolutional unit 2, convolutional unit 3, convolutional unit 4, a global adaptive average pooling layer, a fully connected regression head, and an output layer are connected sequentially. The skin region segmentation network adopts a U-Net structure, the overall structure of which is as follows: starting from the input layer, it sequentially includes an encoder downsampling section, a bottleneck layer, and a decoder upsampling section, and the encoder features and decoder are fused step by step through skip connections, and finally the output layer generates a binary skin mask of the same size as the input image.

[0015] In one implementation, the structure of the time and context contrast learning module in step (2) is as follows: after inputting the feature sequence, it is sequentially connected to a position encoder, a time encoder, a context contrast projection head, and a time prediction head. The structure of the feature encoder is as follows: starting from the input layer, it is sequentially connected to an initial convolutional block, multiple residual or convolutional downsampling blocks, an adaptive average pooling layer, and a feature projection head.

[0016] Furthermore, in step (1), the image color normalization preprocessing steps are as follows:

[0017] (1.1) Pixel value linearization based on camera response function (CRF): The camera CRF is calibrated in a controlled environment and stored in the database. During online acquisition, the CRF is called according to the device model, and the nonlinear RGB pixel values ​​are converted into linearized energy values ​​through inverse operation;

[0018] (1.2) Color constancy correction based on deep learning: The color vector of the scene light source is regressed using a convolutional neural network (CNN) model, and white balance correction is performed based on the von Klee transform to restore the skin color under the standard light source;

[0019] (1.3) Skin region segmentation and feature extraction: A binary mask for the skin region is generated by the U-Net segmentation network, and the color and texture features of the mask-covered area are extracted to form a feature representation.

[0020] Furthermore, in step (1.1), the pixel value linearization of the camera response function includes the following steps:

[0021] (1.1.1) Calibration phase: For a specific model of smartphone camera, take a set of N images with different and known exposure times t under controlled conditions. j Image sequence {I j The Debevec & Malik algorithm is used to solve the logarithmic form of the inverse function of the camera response function g(Z) by minimizing the objective function, and the solved function g(Z) is stored in the camera CRF database.

[0022] (1.1.2) Application phase: The application identifies the current device model and loads the corresponding function g(Z) from the CRF database;

[0023] (1.1.3) Conversion stage: For the skin region pixels of the acquired image, the nonlinear pixel value Z is converted into a linearized energy value E that is proportional to the scene irradiance through the inverse operation E = exp(g(Z)).

[0024] Furthermore, in step (1.2), the deep learning-based color constancy correction includes the following steps:

[0025] (1.2.1) The linearized image is input into a pre-trained color constancy deep learning model to regress the normalized RGB color vector L = (R... L G L B L );

[0026] (1.2.2) Based on the von Klee diagonal transform model, the linearized color value of each pixel in the image is white-balanced to restore its color under standard light source.

[0027] Furthermore, in step (1.3), skin region segmentation and feature extraction include the following steps:

[0028] (1.3.1) Input the standardized image into the semantic segmentation network to accurately extract the binary mask of the newborn skin region;

[0029] (1.3.2) Extract relevant features such as color and texture only within the skin area covered by the mask to form feature vectors or feature maps for subsequent analysis.

[0030] Furthermore, in step (2), the self-supervised pre-training of the feature encoder based on spatiotemporal contrastive learning includes the following steps:

[0031] (2.1) Design strong and weak data augmentation strategies: Generate a pair of related views for the input time series data samples, namely a weak augmentation view and a strong augmentation view;

[0032] (2.2) Construct a time-context comparison learning module: execute time comparison tasks and context comparison tasks in parallel;

[0033] (2.3) Model pre-training: The feature encoder is pre-trained end-to-end by minimizing the joint loss function consisting of the weighted sum of temporal contrast loss and context contrast loss.

[0034] Furthermore, in step (2.2), the time comparison task constructs a cross-view future state prediction task. Using an autoregressive model, the feature sequence of a weakly enhanced view of a sample before time t is used as input to predict the features of its strongly enhanced view at time t+k in the future. The context comparison task regards the global features extracted from the weakly and strongly enhanced views of the same newborn at the same time point as a positive sample pair, while all view features of other newborns in the data batch are regarded as negative samples. By optimizing the comparison loss function, the distance between positive sample pairs in the feature space is narrowed, and the distance between them and negative samples is widened.

[0035] Furthermore, step (3) includes the following steps:

[0036] (3.1) Initial supervised fine-tuning: On a small number of in-hospital datasets with real clinical labels, a prediction head is connected to the pre-trained encoder for initial supervised fine-tuning;

[0037] (3.2) Generating and filtering pseudo-labels: Using the fine-tuned model, predict bilirubin values ​​for unlabeled family data, and filter out prediction results with confidence levels higher than a preset threshold as high-quality pseudo-labels.

[0038] (3.3) Retraining based on supervised contrastive learning: Mix in-hospital data with real labels and out-of-hospital data with high-quality pseudo-labels, and use supervised contrastive loss function for final model training.

[0039] Furthermore, step (4) includes the following steps:

[0040] (4.1) Time-series bilirubin prediction: Input the encoder feature sequence of the user for several consecutive days into the time-series prediction module, and output the bilirubin trend for the next several days;

[0041] (4.2) Bayesian uncertainty assessment: During prediction, multiple forward propagations are performed using Monte Carlo Dropout to obtain the mean and variance of the prediction results. The variance is used to quantify the confidence level of the model prediction.

[0042] (4.3) Dynamic risk stratification and early warning: Combine clinical standards to set high-risk bilirubin thresholds for different ages, and dynamically adjust the risk boundary using the predicted mean and uncertainty variance. When the preset risk conditions are met, a high-risk warning is pushed to the user.

[0043] The technical concept of this invention is divided into four stages: calibration visual preprocessing, self-supervised pre-training, semi-supervised fine-tuning, and temporal prediction and uncertainty quantification. (1) The steps of the calibration visual preprocessing stage are as follows: linearization of pixel values ​​of camera response function; color constancy correction of deep learning; skin region segmentation and feature extraction. (2) The steps of the self-supervised pre-training stage are as follows: design of strong and weak data augmentation strategies; construction of time and context contrast learning modules; model pre-training. (3) The steps of the semi-supervised fine-tuning stage are as follows: initial supervised fine-tuning; generation and screening of pseudo-labels; retraining based on supervised contrast learning. (4) The steps of the temporal prediction and uncertainty quantification stage are as follows: temporal bilirubin prediction; Bayesian uncertainty assessment; dynamic risk stratification and early warning.

[0044] The beneficial effects of this invention are mainly reflected in the following aspects: By linearizing pixel values ​​based on camera response functions and correcting color constancy based on deep learning, this invention eliminates the need for users to use physical color charts. Simultaneously, it suppresses color measurement errors introduced by differences in mobile device models and varying home lighting from the data source, improving the accuracy and robustness of subsequent jaundice assessments. By combining a training framework of self-supervised pre-training and semi-supervised fine-tuning, it enhances the model's understanding of jaundice temporal features and its generalization performance. By introducing a temporal prediction module and Bayesian uncertainty quantification, this invention not only provides predictions of jaundice development trends over the next few days but also quantifies the reliability of each prediction, enabling dynamic and personalized risk warnings. Compared to traditional single-point, deterministic assessments, this invention better meets the clinical need for early and reliable intervention. Attached Figure Description

[0045] Figure 1 This is the overall flowchart of the method;

[0046] Figure 2 Flowchart for the data preparation phase;

[0047] Figure 3 This is a flowchart of the semi-supervised learning stage. Detailed Implementation

[0048] The present invention will now be further described with reference to the accompanying drawings.

[0049] Reference Figures 1-3 A semi-supervised, non-contact, home-based intelligent early warning method for neonatal jaundice based on image calibration, characterized in that the method includes the following steps:

[0050] (1) A camera response function database is established, and a color constancy depth model and a skin region segmentation network are constructed. The color constancy depth model includes an input layer, convolutional units 1-4, a global adaptive average pooling layer, a fully connected regression head, and an output layer. The convolutional units are used to extract multi-scale color and texture features from the input image; the global adaptive average pooling layer is used to compress spatial features into a fixed-dimensional vector; the fully connected regression head is used to output the normalized RGB color vector of the scene light source; and the output layer is used to output the light source color vector. The skin region segmentation network adopts a U-Net structure, which includes an encoder downsampling section, a bottleneck layer, a decoder upsampling section, and an output layer. The encoder downsampling section is used to extract features step-by-step and reduce resolution; the bottleneck layer is used to fuse contextual information; the decoder upsampling section is used to gradually restore resolution and fuse encoded features; and the output layer is used to generate a binary skin mask.

[0051] (2) Utilizing a large amount of unlabeled home data, a strong and weak enhancement strategy was designed, and a time and context contrast learning module was constructed to perform self-supervised training on the feature encoder. The time and context contrast learning module sequentially includes a position encoder, a time encoder, a context contrast projection head, a time prediction head, and a contrast loss calculation unit. The time encoder is used to extract temporal features; the context contrast projection head is used to map global features and calculate contrast loss; the time prediction head is used to predict future features based on historical features; and the contrast loss calculation unit is used to calculate and weight the joint loss. The feature encoder sequentially includes an input layer, an initial convolutional block, a residual or convolutional downsampling block, an adaptive average pooling layer, and a feature projection head. The initial convolutional block is used to extract preliminary features of the input image; the residual or convolutional downsampling block is used to extract deep features and reduce dimensionality; the adaptive average pooling layer is used to convert the feature map into a fixed vector; and the feature projection head is used to map and normalize the feature vector.

[0052] (3) By combining a small amount of labeled inpatient data with a large amount of unlabeled family data, semi-supervised fine-tuning is achieved through pseudo-label generation and supervised comparative learning;

[0053] (4) Input the image sequences continuously uploaded by users into the model to achieve multi-day prediction of bilirubin levels, and combine the Bayesian method to quantify the prediction uncertainty, dynamically adjust the risk boundary, and provide personalized early warning.

[0054] Furthermore, in step (1), the image color normalization preprocessing steps are as follows:

[0055] In one specific embodiment, the structure of the color constancy deep model is as follows: starting from the input layer, convolutional unit 1, convolutional unit 2, convolutional unit 3, convolutional unit 4, a global adaptive average pooling layer, a fully connected regression head, and an output layer are connected sequentially. The convolutional units are used to extract multi-scale color and texture representations of the input image; the global adaptive average pooling layer maps spatial features to a fixed-dimensional vector; the fully connected regression head maps this vector to a normalized RGB color vector of the scene light source; the output layer outputs this light source color vector for subsequent white balance correction. Specifically, each convolutional unit uses a 3×3 convolutional kernel, followed by a batch normalization layer and a ReLU activation function; convolutional units 2, 3, and 4 are connected to a max-pooling layer at the end; the number of channels in each convolutional unit can be set to 32, 64, 128, and 256 respectively; the fully connected regression head can be...

[0056] The function is composed of FC(256→128)→ReLU→Dropout(0.3)→FC(128→3), and the output is a linear activation.

[0057] The skin region segmentation network adopts a U-Net structure, which consists of an encoder downsampling section, a bottleneck layer, and a decoder upsampling section, starting from the input layer. Encoder features are fused with decoder features at each stage via skip connections, and the output layer generates a binary skin mask of the same size as the input image. Specifically, each encoder stage contains two layers of "convolution + BN + ReLU" and downsampling is achieved through 2×2 max pooling; the bottleneck layer contains two convolutional units with a maximum of 1024 channels; each decoder stage uses transposed convolution or upsampling followed by concatenation of the corresponding encoder features, and then two more layers of "convolution".

[0058] +BN+ReLU”; the output layer uses a 1×1 convolution followed by a Sigmoid activation function to output a binary mask.

[0059] (1.1) Pixel value linearization based on camera response function (CRF): The camera CRF is calibrated in a controlled environment and stored in the database. During online acquisition, the CRF is called according to the device model, and the nonlinear RGB pixel values ​​are converted into linearized energy values ​​through inverse operation;

[0060] (1.2) Color constancy correction based on deep learning: The color vector of the scene light source is regressed using a convolutional neural network (CNN) model, and white balance correction is performed based on the von Klee transform to restore the skin color under the standard light source;

[0061] (1.3) Skin region segmentation and feature extraction: A binary mask for the skin region is generated by the U-Net segmentation network, and the color and texture features of the mask-covered area are extracted to form a feature representation.

[0062] Furthermore, in step (1.1), the pixel value linearization step of the camera response function is as follows:

[0063] (1.1.1) Perform a one-time calibration for a specific smartphone camera model under controlled conditions. Fix the phone in place and take a set of N (6≤N≤10) photos of a static scene with multiple colors and brightness levels, each with a known exposure time t. j The image sequence {I} (where j = 1, 2, ..., N) is a sequence of images. j Since the scene is static and the lighting is constant, the irradiance I received by any pixel i is... i It remains unchanged; the energy E it absorbs is E. ij With exposure time t j Proportional, that is The pixel value recorded in the image is Z. ij The camera response function F describes the mapping from energy to pixel values: Z ij =F(E ij The goal of this step is to solve for the logarithmic form of its inverse function, g(Z) = ln(F). -1 (Z)). According to ln(E) ij ) = ln(I i )+ln(t j ), we can obtain g(Z) ij ) = ln(I i )+ln(t j The Debevec & Malik algorithm is used to solve for g(Z) and the relative irradiance ln(I) of all pixels by minimizing the following quadratic objective function. i ), Where P is the total number of selected pixels, w(Z) is a weighting function used to increase the smoothness of data fitting, and λ is the coefficient of the smoothing term. The solved function g(Z) (which can be fitted as a piecewise polynomial and its coefficients stored) is stored in a camera CRF database as the "color fingerprint" of this camera model;

[0064] (1.1.2) In the online application phase, the user takes a single routine photo of the newborn to obtain a raw JPEG image. The application (App) provided by this invention first identifies the current device model and loads the corresponding function g(Z) from the CRF database;

[0065] (1.1.3) For each pixel in the skin region of the acquired image, obtain its three-channel (R, G, B) pixel values. Through the inverse operation E = exp(g(Z)), accurately convert the nonlinear pixel value Z into a linearized energy value E that is proportional to the scene irradiance. The output is an image in which the RGB values ​​are linearly related to the illumination energy.

[0066] Furthermore, in step (1.2), the steps for color constancy correction based on deep learning are as follows:

[0067] (1.2.1) The image after linearization in step (1.1) is input into a pre-trained color constancy deep learning model. This model (e.g., a convolutional neural network-based model) is able to regress the normalized RGB color vector L = (R0, R0, R0) of the scene light source from the image content. L G L B L ).

[0068] (1.2.2) Based on the von Kriess diagonal transform model, white balance correction is performed on the linearized color values ​​(R,G,B) of each pixel in the image to restore its color (R′,G′,B′) under standard daylight (e.g., D65 light source).

[0069] Furthermore, in step (1.3), the steps for skin region segmentation and feature extraction are as follows:

[0070] (1.3.1) Input the standardized image into a semantic segmentation network (such as U-Net) to accurately extract the binary mask of the newborn skin region to eliminate interference from irrelevant regions such as background and clothing;

[0071] (1.3.2) Extract relevant features such as color and texture only within the skin area covered by the mask to form feature vectors or feature maps for subsequent analysis.

[0072] In step (2), the self-supervised pre-training steps of the feature encoder based on spatiotemporal contrastive learning are as follows:

[0073] In one specific embodiment, the feature encoder is structured as follows: starting from the input layer, it sequentially connects an initial convolutional block, multiple residual or convolutional downsampling blocks, an adaptive average pooling layer, and a feature projection head. Specifically, the initial convolutional block can employ 7×7 convolution, BN, ReLU, and max pooling; the number of channels in the subsequent four downsampling blocks can be 64, 128, 256, and 512 respectively, with each downsampling block containing 2 to 3 residual units. The adaptive average pooling compresses the spatial features into a fixed-dimensional vector, and the feature projection head can be constructed from FC(512→256)→ReLU→FC(256→128), outputting an L2-normalized feature vector.

[0074] In one specific embodiment, the temporal and contextual contrast learning module is structured as follows: after inputting the feature sequence, it is sequentially connected to a position encoder, a temporal encoder, a contextual contrast projection head, and a temporal prediction head. The temporal encoder is preferably a two-layer Transformer Encoder(d... model =512, feedforward layer dimension 2048, 8-head attention), the context projection head consists of two MLP layers, the output dimension is 128 and is L2 normalized; the temporal prediction head can be an autoregressive Transformer Decoder or LSTM.

[0075] (2.1) Design of strong and weak data augmentation strategies: For each time-series data sample, a pair of related views are generated. The weakly augmented view is generated by applying small perturbations such as random cropping and slight color jitter. The strongly augmented view is generated by applying drastic transformations such as color rendering simulating extreme lighting, adding Gaussian noise, or time-series jitter. These two views are simultaneously fed into the subsequent contrastive learning module;

[0076] (2.2) Constructing a Time-Context Comparison Learning Module: This module performs two tasks in parallel: time comparison (future state prediction) and context comparison (simultaneous strong / weak view discrimination). Time comparison uses an autoregressive Transformer to predict future features, while context comparison uses the InfoNCE loss function to constrain the relative distance between positive and negative samples.

[0077] (2.3) Model Pre-training: By optimizing the joint contrastive loss, the model learns stable and discriminative temporal feature representations under unlabeled conditions. This is achieved by minimizing the joint loss function L, which is a weighted sum of temporal contrastive loss and contextual contrastive loss. unsup =λ tc ·L TC +λ cc ·L CCThe feature encoder was pre-trained end-to-end on all available unlabeled family datasets, resulting in a pre-trained encoder with a deep understanding of the temporal features of jaundice.

[0078] Furthermore, in step (2.1), the steps for designing strong and weak data augmentation strategies are as follows:

[0079] (2.1.1) Weak Augmentation applies small perturbations, such as random cropping, small scaling, or slight color jitter. This view preserves the core features of the image;

[0080] (2.1.2) Strong Augmentation applies drastic transformations, specifically referring to color rendering that simulates extreme lighting conditions that may occur in a home, adding Gaussian noise, and for time series data, performing time-series jitter within a limited range of time segments.

[0081] Furthermore, in step (2.2), the steps for constructing the time-context contrastive learning module are as follows:

[0082] (2.2.1) Construct a cross-view future state prediction task. Using an autoregressive model (such as Transformer), the feature sequence of a sample's "weakly enhanced view" before time t is taken as input to predict its features of the "strongly enhanced view" at time t+k. By optimizing the time-contrast loss function, the model is forced to learn the inherent pattern of jaundice development over time.

[0083] (2.2.2) Global features extracted from the "weakly enhanced view" and "strongly enhanced view" of the same newborn at the same time point are considered as a positive sample pair. All view features of other newborns in the data batch (mini-batch) are considered as negative samples. This is achieved by optimizing the contextual contrast loss function L... CC (Such as InfoNCE loss) brings positive sample pairs closer together in the feature space, while pushing them further apart. This forces the model to learn discriminative features that can distinguish different individuals and are insensitive to augmentation operations.

[0084] The steps in step (3) are as follows:

[0085] (3.1) Initial supervision and fine-tuning

[0086] On a small number of in-hospital datasets with real clinical labels, the pre-trained encoder obtained in step (2) is connected to a prediction head (such as a fully connected layer) and preliminarily supervised fine-tuned so that it initially has the specific ability to predict bilirubin values ​​from image features.

[0087] (3.2) Generating and filtering pseudo-tags

[0088] Using a fine-tuned model, bilirubin levels are predicted from massive amounts of unlabeled household data. Simultaneously, the confidence level of the model for each prediction is evaluated (e.g., through the variance of the predicted values ​​or the entropy of the softmax probability). Only predictions with confidence levels above a preset threshold are selected as high-quality "pseudo-labels."

[0089] (3.3) Retraining based on supervised contrastive learning

[0090] The data consisted of both real-label in-hospital data and high-quality pseudo-labeled out-of-hospital data. A supervised contrastive loss function (L...) was employed. SCC The final model training is then performed. In this loss function, a positive sample pair includes not only its own augmented view but also all other samples within the batch with the same or similar bilirubin values ​​(true or false labels). This allows the model to learn more robust intra-class compactness and inter-class separability, greatly improving the final prediction accuracy.

[0091] The specific steps of step (4) are as follows:

[0092] (4.1) Time-series bilirubin prediction

[0093] The encoder feature sequences from multiple consecutive days are input into the time-series prediction module (optionally a two-layer LSTM or Transformer decoder), which outputs the bilirubin trend for the next H days. Example LSTM structure: Two-layer LSTM (hidden = 128) → FC (128 → 64) → ReLU → FC (64 → H). The training loss uses MAE or MSE, and a time-weighted strategy can be added to prioritize near-future prediction accuracy.

[0094] (4.2) Bayesian uncertainty assessment

[0095] During prediction, M forward propagations are performed using Monte Carlo Dropout (MC Dropout) to obtain the mean of the prediction results. and variance σ 2 The variance quantifies the model's confidence in the prediction.

[0096] (4.3) Dynamic risk stratification and early warning

[0097] Using widely adopted clinical standards such as the Bhutani nomogram, high-risk bilirubin thresholds τ were set for different ages. The early warning system does not simply compare... Instead of relying on τ, the risk boundary is dynamically adjusted using the uncertainty σ of the forecast. For example, a risk sensitivity coefficient k (k≥1) is set, and when the condition is met... Even if the predicted mean Even if the threshold has not been exceeded, the system will still send a high-risk warning to the user in advance, considering the high uncertainty that may cover the danger zone, and recommend that they seek medical attention promptly. This approach achieves more reliable and personalized intelligent risk alerts.

[0098] In a preferred embodiment of the present invention, to address the challenge of covering a large number of smartphone models in practical applications, a device clustering strategy can be employed. That is, smartphone models using similar or identical camera sensors and image signal processors (ISPs) are grouped together, sharing a single camera response function (CRF) averaged or optimized from multiple devices within that category. This significantly reduces the complexity and cost of calibration while ensuring basic measurement accuracy. In another preferred embodiment, the present invention can collaborate with specific medical institutions or maternal and child health platforms to provide services for a few mainstream smartphone models that are uniformly recommended or designated by them. Precise calibration ensures the highest monitoring accuracy and reliability within a limited range of devices.

[0099] In the initial deployment phase of this invention, to address the cold start problem caused by sparse unlabeled home data due to a small user base, the self-supervised pre-training in step (2) can employ a transfer learning strategy. Specifically, the feature encoder is first pre-trained on a large-scale, publicly available dataset of natural or skin images (such as ImageNet or DermNet) to learn general and robust visual representation capabilities. Then, targeted secondary pre-training and semi-supervised fine-tuning are performed using existing, limited amounts of labeled jaundice data from hospitals and unlabeled data from outpatient settings. As the application expands and user data accumulates, the model can be periodically updated online or offline to continuously optimize its performance.

Claims

1. A semi-supervised, non-contact, home-based intelligent early warning method for neonatal jaundice based on image calibration, comprising the following steps: (1) A camera response function database is established, and a color constancy depth model and a skin region segmentation network are constructed. The color constancy depth model includes an input layer, convolutional units 1-4, a global adaptive average pooling layer, a fully connected regression head, and an output layer. The convolutional units are used to extract multi-scale color and texture features from the input image; the global adaptive average pooling layer is used to compress spatial features into a fixed-dimensional vector; the fully connected regression head is used to output the normalized RGB color vector of the scene light source; and the output layer is used to output the light source color vector. The skin region segmentation network adopts a U-Net structure, which includes an encoder downsampling section, a bottleneck layer, a decoder upsampling section, and an output layer. The encoder downsampling section is used to extract features step-by-step and reduce resolution; the bottleneck layer is used to fuse contextual information; the decoder upsampling section is used to gradually restore resolution and fuse encoded features; and the output layer is used to generate a binary skin mask. (2) Utilizing a large amount of unlabeled home data, a strong and weak enhancement strategy was designed, and a time and context contrast learning module was constructed to perform self-supervised training on the feature encoder. The time and context contrast learning module sequentially includes a position encoder, a time encoder, a context contrast projection head, a time prediction head, and a contrast loss calculation unit. The time encoder is used to extract temporal features; the context contrast projection head is used to map global features and calculate contrast loss; the time prediction head is used to predict future features based on historical features; and the contrast loss calculation unit is used to calculate and weight the joint loss. The feature encoder sequentially includes an input layer, an initial convolutional block, a residual or convolutional downsampling block, an adaptive average pooling layer, and a feature projection head. The initial convolutional block is used to extract preliminary features of the input image; the residual or convolutional downsampling block is used to extract deep features and reduce dimensionality; the adaptive average pooling layer is used to convert the feature map into a fixed vector; and the feature projection head is used to map and normalize the feature vector. (3) By combining a small amount of labeled inpatient data with a large amount of unlabeled family data, semi-supervised fine-tuning is achieved through pseudo-label generation and supervised comparative learning; (4) Input the image sequences continuously uploaded by users into the model to achieve multi-day prediction of bilirubin levels, and combine the Bayesian method to quantify the prediction uncertainty, dynamically adjust the risk boundary, and provide personalized early warning.

2. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 1, characterized in that, The steps of image color normalization preprocessing in step (1) are as follows: (1.1) Pixel value linearization based on camera response function (CRF): The camera CRF is calibrated in a controlled environment and stored in the database. During online acquisition, the CRF is called according to the device model, and the nonlinear RGB pixel values ​​are converted into linearized energy values ​​through inverse operation. (1.2) Color constancy correction based on deep learning: The color vector of the scene light source is regressed using a convolutional neural network (CNN) model, and white balance correction is performed based on the von Kris transform to restore the skin color under the standard light source; (1.3) Skin region segmentation and feature extraction: A binary mask for the skin region is generated by the U-Net segmentation network, and the color and texture features of the mask-covered area are extracted to form a feature representation.

3. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 2, characterized in that, The pixel value linearization of the camera response function in step (1.1) includes the following steps: (1.1.1) Calibration phase: For a specific model of smartphone camera, take a set of N images with different and known exposure times t under controlled conditions. j Image sequence {I j The Debevec & Malik algorithm is used to solve the logarithmic form of the inverse function of the camera response function g(Z) by minimizing the objective function, and the solved function g(Z) is stored in the camera CRF database. (1.1.2) Application phase: The application identifies the current device model and loads the corresponding function g(Z) from the CRF database; (1.1.3) Conversion stage: For the skin region pixels of the acquired image, the nonlinear pixel value Z is converted into a linearized energy value E that is proportional to the scene irradiance through the inverse operation E = exp(g(Z)).

4. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 2, characterized in that, Step (1.2) of deep learning-based color constancy correction includes the following steps: (1.2.1) The linearized image is input into a pre-trained color constancy deep learning model to regress the normalized RGB color vector L = (R... L G L B L ); (1.2.2) Based on the von Klee diagonal transform model, the linearized color value of each pixel in the image is white-balanced to restore its color under standard light source.

5. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 2, characterized in that, Step (1.3) skin region segmentation and feature extraction includes the following steps: (1.3.1) Input the standardized image into the semantic segmentation network to accurately extract the binary mask of the newborn skin region; (1.3.2) Extract relevant features such as color and texture only within the skin area covered by the mask to form feature vectors or feature maps for subsequent analysis.

6. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 1, characterized in that, In step (2), the self-supervised pre-training of the feature encoder based on spatiotemporal contrastive learning includes the following steps: (2.1) Design strong and weak data augmentation strategies: Generate a pair of related views for the input time series data samples, namely a weak augmentation view and a strong augmentation view; (2.2) Construct a time-context comparison learning module: execute time comparison tasks and context comparison tasks in parallel; (2.3) Model pre-training: The feature encoder is pre-trained end-to-end by minimizing the joint loss function consisting of the weighted sum of temporal contrast loss and context contrast loss.

7. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 6, characterized in that, In step (2.2), the time comparison task constructs a cross-view future state prediction task. Using an autoregressive model, the feature sequence of a sample's weakly enhanced view before time t is used as input to predict the features of its strongly enhanced view at time t+k in the future. The contextual contrast task treats the global features extracted from the weakly and strongly enhanced views of the same newborn at the same time point as a positive sample pair, while all view features of other newborns in the data batch are treated as negative samples. By optimizing the contrastive loss function, the distance between positive sample pairs in the feature space is narrowed, while the distance between them and negative samples is widened.

8. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 2, characterized in that, Step (3) includes the following steps: (3.1) Initial supervised fine-tuning: On a small number of in-hospital datasets with real clinical labels, a prediction head is connected to the pre-trained encoder for initial supervised fine-tuning; (3.2) Generating and filtering pseudo-labels: Using the fine-tuned model, predict bilirubin values ​​for unlabeled family data, and filter out prediction results with confidence levels higher than a preset threshold as high-quality pseudo-labels. (3.3) Retraining based on supervised contrastive learning: Mix in-hospital data with real labels and out-of-hospital data with high-quality pseudo-labels, and use supervised contrastive loss function for final model training.

9. The semi-supervised non-contact home-based intelligent early warning method for neonatal jaundice based on image calibration as described in claim 2, characterized in that, Step (4) includes the following steps: (4.1) Time-series bilirubin prediction: Input the encoder feature sequence of the user for several consecutive days into the time-series prediction module, and output the bilirubin trend for the next several days; (4.2) Bayesian uncertainty assessment: During prediction, multiple forward propagations are performed using Monte Carlo Dropout to obtain the mean and variance of the prediction results. The variance is used to quantify the confidence level of the model prediction. (4.3) Dynamic risk stratification and early warning: Combine clinical standards to set high-risk bilirubin thresholds for different ages, and dynamically adjust the risk boundary using the predicted mean and uncertainty variance. When the preset risk conditions are met, a high-risk warning is pushed to the user.

Images