Bilirubin detection method based on image analysis

Abstract

The invention relates to the technical field of bilirubin detection, and provides a bilirubin detection method based on image analysis, which comprises the following six main steps: firstly, acquiring a forearm skin image of a subject by using photographic equipment with a standardized light source; secondly, preprocessing the image, including denoising, color correction and contrast enhancement; 3, establishing a multi-task learning model, and extracting image features; fourthly, a self-attention mechanism and a multi-scale feature fusion module are introduced, and the precision and robustness of image segmentation are improved; 5, extracting bottom-layer features from the segmented image, wherein the bottom-layer features comprise color, texture and shape features; and finally, fusing various features, and outputting a final predicted value through a bilirubin concentration prediction decoder. In addition, image segmentation and bilirubin concentration prediction tasks are optimized by using a joint loss function, and the overall performance of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of bilirubin detection, and more specifically, to a bilirubin detection method based on image analysis. Background Art

[0002] Bilirubin is an important metabolite in the human body, mainly produced by the decomposition of hemoglobin in aging red blood cells. Under normal physiological conditions, bilirubin is excreted from the bile after being metabolized by the liver. However, when bilirubin is overproduced or metabolism or excretion is impaired, it will lead to increased bilirubin levels in the body and cause a variety of pathological conditions. Accurate measurement of bilirubin concentration is of great significance for the diagnosis and monitoring of a variety of diseases, such as neonatal jaundice, liver dysfunction, hemolytic diseases, etc. Traditional bilirubin detection methods mainly rely on blood sample analysis. Although this method has high accuracy, it has disadvantages such as invasiveness, complex operation, and long detection time, which is not conducive to frequent monitoring and rapid diagnosis.

[0003] In recent years, with the rapid development of computer vision and artificial intelligence technology, non-invasive bilirubin detection technology based on image analysis has attracted widespread attention. This method estimates bilirubin concentration by analyzing skin color changes, and has the advantages of simple operation, strong real-time performance, and good repeatability. Image analysis technology has achieved remarkable results in the fields of medical image processing and computer-aided diagnosis, and its application in bilirubin detection provides new possibilities for clinical practice. This technology usually involves steps such as image acquisition, preprocessing, feature extraction, and machine learning model training. It achieves non-invasive detection by establishing a mapping relationship between skin color features and bilirubin concentration.

[0004] In the existing public literature, in document 1 (Use of a smartphone app to assess neonatal jaundice. Pediatrics, 2017, 140 (3), e20170312), Taylor et al. proposed a method for screening neonatal jaundice based on a smartphone camera, which estimates serum bilirubin levels by analyzing digital images of the forehead skin of newborns. The results of the study showed that this method showed good prospects in clinical applications and provided a new idea for non-invasive bilirubin detection. In document 2 (Smartphone-based neonatal serum bilirubin measurement using multiple colour space analysis. Acta Paediatrica, 2020, 109 (9), 1851-1856), Outram et al. further improved this method and proposed a smartphone bilirubin measurement technology based on multi-color space analysis. This method improves the accuracy and stability of bilirubin concentration estimation by analyzing image features in multiple color spaces.

[0005] However, these studies still have some limitations. For example, Reference 2 mainly focuses on color space analysis, lacks in-depth consideration of image texture and shape features, and cannot fully utilize the multi-dimensional information in skin images; in addition, existing methods still need to be optimized in image segmentation and feature fusion, especially under complex lighting conditions, the segmentation accuracy and robustness of feature extraction still need to be improved; at the same time, the current feature fusion strategy is relatively simple, and it is difficult to effectively integrate image information at different levels, which limits the further improvement of detection performance.

[0006] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art, the present invention provides a bilirubin detection method based on image analysis, which realizes accurate segmentation of skin images by adopting a multi-task learning model for feature extraction, introducing a self-attention mechanism and multi-scale feature fusion in the image segmentation process, and comprehensively utilizing color, texture and shape features, and adopting a joint loss function to optimize model performance, thereby solving the problems of incomplete feature extraction, insufficient segmentation accuracy and simple feature fusion strategy in the prior art, and improving the accuracy and stability of bilirubin detection.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A bilirubin detection method based on image analysis comprises the following steps:

[0010] Step 1: Use a photographic device equipped with a standardized light source to capture images of the subject's forearm skin area, and control the shooting angle and time;

[0011] Step 2: preprocess the collected forearm skin image, use bilateral filtering method to remove noise, use color space transformation to perform color correction, and use adaptive histogram equalization method to enhance contrast;

[0012] Step 3, establishing a multi-task learning model including a shared encoder, an image segmentation decoder and a bilirubin concentration prediction decoder, processing the preprocessed forearm skin image through the shared encoder to extract feature maps;

[0013] Step 4: introduce a self-attention mechanism in the skip connection between the shared encoder and the image segmentation decoder, and use the U-Net deep learning model that introduces multi-scale feature fusion as the image segmentation decoder to decode the feature map output by the shared encoder to generate an image segmentation result;

[0014] Step 5, extracting underlying features from the segmented forearm skin image, wherein the underlying features include color features, texture features, and shape features;

[0015] Step 6: Fusing the image segmentation result, the feature map extracted by the shared encoder and the underlying features, and outputting the final prediction value through the bilirubin concentration prediction decoder, including the following steps:

[0016] Step S1, standardize and fuse the image segmentation result, the feature map extracted by the shared encoder, the color feature, the texture feature and the shape feature to form a comprehensive feature vector, and the fusion formula is:

[0017]

[0018] In the formula, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, S seg is the feature vector of image segmentation result, F high is the feature map vector extracted by the shared encoder, μ C is the mean of the color feature vector, μ T is the mean value of the texture feature vector, μ S is the mean of the shape eigenvector, μ seg is the mean of the feature vector of the image segmentation result, μ high Extract the mean of the feature map vector for the shared encoder, σ C is the standard deviation of the color feature vector, σ T is the standard deviation of the texture feature vector, σ S is the standard deviation of the shape eigenvector, σ concis the standard deviation of the bilirubin concentration feature vector, σ seg is the standard deviation of the feature vector of the image segmentation result, σ high is the standard deviation of the feature map vector extracted by the shared encoder, and concat() is the concatenation operation of the standardized feature vector;

[0019] Step S2, using a bilirubin concentration prediction decoder to process the fused features, the bilirubin concentration prediction decoder includes a fully connected neural network and a regression layer, and outputs a bilirubin concentration prediction value through the regression layer, and its prediction formula is:

[0020]

[0021] In the formula, is the predicted value of bilirubin concentration, S seg is the feature vector of image segmentation result, F high is the feature mapping vector extracted by the shared encoder, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, W1 and W2 are the weight matrices of the fully connected layers, b1 and b2 are the corresponding bias terms, ReLU() is the activation function, which means the rectified linear unit function, and f() is the linear activation function of the regression layer.

[0022] As a further solution of the present invention, in step one, the illumination intensity of the standardized light source is not less than 500 lux; during the acquisition process, the vertical error of the shooting angle relative to the forearm skin area does not exceed ±5 degrees, and the shooting time is controlled to be completed within 1 second.

[0023] As a further solution of the present invention, in step 4, a self-attention mechanism is introduced in the skip connection between the shared encoder and the image segmentation decoder, and a multi-scale feature fusion module is introduced in the image segmentation decoder, including the following steps:

[0024] Step A1, modify the U-Net model structure, insert a self-attention module in the skip connection between the shared encoder and the image segmentation decoder, and introduce a multi-scale feature fusion module in the image segmentation decoder;

[0025] Step A2, inputting the preprocessed forearm skin image into a shared encoder, and extracting forearm skin color feature maps of multiple scales through the encoder;

[0026] Step A3, applying a self-attention mechanism to the obtained forearm skin color feature map, calculating the bilirubin correlation between different positions on the feature map, generating an attention matrix, and using the attention matrix to weight the original feature map to generate an attention-weighted feature map reflecting the distribution of skin bilirubin;

[0027] Step A4, performing element-wise multiplication operation on the generated attention weighted feature map and the original forearm skin color feature map to obtain a weighted bilirubin distribution feature map;

[0028] Step A5, passing the obtained weighted bilirubin distribution feature map to the decoder module corresponding to the current encoder level in the U-Net model through a skip connection;

[0029] Step A6, at each level of the decoder, collect the weighted feature map from the corresponding level of the encoder, the output feature map of the previous decoder, and the feature map of the current level;

[0030] Step A7, adjusting the spatial scales of the feature maps with different resolutions collected in step A6 so that they have the same spatial dimension;

[0031] Step A8, compressing the adjusted multi-scale feature map through a convolution operation to reduce the feature dimension;

[0032] Step A9, using a channel attention mechanism to fuse the compressed multi-scale feature maps to generate a fused feature map;

[0033] Step A10, inputting the fused feature map into the convolutional layer of the current decoder level for further processing and upsampling;

[0034] Step A11, repeating steps A6 to A10 until all decoder levels are processed to generate a final bilirubin distribution area segmentation result.

[0035] As a further solution of the present invention, in step 4, the image segmentation result includes a binary image of the segmented skin area, geometric features of the segmented area, location information of the segmented area, and a shape descriptor of the segmented area.

[0036] As a further solution of the present invention, in step five, the extracted color features are represented as corresponding color feature vectors, including calculating the color average, standard deviation and color histogram of the segmented area.

[0037] As a further solution of the present invention, in step five, the extracted texture features are represented as corresponding texture feature vectors, including using a grayscale co-occurrence matrix to calculate the image contrast, entropy value, uniformity of distribution of grayscale values ​​in the image, and linear correlation between pixel values ​​in the image.

[0038] As a further solution of the present invention, in step five, the extracted shape features are represented as corresponding shape feature vectors, including calculating the perimeter, area, shape moment and contour complexity of the segmented region.

[0039] As a further solution of the present invention, in step 6, a joint loss function is used to optimize the image segmentation and bilirubin concentration prediction tasks, and the weighted sum of the segmentation loss and the concentration prediction loss is defined as:

[0040] L total =α·L seg +β·L reg ;

[0041] Where, L total is the total loss function of the multi-task learning model, L seg is the image segmentation loss, L reg is the bilirubin concentration prediction loss, α and β are the weight factors of segmentation loss and regression loss, respectively, and α+β=1.

[0042] Compared with the prior art, the beneficial effects of the bilirubin detection method based on image analysis of the present invention are:

[0043] The present invention improves the consistency of image quality by controlling lighting conditions and shooting parameters; the introduction of a multi-task learning model realizes the coordinated optimization of feature extraction and image segmentation; the self-attention mechanism significantly enhances the sensitivity to skin bilirubin distribution; the multi-scale feature fusion technology effectively captures skin feature information at different scales; the comprehensive use of color, texture and shape features enriches the basis for bilirubin concentration prediction; and the optimized joint loss function significantly improves the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention is a schematic diagram of the process of a bilirubin detection method based on image analysis. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Example 1

[0047] A bilirubin detection method based on image analysis comprises the following steps:

[0048] Step 1: Use a photographic device equipped with a standardized light source to capture images of the subject's forearm skin area, and control the shooting angle and time;

[0049] Step 2: preprocess the collected forearm skin image, use bilateral filtering method to remove noise, use color space transformation to perform color correction, and use adaptive histogram equalization method to enhance contrast;

[0050] Step 3, establishing a multi-task learning model including a shared encoder, an image segmentation decoder and a bilirubin concentration prediction decoder, processing the preprocessed forearm skin image through the shared encoder to extract feature maps;

[0051] Step 4: introduce a self-attention mechanism in the skip connection between the shared encoder and the image segmentation decoder, and use the U-Net deep learning model that introduces multi-scale feature fusion as the image segmentation decoder to decode the feature map output by the shared encoder to generate an image segmentation result;

[0052] Step 5, extracting underlying features from the segmented forearm skin image, wherein the underlying features include color features, texture features, and shape features;

[0053] Step six, the image segmentation result, the feature map extracted by the shared encoder and the underlying features are fused, and the final prediction value is output through the bilirubin concentration prediction decoder.

[0054] In step one of the embodiment of the present invention, image acquisition is performed using a camera device including but not limited to a digital camera, a smartphone camera or a dedicated medical imaging device. To ensure image quality, the camera device needs to be equipped with a standardized light source with a light intensity of not less than 500 lux. During the acquisition process, the image resolution is required to be not less than 1080x720 pixels to ensure that the details of the skin area in the image are clear. The vertical error of the shooting angle relative to the skin surface does not exceed ±5 degrees to reduce the impact of the shooting angle on image analysis. During shooting, the light must be uniform to avoid the influence of strong light or shadows, and the shooting time must be controlled to be completed within 1 second to avoid image blur caused by movement.

[0055] In step 2 of the implementation of the present invention, a bilateral filtering method is used for denoising. Bilateral filtering can smooth the noise in the image while maintaining the sharpness of the image edges. For each pixel point (x, y) in the image, the color difference and spatial difference of the pixels in the neighborhood are used to perform weighted averaging to obtain the denoised pixel value.

[0056] In step 2 of the implementation of the present invention, the color space conversion technology is used to convert the image from the RGB space to the Lab* color space to remove the influence of ambient light on the color and ensure that the color information of the skin area in the image accurately reflects its bilirubin concentration.

[0057] The color space conversion technology used in the embodiment of the present invention converts the image from the RGB space to the Lab* color space to remove the influence of ambient light on the color, wherein the conversion formula is:

[0058]

[0059] Where, L * is the brightness component, a * and b * is the color component.

[0060] In step 2 of the implementation of the present invention, an adaptive histogram equalization method is used to enhance the subtle color differences in the image to highlight the color features of the skin area and improve the accuracy of feature extraction.

[0061] In step three of the embodiment of the present invention, the main function of the shared encoder is to extract rich feature maps from the preprocessed forearm skin image, including a series of multi-channel two-dimensional matrices, each matrix representing different features of the image; these features include but are not limited to: low-level edge, texture and color information, mid-level shape and structural features, and high-level semantic information.

[0062] The shared encoder uses a deep convolutional neural network structure to gradually extract and combine these features through multi-level convolution, pooling and nonlinear activation operations; in particular, for the bilirubin detection task, the shared encoder focuses on extracting features related to skin color changes, while also capturing factors such as skin texture and vascular distribution that may affect the estimation of bilirubin concentration. These multi-dimensional and multi-scale feature maps provide a comprehensive and rich information basis for subsequent image segmentation and bilirubin concentration prediction tasks. The image segmentation decoder uses these feature maps, combined with the self-attention mechanism and multi-scale feature fusion technology, to achieve accurate segmentation of the skin area; at the same time, the bilirubin concentration prediction decoder is based on the same feature map and uses a deep learning algorithm to establish a complex mapping relationship between skin features and bilirubin concentration.

[0063] In step 4 of the embodiment of the present invention, a self-attention mechanism is introduced in the skip connection between the shared encoder and the image segmentation decoder, and a multi-scale feature fusion module is introduced in the image segmentation decoder, including the following steps:

[0064] Step A1, modify the U-Net model structure, insert a self-attention module in the skip connection between the shared encoder and the image segmentation decoder, and introduce a multi-scale feature fusion module in the image segmentation decoder;

[0065] Step A2, inputting the preprocessed forearm skin image into a shared encoder, and extracting forearm skin color feature maps of multiple scales through the encoder;

[0066] Step A3, applying a self-attention mechanism to the obtained forearm skin color feature map, calculating the bilirubin correlation between different positions on the feature map, generating an attention matrix, and using the attention matrix to weight the original feature map to generate an attention-weighted feature map reflecting the distribution of skin bilirubin;

[0067] Step A4, performing element-wise multiplication operation on the generated attention weighted feature map and the original forearm skin color feature map to obtain a weighted bilirubin distribution feature map;

[0068] Step A5, passing the obtained weighted bilirubin distribution feature map to the decoder module corresponding to the current encoder level in the U-Net model through a skip connection;

[0069] Step A6, at each level of the decoder, collect the weighted feature map from the corresponding level of the encoder, the output feature map of the previous decoder, and the feature map of the current level;

[0070] Step A7, adjusting the spatial scales of the feature maps with different resolutions collected in step A6 so that they have the same spatial dimension;

[0071] Step A8, compressing the adjusted multi-scale feature map through a convolution operation to reduce the feature dimension;

[0072] Step A9, using a channel attention mechanism to fuse the compressed multi-scale feature maps to generate a fused feature map;

[0073] Step A10, inputting the fused feature map into the convolutional layer of the current decoder level for further processing and upsampling;

[0074] Step A11, repeating steps A6 to A10 until all decoder levels are processed to generate a final bilirubin distribution area segmentation result.

[0075] In the embodiment of the present invention, a self-attention module is added to the jump connection, so that the model can pay more attention to the key feature information of the forearm skin, especially in low-contrast images, and improve the segmentation accuracy of the model for small targets; a multi-scale feature fusion module is introduced in the decoder part, and by fusing feature maps from different resolutions, the model can capture information of targets of different scales, and improve the segmentation accuracy of the model on the bilirubin distribution boundary of the forearm skin, especially for bilirubin deposition areas with complex edges.

[0076] The self-attention mechanism in the embodiment of the present invention is implemented by calculating the query, key and value matrices, where the dot product of the query and the key is used to generate the attention weight, and the product of the value and the attention weight is used to generate the final attention output.

[0077] The multi-scale feature fusion in the embodiment of the present invention is achieved by the following steps: first, spatial scale adjustment is performed on feature maps of different scales, then the number of channels is reduced using 1x1 convolution, and finally the adjusted features are weightedly fused through a channel attention mechanism.

[0078] In step 4 of the embodiment of the present invention, the image segmentation result includes a binary image of the segmented skin area, geometric features of the segmented area, location information of the segmented area, and a shape descriptor of the segmented area.

[0079] In step five of the embodiment of the present invention, the extracted color features are represented as corresponding color feature vectors, including calculating the color average, standard deviation and color histogram of the segmented area.

[0080] In step 5 of the embodiment of the present invention, the extracted texture features are expressed as corresponding texture feature vectors, including using a gray level co-occurrence matrix to calculate the contrast, entropy, uniformity and correlation of the image, wherein the image contrast is calculated to measure the degree of difference in pixel gray values ​​in the image, and the formula is:

[0081]

[0082] Where i and j are the row and column indices in the gray-level co-occurrence matrix, N is the number of gray levels, and p(i,j) is the element value of the i-th row and j-th column in the gray-level co-occurrence matrix.

[0083] The image entropy value is calculated to measure the complexity of the image information. The formula is:

[0084]

[0085] Where i and j are the row and column indices in the gray-level co-occurrence matrix, N is the number of gray levels, and p(i,j) is the element value of the i-th row and j-th column in the gray-level co-occurrence matrix.

[0086] Image uniformity is calculated to indicate the uniformity of the distribution of gray values ​​in the image. The formula is:

[0087]

[0088] Where i and j are the row and column indices in the gray-level co-occurrence matrix, N is the number of gray levels, and p(i,j) is the element value of the i-th row and j-th column in the gray-level co-occurrence matrix.

[0089] Image correlation is calculated to measure the degree of linear correlation between pixel values ​​in an image. The formula is:

[0090]

[0091] Where i and j are the row and column indices in the gray-level co-occurrence matrix, N is the number of gray levels, p(i,j) is the element value of the i-th row and j-th column in the gray-level co-occurrence matrix, and μ i and μ j is the mean of the rows and columns, σ i and σ j are the row and column standard deviations.

[0092] In step five of the embodiment of the present invention, the extracted shape features are represented as corresponding shape feature vectors, including calculating the perimeter, area, shape moment and contour complexity of the segmented region.

[0093] Step six in the embodiment of the present invention includes the following steps:

[0094] Step S1, standardize and fuse the image segmentation result, the feature map extracted by the shared encoder, the color feature, the texture feature and the shape feature to form a comprehensive feature vector, and the fusion formula is:

[0095]

[0096] In the formula, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, S seg is the feature vector of image segmentation result, E high is the feature map vector extracted by the shared encoder, μ C is the mean of the color feature vector, μ T is the mean value of the texture feature vector, μ S is the mean of the shape eigenvector, μ seg is the mean of the feature vector of the image segmentation result, μ high Extract the mean of the feature map vector for the shared encoder, σ C is the standard deviation of the color feature vector, σ T is the standard deviation of the texture feature vector, σ S is the standard deviation of the shape eigenvector, σ conc is the standard deviation of the bilirubin concentration feature vector, σ seg is the standard deviation of the feature vector of the image segmentation result, σ high is the standard deviation of the feature map vector extracted by the shared encoder, and concat() is the concatenation operation of the standardized feature vector;

[0097] Step S2, using a bilirubin concentration prediction decoder to process the fused features, the bilirubin concentration prediction decoder includes a fully connected neural network and a regression layer, and outputs a bilirubin concentration prediction value through the regression layer, and its prediction formula is:

[0098]

[0099] In the formula, is the predicted value of bilirubin concentration, S seg is the feature vector of image segmentation result, F high is the feature mapping vector extracted by the shared encoder, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, W1 and W2 are the weight matrices of the fully connected layers, b1 and b2 are the corresponding bias terms, ReLU() is the activation function, which means the rectified linear unit function, and f() is the linear activation function of the regression layer.

[0100] In step six of the embodiment of the present invention, a comprehensive integrated feature vector is formed by standardizing and fusing the image segmentation results, the high-level feature maps extracted by the shared encoder, and the multidimensional features such as color, texture and shape. This fusion strategy not only solves the problem of the simple feature fusion strategy mentioned in the background technology, but also ensures the equal contribution of different types of features in the fusion process. The standardization processing used effectively eliminates the influence of different feature scales, and the concat() operation realizes the lossless splicing of features. Subsequently, a bilirubin concentration prediction decoder including a fully connected neural network and a regression layer is designed to output the predicted value of bilirubin concentration by processing the fused features, wherein the use of the ReLU activation function enhances the nonlinear expression ability of the model, and the linear activation function of the last layer is suitable for the output of the regression task. This structural design makes full use of the high-level features extracted by the shared encoder in the multi-task learning model, and combines the traditional color, texture and shape features to realize the accurate mapping from multi-angle and multi-scale features to bilirubin concentration, thereby improving the accuracy and robustness of the prediction.

[0101] In step six of the embodiment of the present invention, a joint loss function is used to optimize the image segmentation and bilirubin concentration prediction tasks, and the weighted sum of the segmentation loss and the concentration prediction loss is defined as:

[0102] L total =α*L seg +β·L reg ;

[0103] Where, L total is the total loss function of the multi-task learning model, L seg is the image segmentation loss, L reg is the bilirubin concentration prediction loss, α and β are the weight factors of segmentation loss and regression loss, respectively, and α+β=1.

[0104] In step six of the embodiment of the present invention, a joint loss function is used to optimize the image segmentation and bilirubin concentration prediction tasks, and the total loss function is composed of a weighted image segmentation loss and a bilirubin concentration prediction loss, where the weight factor is used to adjust the relative importance of the two tasks, and the sum is 1; this design can balance the learning objectives of the two tasks and improve the overall performance.

[0105] The image segmentation loss in the embodiment of the present invention adopts Dice loss to measure the difference between the predicted segmentation mask and the true annotation, and its calculation formula is:

[0106]

[0107] Where X is the predicted segmentation mask, i.e., the output of the image segmentation decoder, Y is the true annotation, i.e., the correct segmentation annotation provided in the dataset, which is used to compare with the predicted result, and ε is a smoothing term used to prevent the denominator from being zero.

[0108] The bilirubin concentration prediction loss in the embodiment of the present invention uses the mean square error to measure the deviation between the predicted bilirubin concentration and the true value, and its calculation formula is:

[0109]

[0110] Where B is the actual bilirubin concentration, is the predicted value, and n is the number of samples.

[0111] In step six of the embodiment of the present invention, during the training process, the learning of the two tasks is balanced by adjusting the values ​​of α and β; the initial setting is α=β=0.5, and then dynamically adjusted according to the performance of the model on the validation set.

[0112] The following is a Python code example that uses Python to implement the loss function. Please note that this example is only a starting point and may need to be adjusted according to actual conditions and device interfaces in actual applications;

[0113] Define a custom loss function class:

[0114] class JointLoss(nn.Module):

[0115] def__init__(self,alpha=0.5,beta=0.5):

[0116] super(JointLoss,self).__init__()

[0117] self.alpha = alpha

[0118] self.beta = beta

[0119] self.seg_loss=DiceLoss()#Assuming Dice loss is used

[0120] self.reg_loss = nn.MSELoss() #Assume that MSE loss is used

[0121] def forward(self,seg_pred,seg_true,reg_pred,reg_true):

[0122] seg_loss=self.seg_loss(seg_pred,seg_true)

[0123] reg_loss=self.reg_loss(reg_pred,reg_true)

[0124] return self.alpha*seg_loss+self.beta*reg_loss

[0125] To find the optimal α and β values, use sklearn's GridSearchCV to perform a grid search:

[0126] from sklearn.model_selection import GridSearchCV

[0127] param_grid={'alpha':[0.3,0.4,0.5,0.6,0.7],

[0128] 'beta':[0.3,0.4,0.5,0.6,0.7]}

[0129] grid_search=GridSearchCV(estimator=model,param_grid=param_grid,cv=5)

[0130] grid_search.fit(X_train,y_train)

[0131] best_alpha=grid_search.best_params_['alpha']

[0132] best_beta=grid_search.best_params_['beta']

[0133] In addition, a learning rate scheduler is designed to increase the α value in the early stage of training to prioritize the learning of the segmentation task, and then gradually increase the β value to strengthen the learning of concentration prediction:

[0134] def adjust_loss_weights(epoch,max_epochs):

[0135] alpha=max(0.5,1-epoch / max_epochs)

[0136] beta = 1-alpha

[0137] return alpha,beta

[0138] #In the training loop

[0139] for epoch in range(max_epochs):

[0140] alpha,beta=adjust_loss_weights(epoch,max_epochs)

[0141] criterion=JointLoss(alpha=alpha,beta=beta)

[0142] # Training steps...

[0143] This code is only used as an example. In actual applications, appropriate modifications and adjustments need to be made according to specific circumstances.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0145] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bilirubin detection method based on image analysis, characterized in that: The following steps are involved: Step 1: Use a photographic device equipped with a standardized light source to capture images of the subject's forearm skin area, and control the shooting angle and time; Step 2: preprocess the collected forearm skin image, use bilateral filtering method to remove noise, use color space transformation to perform color correction, and use adaptive histogram equalization method to enhance contrast; Step 3, establishing a multi-task learning model including a shared encoder, an image segmentation decoder and a bilirubin concentration prediction decoder, processing the preprocessed forearm skin image through the shared encoder to extract feature maps; Step 4: introduce a self-attention mechanism in the skip connection between the shared encoder and the image segmentation decoder, use the U-Net deep learning model that introduces multi-scale feature fusion as the image segmentation decoder, decode the feature map output by the shared encoder, and generate the image segmentation result; Step 5, extracting underlying features from the segmented forearm skin image, wherein the underlying features include color features, texture features, and shape features; Step 6: Fusing the image segmentation result, the feature map extracted by the shared encoder and the underlying features, and outputting the final prediction value through the bilirubin concentration prediction decoder, including the following steps: Step S1, standardize and fuse the image segmentation result, the feature map extracted by the shared encoder, the color feature, the texture feature and the shape feature to form a comprehensive feature vector, and the fusion formula is: In the formula, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, S seg is the feature vector of image segmentation result, F gigh is the feature map vector extracted by the shared encoder, μ C is the mean of the color feature vector, μ T is the mean value of the texture feature vector, μ S is the mean of the shape eigenvector, μ seg is the mean of the feature vector of the image segmentation result, μ high Extract the mean of the feature map vector for the shared encoder, σ C is the standard deviation of the color feature vector, σ T is the standard deviation of the texture feature vector, σ S is the standard deviation of the shape eigenvector, σ conc is the standard deviation of the bilirubin concentration feature vector, σ seg is the standard deviation of the feature vector of the image segmentation result, σ high is the standard deviation of the feature map vector extracted by the shared encoder, and concat() is the concatenation operation of the standardized feature vector; Step S2, using a bilirubin concentration prediction decoder to process the fused features, the bilirubin concentration prediction decoder includes a fully connected neural network and a regression layer, and outputs a bilirubin concentration prediction value through the regression layer, and its prediction formula is: In the formula, is the predicted value of bilirubin concentration, S seg is the feature vector of image segmentation result, F high is the feature mapping vector extracted by the shared encoder, C is the color feature vector, T is the texture feature vector, S is the shape feature vector, W1 and W2 are the weight matrices of the fully connected layers, b1 and b2 are the corresponding bias terms, ReLU() is the activation function, which means the rectified linear unit function, and f() is the linear activation function of the regression layer.

2. A bilirubin detection method based on image analysis according to claim 1, characterized in that: In step 1, the illumination intensity of the standardized light source is not less than 500 lux; during the acquisition process, the vertical error of the shooting angle relative to the forearm skin area does not exceed ±5 degrees, and the shooting time is controlled to be completed within 1 second.

3. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step 4, a self-attention mechanism is introduced in the skip connection between the shared encoder and the image segmentation decoder, and a multi-scale feature fusion module is introduced in the image segmentation decoder, including the following steps: Step A1, modify the U-Net model structure, insert a self-attention module in the skip connection between the shared encoder and the image segmentation decoder, and introduce a multi-scale feature fusion module in the image segmentation decoder; Step A2, inputting the preprocessed forearm skin image into a shared encoder, and extracting forearm skin color feature maps of multiple scales through the encoder; Step A3, applying a self-attention mechanism to the obtained forearm skin color feature map, calculating the bilirubin correlation between different positions on the feature map, generating an attention matrix, and using the attention matrix to weight the original feature map to generate an attention-weighted feature map reflecting the distribution of skin bilirubin; Step A4, performing element-wise multiplication operation on the generated attention weighted feature map and the original forearm skin color feature map to obtain a weighted bilirubin distribution feature map; Step A5, passing the obtained weighted bilirubin distribution feature map to the decoder module corresponding to the current encoder level in the U-Net model through a skip connection; Step A6, at each level of the decoder, collect the weighted feature map from the corresponding level of the encoder, the output feature map of the previous decoder, and the feature map of the current level; Step A7, adjusting the spatial scales of the feature maps with different resolutions collected in step A6 so that they have the same spatial dimension; Step A8, compressing the adjusted multi-scale feature map through a convolution operation to reduce the feature dimension; Step A9, using a channel attention mechanism to fuse the compressed multi-scale feature maps to generate a fused feature map; Step A10, inputting the fused feature map into the convolutional layer of the current decoder level for further processing and upsampling; Step A11, repeating steps A6 to A10 until all decoder levels are processed to generate a final bilirubin distribution area segmentation result.

4. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step 4, the image segmentation result includes a binary image of the segmented skin area, geometric features of the segmented area, location information of the segmented area, and a shape descriptor of the segmented area.

5. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step five, the extracted color features are expressed as color feature vectors, including calculating the color average, standard deviation and color histogram of the segmented area.

6. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step five, the extracted texture features are expressed as texture feature vectors, including using a gray-level co-occurrence matrix to calculate the image contrast, entropy value, uniformity of distribution of gray values ​​in the image, and linear correlation between pixel values ​​in the image.

7. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step five, the extracted shape features are expressed as shape feature vectors, including calculating the perimeter, area, shape moment and contour complexity of the segmented region.

8. The bilirubin detection method based on image analysis according to claim 1, characterized in that: In step 6, the joint loss function is used to optimize the image segmentation and bilirubin concentration prediction tasks, and the weighted sum of the segmentation loss and the concentration prediction loss is defined as: L total =α·L seg +β·L reg ; Where, L total is the total loss function of the multi-task learning model, L seg is the image segmentation loss, L reg is the bilirubin concentration prediction loss, α and β are the weight factors of segmentation loss and regression loss, respectively, and α+β=1.

Images