An eyelid morphological parameter measurement method based on a reconstruction-segmentation network of a connected architecture

Abstract

The application discloses a kind of based on the reconstruction-segmentation network of conjoined architecture eyelid morphological parameter measurement method, it is related to image processing field, including: obtaining ocular surface image, and respectively constructing ocular surface image classification dataset and ocular surface image segmentation dataset;Reconstruction-segmentation network of conjoined architecture based on dynamic parameter convolution is constructed;Reconstruction-segmentation network includes reconstruction task part and segmentation task part;Utilize the reconstruction task part of ocular surface image classification dataset training;The encoder of trained reconstruction task part is embedded as dynamic convolution module to segmentation task part, and reconstruction-segmentation network is trained using ocular surface image segmentation dataset, and the trained reconstruction-segmentation network is obtained;The ocular surface image to be detected is input to the trained reconstruction-segmentation network, and corneal region and palpebral fissure region are segmented out, and the measurement module is used to measure eyelid morphological parameter.The application improves the accuracy of image segmentation and eyelid morphological parameter detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network. Background Technology

[0002] Eyelid morphological parameters have extremely high clinical value in the diagnosis and evaluation of ocular surface diseases, such as the diagnosis of ophthalmic diseases and the evaluation of preoperative and postoperative effects of ophthalmic surgery.

[0003] However, traditional eyelid morphology measurement relies on doctors manually using measuring tools, a method susceptible to subjective factors and inefficient. With the rapid development of computer image processing and deep learning technologies, more and more research is exploring automated methods for measuring eyelid morphological parameters. For example, using ResNet50 or modified ResNet18 as backbones or encoders combined with networks like U-Net, or improving models like DeepLabV3, allows for the measurement of various eyelid morphological parameters such as medial canthus height, lacrimal meniscus height, and palpebral fissure length by segmenting different eye regions. However, most existing image processing and deep learning research improves segmentation accuracy by upgrading different parts of U-Net, often neglecting the effective construction of long-range dependency information. Furthermore, the datasets used are usually limited to segmentation datasets, restricting the generalization ability of the segmentation model and resulting in insufficient accuracy in ocular surface image segmentation.

[0004] Therefore, how to improve the accuracy of image segmentation, thereby improving the accuracy of eyelid morphological parameter detection, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network. An improved TransUNet model TB-Net is proposed, and a reconstruction-segmentation network STB-Net based on dynamic parameter convolution is implemented on the basis of the TB-Net model for image segmentation. Finally, the segmented corneal and palpebral fissure regions are input into the measurement module to automatically obtain the required eyelid morphological parameters.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network, the steps of which are as follows:

[0008] Obtain ocular surface images and construct ocular surface image classification dataset and ocular surface image segmentation dataset respectively;

[0009] A reconstruction-segmentation network based on a conjoined architecture with dynamic parameter convolution is constructed. The reconstruction-segmentation network includes a reconstruction task part and a segmentation task part, both of which are constructed based on TB-Net.

[0010] The reconstruction task part is trained unsupervised using the ocular surface image classification dataset; the encoder of the trained reconstruction task part is embedded as a dynamic convolution module into the segmentation task part, and the reconstruction-segmentation network is trained in a supervised manner using the ocular surface image segmentation dataset to obtain the trained reconstruction-segmentation network.

[0011] The ocular surface image to be detected is input into the trained reconstruction-segmentation network to segment the corneal region and palpebral fissure region, and the eyelid morphological parameters are measured using the measurement module.

[0012] Furthermore, the TB-Net includes an encoder and a decoder; the encoder includes a first bottleneck module, a second bottleneck module, a third bottleneck module, and 12 Transformer Layers connected in sequence.

[0013] The three-stage bottleneck module extracts features from the input image or feature map to obtain CNN feature maps of different resolutions.

[0014] The CNN feature map output by the third bottleneck module is converted into the feature format of Transformer through linear mapping, and the features are encoded through the 12 Transformer Layers to obtain the output feature map of the encoder.

[0015] Furthermore, the decoder includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and an output layer connected in sequence;

[0016] The output feature map of the encoder is converted into a feature format and used as the input feature of the decoder, which is then input into the first deconvolution layer.

[0017] The first deconvolutional layer, the second deconvolutional layer, and the third deconvolutional layer correspond one-to-one with the third bottleneck module, the second bottleneck module, and the first bottleneck module, respectively. The deconvolutional layer first performs an upsampling operation on the input feature map, then fuses the upsampled feature map with the CNN feature map output by the bottleneck module, then performs a convolution operation on the fused feature map, and finally outputs it after a non-linear activation operation of the ReLU activation function.

[0018] The output layer performs convolution and nonlinear activation operations on the feature map output by the third deconvolution layer before outputting it.

[0019] Furthermore, the decoder also includes a BLAM module, which includes a first BLAM layer, a second BLAM layer, a third BLAM layer, and a fourth BLAM layer connected in sequence; the first BLAM layer, the second BLAM layer, the third BLAM layer, and the fourth BLAM layer correspond one-to-one with the first deconvolution layer, the second deconvolution layer, the third deconvolution layer, and the output layer, respectively.

[0020] The deconvolutional layer utilizes a local channel attention mechanism to aggregate the input small-scale feature map and large-scale feature map and output a cross-layer fused feature map; the small-scale feature map is the input feature map of the decoder or the feature map output by the previous BLAM layer, and the large-scale feature map is the feature map output by the deconvolutional layer corresponding to the BLAM layer.

[0021] The outputs of all BLAM layers are concatenated through channels to output the final result.

[0022] Furthermore, the aggregation formula in the BLAM layer is as follows:

[0023] L(M)=σ(δ(B(PWConv2(δ(B(PWConv1(M))))))));

[0024]

[0025] Where L(M) represents the attention weight map; PWConv1 and PWConv2 represent the first and second pointwise convolutions, respectively; σ and δ represent the Sigmoid function and ReLU activation function, respectively; B represents the normalization operation; M, N, and Z represent the low-level features of the large-scale feature map, the high-level features of the small-scale feature map, and the cross-layer fusion features, respectively. This represents element-wise multiplication.

[0026] Furthermore, the dynamic parameter convolution process performed within the dynamic convolution module is specifically as follows:

[0027] The feature map output by the encoder in the reconstruction task is subjected to convolution, batch normalization and ReLU nonlinear combination operations, and then parameter deformation is performed to determine the parameters of the dynamic convolution kernel.

[0028] The feature map output by the encoder of the segmentation task is subjected to convolution, batch normalization and ReLU nonlinear combination operations, and used as the convolution object;

[0029] The convolutional object is convolved using the dynamic convolutional kernel, and the resulting feature map is then added to the feature map output by the encoder of the segmentation task part to obtain the final output feature map of the encoder of the segmentation task part.

[0030] Furthermore, the formula for the dynamic parameter convolution process is as follows:

[0031] DPConv(X,Y)=Conv2d(γX,ψ(γ(Y)));

[0032] Where γ represents the combination of convolution, batch normalization and ReLU nonlinearity, X and Y represent the feature maps output by the encoders for the segmentation task and reconstruction task, respectively, ψ represents parameter deformation, and Conv2d represents two-dimensional convolution.

[0033] Furthermore, the step of determining the parameters of the dynamic convolution kernel through parameter deformation specifically involves: performing parameter deformation ψ based on the feature map output by the encoder of the reconstruction task to obtain a one-dimensional vector θ of size n. n According to θ n The n parameters respectively determine the expected number of target channels, the size of the convolution kernel, and the number of output channels of the dynamic convolution kernel;

[0034] The semantic transformation first uses a 1×1 convolution. Adaptive average pooling P is used to adjust the channel dimensions, followed by another 1×1 convolution. Reshape the features into θ n And change the channel from z to n, the specific formula is:

[0035]

[0036] Furthermore, the eyelid morphology parameters are divided into three categories: palpebral fissure height, palpebral fissure width, and palpebral fissure area. The palpebral fissure height includes the left palpebral fissure height, the central palpebral fissure height, and the right palpebral fissure height.

[0037] As can be seen from the above technical solution, compared with the prior art, this invention discloses a method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network. It proposes an improved TransUNet model, TB-Net, and constructs a reconstruction-segmentation network STB-Net based on dynamic parameter convolution using a conjoined architecture. This network is used to accurately segment the cornea and palpebral fissure regions in ocular surface images. Finally, based on the obtained segmentation results, the corresponding left palpebral fissure height, central palpebral fissure height, right palpebral fissure height, palpebral fissure width, and palpebral fissure area are measured. The TB-Net model improves the TransUNet decoder by introducing a bottom-up Local Attention Modulation (BLAM) module, effectively solving the problem of missing fine and edge local information in ocular surface image segmentation and improving segmentation accuracy. The STB-Net model adopts the SRSNetwork architecture, combining reconstruction and segmentation tasks, and uses dynamic parameter convolution to generate adaptive convolution kernels, thereby optimizing the segmentation effect on limited labeled data. This invention uses a conjoined architecture reconstruction-segmentation network based on dynamic parameter convolution to segment ocular surface images, improving image segmentation accuracy and further enhancing the accuracy of eyelid morphological parameter detection. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the BLAM structure according to an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the TB-Net structure according to an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram illustrating the principle of the dynamic convolution module in an embodiment of the present invention.

[0043] Figure 5(a) is a schematic diagram of the original image of an embodiment of the present invention.

[0044] Figure 5(b) is a schematic diagram of the palpebral fissure label according to an embodiment of the present invention.

[0045] Figure 5(c) is a schematic diagram of the corneal tag according to an embodiment of the present invention.

[0046] Figure 6This is a test result diagram of the reconstruction task according to an embodiment of the present invention.

[0047] Figure 7 This is a comparison diagram of the corneal segmentation results of various models in the embodiments of the present invention.

[0048] Figure 8 This is a comparison diagram of the palpebral fissure segmentation results of various models in the embodiments of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This invention discloses a method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network, such as... Figure 1 As shown, the steps are as follows:

[0051] Obtain ocular surface images and construct ocular surface image classification dataset and ocular surface image segmentation dataset respectively;

[0052] A reconstruction-segmentation network based on a conjoined architecture with dynamic parameter convolution is constructed. The reconstruction-segmentation network consists of a reconstruction task part and a segmentation task part, both of which are constructed based on TB-Net.

[0053] The reconstruction task part is trained unsupervised using an ocular surface image classification dataset; the encoder of the trained reconstruction task part is embedded as a dynamic convolutional module into the segmentation task part, and the reconstruction-segmentation network is trained in a supervised manner using an ocular surface image segmentation dataset to obtain the trained reconstruction-segmentation network.

[0054] The ocular surface image to be detected is input into the trained reconstruction-segmentation network to segment the corneal region and palpebral fissure region, and the eyelid morphological parameters are measured using the measurement module.

[0055] In a specific embodiment, such as Figure 3 As shown, TB-Net includes encoder and decoder sections; the encoder includes a first bottleneck module, a second bottleneck module, a third bottleneck module, and 12 Transformer Layers connected in sequence.

[0056] The three-stage bottleneck module extracts features from the input image or feature map to obtain CNN feature maps of different resolutions.

[0057] The CNN feature map output by the third bottleneck module is converted into the feature format of Transformer by linear mapping, and the features are encoded by 12 Transformer Layers to obtain the output feature map of the encoder.

[0058] In one specific embodiment, the decoder includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and an output layer connected in sequence;

[0059] The encoder's output feature map is converted into a feature format and used as the input feature of the decoder, which is then input into the first deconvolution layer.

[0060] The first, second, and third deconvolutional layers correspond one-to-one with the third, second, and first bottleneck modules, respectively. The deconvolutional layer first upsamples the input feature map, then fuses the upsampled feature map with the CNN feature map output by the bottleneck module, then performs a convolution operation on the fused feature map, and finally outputs the result after a non-linear activation operation of the ReLU activation function.

[0061] The output layer performs convolution and non-linear activation operations on the feature map output by the third deconvolution layer before outputting the result.

[0062] Specifically, in the field of medical image segmentation, the U-Net network is a classic and widely used deep learning model. The U-Net network is a typical encoder-decoder structure, forming a U-shape. With subsequent research, the U-Net network has undergone several development stages, resulting in different variants and derivatives. U-Net++ is a typical improvement, which increases the model's feature representation and detail recovery capabilities by introducing multi-layered network structures in skip connections, further improving segmentation accuracy. In recent years, variants of U-Net have also incorporated Transformers, introducing a self-attention mechanism at the encoder level to improve the model's shortcomings in constructing long-range dependency information in traditional convolutional neural networks. The most representative example is TransUNet. However, in the feature transfer of the TransUNet decoder itself, it does not consider fusing high-level semantic information with low-level semantic information, leading to local information limitations. Furthermore, during feature extraction, most convolutional networks learn high-level semantic features by gradually decaying the feature map size, causing local information to be easily overwhelmed by surrounding background features in deeper layers, which is not conducive to the local reproduction of the segmented target's edges.

[0063] Therefore, this invention utilizes a bottom-up local attentional modulation (BLAM) module to integrate small-scale subtle features from low-level features into deeper, high-level features. Low-level features are upsampled and then interact with high-level features within the BLAM, achieving dynamic weighting of local information attention from low-level features to high-level features. Through layer-by-layer enhancement using multiple BLAM modules, TB-Net achieves efficient feature progressive optimization in the decoder, enabling features at different levels to exhibit stronger targeting and local attention capabilities during fusion. Each BLAM module dynamically adjusts the response of high-level semantic features to low-level detail information, gradually reducing the deep network's over-reliance on background information, thereby highlighting the salience of the target region. This layer-by-layer enhancement mechanism ensures that the segmentation network can effectively capture the local features of the target while preserving spatial details, improving the robustness and accuracy of segmentation in complex scenes. Applying BLAM to the decoder module in the segmentation part effectively enhances the decoder's attention to subtle and edge-related local information, helping to more accurately recover the boundaries of complex structures in medical image segmentation and improving the segmentation accuracy of small lesions and blurred boundaries.

[0064] In one specific embodiment, the decoder further includes a BLAM module, which includes a first BLAM layer, a second BLAM layer, a third BLAM layer, and a fourth BLAM layer connected in sequence; the first BLAM layer, the second BLAM layer, the third BLAM layer, and the fourth BLAM layer correspond one-to-one with the first deconvolution layer, the second deconvolution layer, the third deconvolution layer, and the output layer, respectively.

[0065] The deconvolutional layer utilizes a local channel attention mechanism to aggregate the small-scale and large-scale feature maps of the input and output a cross-layer fused feature map. The small-scale feature map is the input feature map of the decoder or the feature map output by the previous BLAM layer, and the large-scale feature map is the feature map output by the deconvolutional layer corresponding to the BLAM layer.

[0066] The outputs of all BLAM layers are concatenated through channels to output the final result.

[0067] In one specific embodiment, the aggregation formula in the BLAM layer is:

[0068] L(M)=σ(δ(B(PWConv2(δ(B(PWConv1(M))))))));

[0069]

[0070] Where L(M) represents the attention weight map; PWConv1 and PWConv2 represent the first and second pointwise convolutions, respectively; σ and δ represent the Sigmoid function and ReLU activation function, respectively; B represents the normalization operation; M, N, and Z represent the low-level features of the large-scale feature map, the high-level features of the small-scale feature map, and the cross-layer fusion features, respectively. This represents element-wise multiplication.

[0071] Specifically, such as Figure 2 As shown (Point-wise Conv in the figure represents point-wise convolution), the local channel attention mechanism L aggregates the channel context features at each spatial location. The kernel sizes of PWConv1 and PWConv2 are C / 4×C×1×1 and C×C / 4×1×1, respectively, similar to a bottleneck structure. Attention weight map L(X)∈R C×H×W Having the same shape as the input feature map, subtle details can be highlighted element-wise (spatially and across channels). The BLAM module is motivated by embedding small-scale details into a high-level coarse feature map, achieved through dynamic weighted modulation of high-level features guided by low-level features. Given X as a low-level feature and Y as a high-level feature, a cross-layer fused feature Z∈R can be obtained through a bottom-up local attention modulation module. C×H×W R C×H×W This represents a three-dimensional real tensor, where C, H, and W represent the number of channels, height, and width, respectively.

[0072] In a specific embodiment, the dynamic parameter convolution process performed within the dynamic convolution module is as follows:

[0073] The feature map output by the encoder in the reconstruction task is subjected to convolution, batch normalization and ReLU nonlinear combination operations, and then parameter deformation is performed to determine the parameters of the dynamic convolution kernel.

[0074] The feature map output by the encoder in the segmentation task is subjected to convolution, batch normalization, and ReLU nonlinear combination operations, and used as the convolution object.

[0075] The convolution operation is performed on the convolutional object using a dynamic convolution kernel. The resulting feature map is then added to the feature map output by the encoder of the segmentation task part, which is used as the final output feature map of the encoder of the segmentation task part.

[0076] Specifically, this invention uses TB-Net for both reconstruction and segmentation tasks, forming a new model named STB-Net. STB-Net leverages the unsupervised reconstruction task to learn high-level semantic information from similar medical image datasets without segmentation labels, providing more reliable dynamically parameterized convolutional kernels for the segmentation task, thereby improving the performance and accuracy of the segmentation model. The reconstruction and segmentation tasks are linked, forming a symmetrical conjoined architecture. The connection between the reconstruction and segmentation tasks is implemented through the dynamic convolutional module DPConv. The working principle of DPConv is as follows... Figure 4 As shown, where E r E s D s These represent the encoder of the reconstruction model, the encoder of the segmentation model, and the decoder of the segmentation model, respectively. The semantic features output from the upper part are transformed into convolutional kernels through parameter deformation, while the feature maps generated in the lower part serve as the convolution object. The two are convolved, and the output feature map is then added to the original feature map and passed to the decoder. This is the core of the entire network's operation. It's important to note that DPConv is only used at the end of the encoder to generate additional new feature outputs, which are then added to the original encoder output. It does not change the number of channels in the feature transmission process. This method, similar to residual connections, ensures that the features originally transmitted by the segmentation model remain effective and are further enhanced after the incorporation of new features.

[0077] In a specific embodiment, the formula for the dynamic parameter convolution process is:

[0078] DPConv(X,Y)=Conv2d(γX,ψ(γ(Y)));

[0079] Where γ represents the combination of convolution, batch normalization and ReLU nonlinearity, X and Y represent the feature maps output by the encoders for the segmentation task and reconstruction task, respectively, ψ represents parameter deformation, and Conv2d represents two-dimensional convolution.

[0080] In a specific embodiment, parameter deformation is performed to determine the parameters of the dynamic convolution kernel. Specifically, based on the feature map output by the encoder of the reconstruction task, parameter deformation ψ is performed to obtain a one-dimensional vector θ of size n. n According to θ n The n parameters determine the expected number of target channels, the size of the convolution kernel, and the number of output channels of the dynamic convolution kernel, respectively.

[0081] Semantic transformation begins with a 1×1 convolution. Adaptive average pooling P is used to adjust the channel dimensions, followed by another 1×1 convolution. Reshape the features into θ n And change the channel from z to n, the specific formula is:

[0082]

[0083] Specifically, in order to perform the DPConv operation, the convolution kernel must have specific parameters: the desired number of target channels is denoted by M, the kernel size by K, and the output channels by N. The total number of parameters to be generated can be expressed as n = M × N × K. 2 Therefore, in order to generate n parameters, we choose to construct a one-dimensional vector θ of size n. n In order to obtain θ n Semantic features Y∈R z ＝C×H×W It needs to be converted into a one-dimensional vector θ n However, if we directly use average pooling to transform Y into θ... n This will lead to network non-convergence.

[0084] In one specific embodiment, the eyelid morphological parameters are divided into three categories, including palpebral fissure height, palpebral fissure width, and palpebral fissure area. The palpebral fissure height includes the left palpebral fissure height, the central palpebral fissure height, and the right palpebral fissure height.

[0085] Specifically, the final step of this invention is to measure three types of eyelid morphological parameters based on the corneal segmentation results and palpebral fissure segmentation results obtained from the segmentation algorithm described above. In this embodiment, the resolution of the ocular surface image is 2974×1984. After professional ophthalmologists measured the height and width of all ocular surface images in the experiment, the average width was found to be 14.65cm and the average height 9.77cm. The image obtained by the instrument used to capture the ocular surface image is four times the actual size, resulting in an average actual width of 3.6625cm and an average height of 2.4425cm. Therefore, in the original image, each pixel has a width of 0.012306mm and a height of 0.012310mm. Due to the limitations of the Transformer layer, there are restrictions on the size of the input image. An excessively large input size can lead to insufficient GPU memory during training, while an excessively small input size can result in lower segmentation accuracy. Therefore, the input image size is reduced to 384×256, resulting in a pixel width of 0.09537mm and a height of 0.09541mm. For each ocular surface image in the test set, the left, right, and center coordinates of the corneal region in the corneal segmentation map are recorded. The number of target pixels corresponding to these three coordinates in the palpebral fissure segmentation map is then found and multiplied by the corresponding pixel height to obtain the left, central, and right palpebral fissure heights. The palpebral fissure width is calculated by multiplying the difference between the leftmost and rightmost coordinates in the palpebral fissure segmentation map by the pixel width. The palpebral fissure area is obtained by directly calculating the total number of target pixels and multiplying by the area of ​​a single pixel.

[0086] In a specific embodiment, a concrete example is used to illustrate the process of image dataset acquisition and processing, model training and evaluation, and result assessment and analysis. Specifically:

[0087] Because this invention's algorithm includes reconstruction and segmentation tasks, the datasets involved are of two types: ocular surface image classification datasets and ocular surface image segmentation datasets. These two types of images were captured using different instruments, resulting in differences in their actual size and image resolution. However, the ocular surface image classification dataset, being an ocular surface image dataset, has high reuse value and can therefore be used together with the ocular surface image segmentation dataset of this invention for the reconstruction task. Furthermore, STB-Net comprises both reconstruction and segmentation parts. Reconstruction is an unsupervised learning process, allowing the use of non-matching data—the classification dataset—for unsupervised network training to extract features from similar data (ocular surface image data). The encoder obtained from the reconstruction task generates adaptive dynamic convolution parameters for the segmentation network, indirectly compensating for the accuracy deficiencies caused by insufficient segmentation datasets. Regarding the similar datasets, the experiment used existing ocular surface classification datasets, but without utilizing their classification labels.

[0088] First, the ocular surface image segmentation dataset was provided by Shenzhen Eye Hospital, containing 250 ocular surface images with corneal and palpebral fissure labels, at a resolution of 2976×1984. All images were captured under the same conditions. Second, the ocular surface image classification dataset was provided by the Ocular Surface Disease Center of Nanjing Medical University Affiliated Eye Hospital, containing 2855 ocular surface disease images, also at a resolution of 2976×1984, categorized into three main types: normal ocular surface, ocular surface hemorrhage, and pterygium. All images have been anonymized and do not contain any patient privacy information.

[0089] In the reconstruction task, both the ocular surface image classification dataset and the measurement dataset were used together, totaling 3105 ocular surface images. Since the reconstruction task is used to assist the segmentation task, no testing or validation was required. Therefore, the dataset was divided into training and validation sets in a 9:1 ratio, resulting in a training set of 2795 images and a validation set of 310 images. In the segmentation task, the ocular surface image segmentation dataset was divided into training, validation, and test sets in a 7:1:2 ratio, following the same stratified sampling principle, resulting in a training set of 174 images, a validation set of 25 images, and a test set of 51 images.

[0090] Secondly, since the width and height of the image are not the same, and the optimal input image size for the algorithm of this invention is 384×384, directly stretching the image to the same scale to meet the network's input requirements does not conform to the original proportions of the image and can easily lead to distortion of the target shape, thus affecting the network's understanding of the image and segmentation accuracy. Therefore, a proportional scaling method is adopted, scaling the image to the required scale along the long side and padding the short side with zeros to the same size. This preserves the original proportions of the image and avoids information loss caused by deformation. The data augmentation methods used include random size stretching, random horizontal flipping, and random vertical flipping. The segmentation labels of this invention are all completed under the guidance of doctors, using LabelMe annotation software. The preprocessing method for the labels is the same as that for the original images. The ocular surface images and corresponding labels are shown in Figures 5(a), 5(b), and 5(c).

[0091] In the specific training process, the reconstruction task trains TB-Net. Since it is necessary to simultaneously restore the RGB channels of the image and further evaluate the difference between the model's predictions and the true values, Mean Squared Error (MSE) is used as the loss function. The specific formula is:

[0092]

[0093] Among them, N is the total number of pixels in the three channels, y i It is the true value of the i-th sample. It is the predicted value of the i-th sample.

[0094] In addition, stochastic gradient descent was used as the optimizer, with impulse set to 0.9, weight decay set to 0.0001, learning rate set to 0.001, batch size set to 4, and number of iterations set to 200.

[0095] The segmentation task trains STB-Net, using a weighted sum of BCE Loss and Dice Loss as the loss function, calculated as follows:

[0096]

[0097] BCEDiceLoss=αDiceLoss+βBCELoss;

[0098] Where N represents the total number of pixels, y i It is the true value of the i-th sample. is the predicted value of the i-th sample, and α and β represent the weights of the two loss functions, both set to 0.5.

[0099] Furthermore, Adam was used as the optimizer, with weight decay set to 0.0003, learning rate set to 0.0001, batch size set to 8, iterations to 300, and a cosine annealing learning rate adjustment strategy employed. This invention was implemented using the PyTorch deep learning framework, with the following hardware configuration: Intel i7-14700KF 5.6GHz, NVIDIA RTX4090 24GB. Software configuration: Windows 11, Python 3.8, PyTorch 2.3.1, OpenCV 4.5.5.

[0100] To measure the segmentation performance of the model, the Dice coefficient, Intersection over Union (IoU), and Global Accuracy (GA) are used as evaluation metrics. The calculation formulas are shown below:

[0101]

[0102] In the formula, y represents the label pixel matrix. This represents the predicted pixel matrix, where TP represents the number of pixels correctly predicted as the target, TN represents the number of pixels correctly predicted as the background, FP represents the number of pixels incorrectly predicted as the target, and FN represents the number of pixels incorrectly predicted as the background. The Dice coefficient describes the similarity between the predicted and actual results; the higher the similarity, the closer the value is to 1. IoU measures the overlap between the predicted and actual results; similarly, the closer the ratio is to 1, the higher the overlap. GA represents the proportion of correctly predicted pixel values ​​among all pixels, and is an important indicator for evaluating the model's ability to distinguish between the target and the background.

[0103] This invention also employs two methods—linear regression analysis and Bland-Altman consistency analysis—to quantitatively evaluate the measurement results. In linear regression analysis, a regression line is fitted and the correlation coefficient r is calculated. 2 Value, the relationship and goodness of fit between the measured value and the actual value, r 2 The closer the value is to 1, the higher the degree of agreement between the system's predicted value and the actual value, indicating better measurement accuracy. Furthermore, Bland-Altman consistency analysis is used to further evaluate the consistency between the system and physician measurement results, analyzing the error distribution and the limits of agreement (LoA) between the two, visually revealing their biases and reliability. These two analytical methods complement each other; linear regression focuses on the correlation and accuracy of the results, while consistency analysis focuses on the stability and consistency of the measurement results in practical applications, ensuring the reliability and effectiveness of the system in clinical use.

[0104] This invention uses the constructed TB-Net for reconstruction tasks. The optimal reconstruction model obtained through training has an average mean square error of 0.00311 on the validation set. Some test results are shown in the figure below. Figure 6 As shown, the reconstruction task generally demonstrates good image restoration performance. Although some areas of the image show spots and voids in certain reflective regions, the overall reconstruction of the entire ocular surface is quite accurate. In particular, it exhibits high quality in the restoration of texture in the corneal region and blood vessels and colors in the palpebral fissure region. This indicates that TB-Net has learned the features of ocular surface images quite well in the reconstruction task, laying a solid foundation for subsequent analysis and applications.

[0105] In the segmentation task of this invention, in addition to comparing and analyzing TransUNet, TB-Net, and STB-Net, two classic segmentation models, U-Net and U-Net++, were also selected for comparison. All models were tested on the same dataset and trained multiple times to reach their optimal performance, thus measuring the effectiveness of the proposed method. As shown in Table 1, for the segmentation of the palpebral fissure region, the proposed model STB-Net outperformed the other models in all metrics, with Dice, GA, and IoU of 0.9875, 0.9955, and 0.9767, respectively. TB-Net showed improvements in all three metrics compared to TransUNet, with Dice improved by 0.08%, GA by 0.03%, and IoU by 0.15%, indicating that introducing BLAM into the TransUNet decoder is effective. STB-Net improved Dice by 0.06%, GA by 0.02%, and IoU by 0.12% compared to TB-Net, also demonstrating the effectiveness of using a conjoined architecture based on TB-Net.

[0106] Table 1 Evaluation Indicators for Palpebral fissure segmentation

[0107]

[0108]

[0109] Table 2 Corneal Segmentation Evaluation Indicators

[0110]

[0111] Similarly, as shown in Table 2, for corneal region segmentation, the proposed model STB-Net also achieved the highest scores in all three metrics. GA (Gross GA) was tied for the highest with U-Net++ and TB-Net at 0.9978, Dice was 0.9891, and IoU was 0.9790. Compared to TransUNet, TB-Net showed improvements in Dice, GA, and IoU. Compared to TB-Net, STB-Net maintained the same GA but further improved Dice and IoU. Therefore, STB-Net can effectively segment the palpebral fissure and corneal regions on ocular surface images.

[0112] In addition to using evaluation metrics to measure the model's segmentation performance on ocular surface images, this invention also visually demonstrates the advantages of the proposed model by showcasing the segmentation results of each model on ocular surface images. For example... Figure 8 As shown, it can be observed that STB-Net can segment the palpebral fissure region more accurately. Firstly, regarding the segmentation details of the palpebral fissure angle, other models exhibit varying degrees of loss or breakage. For example, in the first set of segmentation results, STB-Net maximizes the restoration of details at the palpebral fissure angle, while other models either do not consider it part of the palpebral fissure or show incomplete segmentation. Secondly, regarding the segmentation details of polyps on the upper and lower eyelids, other models misclassify polyps. For example, in the third set of segmentation results, STB-Net can distinguish polyps, treating them as background, while other models treat polyps as part of the palpebral fissure. Similarly, as... Figure 7 As shown, STB-Net is also more accurate in corneal segmentation. For example, in the first set of segmentation results, STB-Net segmented the cornea very well, while other models showed gaps due to the influence of polyps. In the second set of segmentation results, STB-Net was not affected by eyelashes, while other models were affected and treated areas not belonging to the cornea as targets.

[0113] In summary, this invention fully validates the advantages of STB-Net in ocular surface image segmentation tasks through quantitative evaluation metrics and intuitive comparison of segmentation results. STB-Net not only outperforms other models in segmentation accuracy for the palpebral fissure and cornea, but also demonstrates superior detail capture and anti-interference capabilities. Specifically, in the segmentation of complex structures (such as the palpebral fissure angle, upper and lower eyelid polyps, and eyelash regions), STB-Net accurately restores details, avoids common detail loss or misjudgment, effectively resists background interference, and ensures the integrity and accuracy of the segmentation results.

[0114] Furthermore, the segmentation results of STB-Net were measured and the relative errors were calculated by comparing them with the actual measured values. Specifically, the relative error for left palpebral fissure height was 2%, for central palpebral fissure height was 0.78%, for right palpebral fissure height was 1.93%, for palpebral fissure width was 1.31%, and for palpebral fissure area was 0.98%. Measurement values ​​for some samples are shown in Tables 3 and 4, where LH, MH, RH, PW, and PA represent left palpebral fissure height, central palpebral fissure height, right palpebral fissure height, palpebral fissure width, and palpebral fissure area, respectively. This further verifies that STB-Net can accurately segment key regions and achieve precise measurements in most samples.

[0115] Table 3 shows the actual measurement values ​​of some samples.

[0116]

[0117] Table 4 shows the STB-Net segmentation measurements for some samples.

[0118]

[0119]

[0120] This invention employs the Bland-Altman analysis method to assess the consistency between measured and true values ​​of various eyelid morphological parameters. In the analysis, the average value of the two sets of data is used as the horizontal axis, and the difference as the vertical axis, generating a Bland-Altman test chart. Statistically, if the error falls within the 95% consistency interval, it is considered within an acceptable range. The results show that three sample points on the left side exceeded the consistency interval for palpebral fissure height, and three for the central palpebral fissure height; one on the right side exceeded the consistency interval for palpebral fissure height; four exceeded the consistency interval for palpebral fissure width; and three exceeded the consistency interval for palpebral fissure area. This indicates that the eyelid morphological parameter measurement method proposed in this study has good agreement with the true values, demonstrating its potential value in practical applications.

[0121] In summary, this invention proposes an improved TransUNet model, TB-Net, and based on this, constructs a reconstruction-segmentation network, STB-Net, with a connected architecture based on dynamic parameter convolution. This network is used to accurately segment the cornea and palpebral fissure regions in ocular surface images, and measures the corresponding left palpebral fissure height, central palpebral fissure height, right palpebral fissure height, palpebral fissure width, and palpebral fissure area based on the segmentation results. The TB-Net model improves upon the TransUNet decoder by introducing a bottom-up Local Attention Modulation (BLAM) module, effectively addressing the lack of fine and edge local information in ocular surface image segmentation and improving segmentation accuracy. The STB-Net model employs the SRSNetwork architecture, combining reconstruction and segmentation tasks, and utilizes dynamic parameter convolution to generate adaptive convolution kernels, thereby optimizing segmentation performance on limited labeled data.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0123] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network, characterized in that, The steps are as follows: Obtain ocular surface images and construct ocular surface image classification dataset and ocular surface image segmentation dataset respectively; A reconstruction-segmentation network based on a conjoined architecture with dynamic parameter convolution is constructed. The reconstruction-segmentation network includes a reconstruction task part and a segmentation task part, both of which are constructed based on TB-Net. The reconstruction task portion is trained and learned unsupervised using the ocular surface image classification dataset. The encoder of the reconstruction task part, which has been trained, is embedded into the segmentation task part as a dynamic convolution module, and the reconstruction-segmentation network is trained and learned in a supervised manner using the ocular surface image segmentation dataset to obtain the trained reconstruction-segmentation network. The ocular surface image to be detected is input into the trained reconstruction-segmentation network to segment the corneal region and palpebral fissure region, and the eyelid morphological parameters are measured using the measurement module. The TB-Net includes an encoder and a decoder; the encoder includes a first bottleneck module, a second bottleneck module, a third bottleneck module, and 12 Transformer Layers connected in sequence. The three-stage bottleneck module extracts features from the input image or feature map to obtain CNN feature maps of different resolutions. The CNN feature map output by the third bottleneck module is converted into the feature format of Transformer through linear mapping, and the features are encoded through the 12 Transformer Layers to obtain the output feature map of the encoder. The decoder includes a first deconvolution layer, a second deconvolution layer, a third deconvolution layer, and an output layer connected in sequence; The output feature map of the encoder is converted into a feature format and used as the input feature of the decoder, which is then input into the first deconvolution layer. The first deconvolutional layer, the second deconvolutional layer, and the third deconvolutional layer correspond one-to-one with the third bottleneck module, the second bottleneck module, and the first bottleneck module, respectively. The deconvolutional layer first performs an upsampling operation on the input feature map, then fuses the upsampled feature map with the CNN feature map output by the bottleneck module, then performs a convolution operation on the fused feature map, and finally outputs it after a non-linear activation operation of the ReLU activation function. The output layer performs convolution and non-linear activation operations on the feature map output by the third deconvolution layer before outputting it. The decoder also includes a BLAM module, which includes a first BLAM layer, a second BLAM layer, a third BLAM layer, and a fourth BLAM layer connected in sequence; the first BLAM layer, the second BLAM layer, the third BLAM layer, and the fourth BLAM layer correspond one-to-one with the first deconvolution layer, the second deconvolution layer, the third deconvolution layer, and the output layer, respectively. The deconvolutional layer utilizes a local channel attention mechanism to aggregate the input small-scale feature map and large-scale feature map and output a cross-layer fused feature map; the small-scale feature map is the input feature map of the decoder or the feature map output by the previous BLAM layer, and the large-scale feature map is the feature map output by the deconvolutional layer corresponding to the BLAM layer. The outputs of all BLAM layers are concatenated through channels to output the final result.

2. The method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network according to claim 1, characterized in that, The aggregation formula in the BLAM layer is: ； ； in, Represents the attention weights graph; and These represent the first pointwise convolution and the second pointwise convolution, respectively. , These represent the Sigmoid function and the ReLU activation function, respectively. This indicates a normalization operation; , , These represent low-level features of large-scale feature maps, high-level features of small-scale feature maps, and cross-layer fusion features, respectively. This represents element-wise multiplication.

3. The method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network according to claim 1, characterized in that, The dynamic parameter convolution process performed within the dynamic convolution module is as follows: The feature map output by the encoder in the reconstruction task is subjected to convolution, batch normalization and ReLU nonlinear combination operations, and then parameter deformation is performed to determine the parameters of the dynamic convolution kernel. The feature map output by the encoder of the segmentation task is subjected to convolution, batch normalization and ReLU nonlinear combination operations, and used as the convolution object; The convolutional object is convolved using the dynamic convolutional kernel, and the resulting feature map is then added to the feature map output by the encoder of the segmentation task part to obtain the final output feature map of the encoder of the segmentation task part.

4. The method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network according to claim 3, characterized in that, The formula for the dynamic parameter convolution process is: ； in, This represents a combination of convolution, batch normalization, and ReLU nonlinearity. X , Y These represent the feature maps output by the encoders for the segmentation and reconstruction tasks, respectively. Indicates parameter deformation, This represents a two-dimensional convolution.

5. The method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network according to claim 3, characterized in that, The parameter deformation to determine the parameters of the dynamic convolution kernel specifically involves: performing parameter deformation based on the feature map output by the encoder of the reconstruction task. , to obtain a size of n a one-dimensional vector ;according to of n The parameters determine the desired number of target channels, the size of the convolution kernel, and the number of output channels of the dynamic convolution kernel, respectively. The parameter deformation first uses a 1×1 convolution. and adaptive average pooling P Adjust the channel dimensions, then use another 1×1 convolution. Reshape features to and the channel from z Change to n， The specific formula is as follows: 。 6. The method for measuring eyelid morphological parameters based on a conjoined architecture reconstruction-segmentation network according to claim 1, characterized in that, The eyelid morphology parameters are divided into three categories: palpebral fissure height, palpebral fissure width, and palpebral fissure area. The palpebral fissure height includes the left palpebral fissure height, the central palpebral fissure height, and the right palpebral fissure height.

Images