A method for eyelid margin feature classification

Abstract

The present invention relates to the field of image processing technology, and in particular to a method for eyelid margin feature classification, comprising: inputting an original feature map into an eyelid margin segmentation model to obtain a segmented image; dividing the segmented image into windows to obtain initialization position and size information corresponding to each window, and then inputting the segmented image into a preliminary feature extraction model to obtain an initial image; re-windowing the segmented image based on the initialization position and size information corresponding to each window and the initial image; inputting the segmented image into a pre-set feature classification model, which obtains a feature image based on the segmented image and image block size, as well as a first target dimension; and inputting the feature image into a classification head to obtain a feature classification result. The method has the beneficial effect of utilizing segmentation information to assist feature classification and optimizing it based on the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an eyelid margin feature classification method. Background Art

[0002] The eyelid margin is the junction between the eyelid skin and the conjunctiva. Maintaining a healthy eyelid margin is crucial for overall ocular surface health. Blepharitis is one of the most common eye diseases, affecting approximately 47% of ophthalmology patients. Blepharitis often leads to meibomian gland dysfunction, dry eye, and blepharokeratoconjunctivitis (BKC). BKC can cause astigmatism, corneal lesions, amblyopia, vision loss, or corneal perforation. Therefore, early diagnosis and treatment of blepharitis are crucial to preventing vision loss. According to the American Academy of Ophthalmology classification, blepharitis is categorized by anatomical location as anterior blepharitis and posterior blepharitis. Anterior blepharitis affects the eyelid skin, eyelash base, and eyelash follicles, while posterior blepharitis affects the meibomian glands. The primary clinical feature of blepharitis is redness of the eyelid margin, while the hallmark of posterior blepharitis is obstruction of the meibomian gland openings.

[0003] In recent years, deep learning has shown significant potential in medical image segmentation and classification. One study proposed an automated and objective method for assessing eyelid morphology, while another used deep learning to analyze eyelid morphology in patients with thyroid-related eye disease, demonstrating strong agreement between automated and manual measurements.

[0004] However, these methods all regard segmentation and classification as independent tasks and fail to fully utilize segmentation information to assist feature classification, resulting in limited accuracy of classification results. Summary of the Invention

[0005] (1) Technical issues to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an eyelid margin feature classification method, which solves the technical problem that the prior art regards segmentation and classification as independent tasks, fails to fully utilize segmentation information to assist feature classification, and results in limited accuracy of classification results.

[0007] (2) Technical solution

[0008] In order to achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for classifying eyelid margin features, comprising:

[0010] S1. Inputting the original feature map into a preset eyelid margin segmentation model to obtain a segmented image corresponding to the original feature map, wherein the segmented image is an image in which the eyelid margin area is extracted;

[0011] S2. Inputting the segmented image into a preset preliminary feature extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image;

[0012] S3, dividing the segmented image into windows according to a preset first window size, and obtaining initialization position and size information corresponding to each window;

[0013] Based on the initialization position and size information corresponding to each window, the initial image is globally pooled to generate a global feature description for each window;

[0014] According to the pre-set self-attention layer and the global description of each window, the feature weight of each window is obtained;

[0015] Obtaining a feature importance map corresponding to the segmented image according to the feature weight of each window, wherein the feature importance map includes the relative importance of each window area;

[0016] Adjust the window size in the feature importance map according to a preset importance threshold, and re-window the segmented image based on the window size in the adjusted feature importance map;

[0017] S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image;

[0018] During this period, a feature image is obtained according to the segmented image after the window is re-divided and the pre-set first target dimension;

[0019] S5, inputting the feature image into a preset classification head to obtain a feature classification result;

[0020] The eyelid margin segmentation model is based on the U2-Net model, including an encoder and a decoder, and the encoder and the decoder are connected by a jump connection.

[0021] Optionally, the S4 includes:

[0022] S4-1. Dividing the segmented image into at least one image block according to the window position and window size of each window in the newly windowed segmented image, and adjusting the size of all the image blocks based on a preset adaptive pooling layer so that the size of each image block is the same;

[0023] S4-2. After flattening any image block into a vector based on the local features of the image block, map the image block to the first target dimension using a preset linear transformation formula and a preset first target dimension; the linear transformation formula is:

[0024] E=Wpatch ·patch;

[0025] Among them, W patch is the preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block;

[0026] S4-3, after all image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained according to a preset in-window self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block;

[0027] At this time, the S5 includes: inputting the first feature image into a preset classification head to obtain a feature classification result.

[0028] Optionally, after S4-3 and before S5, the following steps are further included:

[0029] S4-4, according to the preset downsampling multiple N, merge the image blocks in the first feature image to obtain the N downsampling multiple of the first feature image. 2 A first process graph, N is a non-zero even number;

[0030] S4-5, after concatenating all the first process graphs according to the preset second target dimension, input the pre-set 1*1 convolution layer to obtain the second process graph;

[0031] S4-6, according to the preset window self-attention mechanism and the second process map, obtain a second feature image; the size of the second feature image is 1 / 2 of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image;

[0032] The self-attention within the window is:

[0033]

[0034] Where Q = W Q x, K = W K x,V=W V x, Q is the query vector, K is the key vector, V is the value vector, W Q 、W K and W V are all pre-set weight matrices, is a preset scaling factor, and T is a preset smoothness constant;

[0035] At this time, the S5 includes: inputting the second feature image into a preset classification head to obtain a feature classification result.

[0036] Optionally, after S4-6 and before S5, the following steps are further included:

[0037] S4-7, jump to S4-4, and replace the first feature image with the latest feature image, repeating the preset number of times;

[0038] S4-8. Obtain M feature images of different scales, and align the sizes of all feature images according to a preset bilinear difference formula; the bilinear difference formula is:

[0039]

[0040] Among them, (x3, y3) is the position where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight;

[0041] S4-9. After all the size-aligned feature images are spliced ​​according to a preset third target dimension, each feature image is linearly transformed based on a preset linear layer, and an attention score corresponding to each feature image is obtained according to a preset attention mechanism.

[0042] S4-10, according to the attention score corresponding to each feature image and the preset Sotfmax function, obtain the attention weight corresponding to each feature image, and perform weighted summation according to the attention weight corresponding to each feature image to obtain the fused feature image;

[0043] At this time, S5 includes inputting the fused feature image into a preset classification head to obtain a feature classification result.

[0044] Optionally, the S5 includes:

[0045] The feature image is input into the classification head, and after the feature image is pooled based on a preset global average pooling layer, fine-grained feature classification is performed based on the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding level of eyelid margin opening or eyelid margin congestion.

[0046] Optionally, the S4-3 further includes:

[0047] Inputting the first feature image into a preset feedforward network for linear activation processing;

[0048] The feedforward network is:

[0049] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0050] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias terms, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0051] Optionally, the eyelid margin segmentation model includes an L-level encoder and an L-1-level decoder, where L is a positive integer greater than 1;

[0052] Then the S1 includes:

[0053] S1-1. Input the original feature image into the eyelid margin segmentation model. The outputs of the last-level encoder and all decoders are respectively subjected to 3×3 convolution and Sigmoid function to obtain the saliency probability maps corresponding to the last-level encoder and all decoders.

[0054] S1-2, upsample all output saliency probability maps to the same size as the original feature image and fuse them through a cascade operation;

[0055] S1-3. Generate the segmentation image corresponding to the original feature image through a 1×1 convolution layer and a Sigmoid function.

[0056] Optionally, the S1-1 further includes:

[0057] When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is:

[0058] y1=ReLU(BN(K*x1+b));

[0059] Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization;

[0060] The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is:

[0061] y2=MaxPool(x2);

[0062] Among them, x2 is the third process image, and y2 is the output result after downsampling.

[0063] Optionally, the S1-1 further includes: when any decoder receives input data, upsampling the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is:

[0064]

[0065] Among them, (x3, y3) is the position where the difference is performed in the input data, Ii,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

[0066] Optionally, before S1, the step further includes:

[0067] S0. constructing a first data training set based on a preset first eye image dataset, constructing a second data training set based on a preset second eye image dataset, and constructing a third data training set based on a preset third eye image dataset;

[0068] The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix;

[0069] Preprocessing all image data in the first data training set, the second data training set, and the third data training set respectively, wherein the preprocessing includes normalization, size scaling, and noise removal;

[0070] Training a pre-built eyelid margin segmentation model according to the first data training set to obtain a trained eyelid margin segmentation model;

[0071] Training a pre-built feature classification model according to the second data training set to obtain a trained feature classification model;

[0072] The preliminary feature extraction model is trained according to the third data training set to obtain a trained preliminary feature extraction model.

[0073] (3) Beneficial effects

[0074] The beneficial effects of the present invention are as follows: a method for classifying eyelid margin features of the present invention, while utilizing an adaptive window mechanism, realizes the use of segmentation information to assist feature classification through an eyelid margin segmentation model and a feature classification model, and optimizes the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flow chart of a method for classifying eyelid margin features;

[0076] Figure 2 This is the structure diagram of the eyelid margin segmentation model;

[0077] Figure 3 This is the structure diagram of the feature classification model. DETAILED DESCRIPTION

[0078] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0079] An eyelid margin feature classification method proposed in an embodiment of the present invention utilizes an adaptive window mechanism while using an eyelid margin segmentation model and a feature classification model to utilize segmentation information to assist feature classification, and optimizes the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features.

[0080] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0081] Example 1

[0082] This embodiment provides a method for classifying eyelid margin features, such as Figure 1 Shown, including:

[0083] S1. The original feature map is input into the pre-set eyelid margin segmentation model to obtain the segmentation image corresponding to the original feature map;

[0084] That is, when the preset eyelid margin segmentation model receives the original feature image, the outputs of the last-level encoder and all decoders are respectively input into the preset 3×3 convolution and Sigmoid function to obtain the corresponding saliency probability maps; all output saliency probability maps are upsampled to the same size as the original feature map and fused through the cascade operation box; and the segmentation image corresponding to the original feature image is generated through a 1×1 convolution layer and a Sigmoid function.

[0085] S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image;

[0086] During this period, a feature image is obtained according to the segmented image after the window is re-divided and the pre-set first target dimension;

[0087] That is, the segmented image is input into a preset feature classification model, and the feature classification model divides the segmented image into at least one image patch according to the size of the segmented image and a preset image patch size;

[0088] After flattening any image block into a vector based on the local features of the image block, the image block is mapped to the first target dimension using a preset linear transformation formula and a preset first target dimension; the linear transformation formula is:

[0089] E=W patch ·patch;

[0090] Among them, W patch is a preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block.

[0091] After all image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained based on a pre-set windowed self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block. Specifically, the first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block, a pre-set windowed multi-head self-attention mechanism, and a pre-set moving window multi-head self-attention mechanism.

[0092] According to the preset downsampling multiple N, the image blocks in the first feature image are merged to obtain the N downsampling multiple of the first feature image. 2 first process graphs, where N is a non-zero even number; after concatenating all the first process graphs according to a preset second target dimension, inputting the convolutional layer into a preset 1×1 convolutional layer to obtain a second process graph;

[0093] According to the preset window self-attention mechanism and the second process map, a second feature image is obtained; the size of the second feature image is 1 / 2 of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image;

[0094] The self-attention within the window is:

[0095]

[0096] Where Q = W Q x, K = W K x,V=W V x, Q is the query vector, K is the key vector, V is the value vector, W Q 、W K and W V are all pre-set weight matrices, is a preset scaling factor, and T is a preset smoothness constant. The in-window self-attention mechanism includes: window multi-head self-attention mechanism and moving window multi-head self-attention mechanism.

[0097] After the first feature image or the second feature image is generated, a linear activation process is performed on the first feature image or the second feature image through a pre-set feedforward network;

[0098] The feedforward network is:

[0099] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0100] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias terms, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0101] The first feature image and the second feature image are aligned in size according to a preset bilinear difference formula; the bilinear difference formula is:

[0102]

[0103] Among them, (x3, y3) is the position where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight;

[0104] After all the aligned feature images are spliced ​​according to the preset third target dimension, each feature image is linearly transformed based on the preset linear layer, and the attention score corresponding to each feature image is obtained according to the preset attention mechanism;

[0105] According to the attention score corresponding to each feature image and the preset Sotfmax function, the attention weight corresponding to each feature image is obtained, and the weighted summation is performed according to the attention weight corresponding to each feature image to obtain the fused feature image.

[0106] S5, inputting the feature image into a preset classification head to obtain a feature classification result;

[0107] That is, when the preset classification head receives the fused feature image, the feature image is pooled based on the preset global average pooling layer, and then fine-grained feature classification is performed according to the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding level of eyelid margin opening or eyelid margin congestion.

[0108] Before inputting the segmented image into the feature classification model, an adaptive window mechanism can be added so that the model can dynamically adjust the size and position of the window, thereby improving the flexibility of the model in processing targets of different sizes, that is, adding step S2, inputting the segmented image into a preset feature preliminary extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image; and S3, dividing the segmented image into windows according to the preset first window size, and obtaining the initialization position and size information corresponding to each window; performing global pooling on the initial image according to the initialization position and size information corresponding to each window, and generating a global feature description of each window; obtaining the feature weight of each window according to the preset self-attention layer and the global description of each window; obtaining the feature importance map corresponding to the segmented image according to the feature weight of each window, and the feature importance map including the relative importance of each window area; adjusting the window size in the feature importance map according to the preset importance threshold, and re-windowing the segmented image based on the window size in the adjusted feature importance map.

[0109] The specific steps include:

[0110] Window Initialization: Define the size of the base window. Initialize a set of base windows based on the size of the input segmented image. For example, you can choose smaller windows such as 8×8 or 16×16. Each window will initially be evenly spaced across the image. Initialize Position and Size: Initialize the position and size of each window, which will be used for subsequent adjustments.

[0111] Feature extraction: The traditional Swin Transformer is applied to perform preliminary feature extraction. This stage uses a fixed-size window to process the image data and calculate the feature map (i.e., obtain the initial image).

[0112] Adaptive Adjustment: Feature Importance Calculation:

[0113] Global pooling: Global pooling is performed on the feature map to generate a global description of each window area. Global pooling can include global average pooling or global maximum pooling.

[0114] Attention Mechanism: Use attention mechanism to calculate the importance of features. For example, use a self-attention layer to evaluate the feature weights of each window region.

[0115] Dynamically adjust the window:

[0116] Feature Importance Analysis: Based on the calculated feature importance, the feature content of each window is analyzed. For example, for areas with high feature importance, the window size can be reduced to capture more details.

[0117] Adjustment strategy: Design an adjustment strategy, such as using specific rules or algorithms to dynamically modify the size and position of the window. You can decide whether to increase or decrease the window based on the feature importance distribution.

[0118] Local Adaptation: For each window region, an adaptive adjustment mechanism is applied to ensure that the window size and position can reflect the feature distribution of the current region.

[0119] Update Window Division: Adjust Window: Update the size and position of the window based on the results of adaptive adjustment. Interpolation methods or other algorithms can be used to accurately adjust the window division.

[0120] Apply new window partitioning: Apply the adjusted window partitioning to the feature classification model.

[0121] That is, the segmented image is divided into windows according to a preset first window size, the initialization position and size information corresponding to each window is obtained, and the segmented image is input into a preset SwinTransformer model, and feature extraction is performed on the segmented image according to the first window size to obtain an initial image corresponding to the segmented image; based on the initialization position and size information corresponding to each window, the initial image is globally pooled to generate a global feature description for each window;

[0122] According to the pre-set self-attention layer and the global description of each window, the feature weight of each window is obtained;

[0123] Obtaining a feature importance map corresponding to the segmented image according to the feature weight of each window, wherein the feature importance map includes the relative importance of each window area;

[0124] Adjust the window size in the feature importance map according to a preset importance threshold, and re-window the segmented image based on the window size in the adjusted feature importance map;

[0125] At this time, the feature classification model inputs the segmented image after re-window division into the pre-set feature classification model, and then divides the segmented image into at least one image block according to the window position and window size of each window in the segmented image after the new window division, and adjusts the size of all image blocks based on the pre-set adaptive pooling layer to make the size of each image block the same before performing subsequent steps.

[0126] This embodiment provides a method for eyelid margin feature classification. By using an eyelid margin segmentation model and a feature classification model, segmentation information is used to assist feature classification, and the method is optimized based on the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features.

[0127] Example 2

[0128] This embodiment provides a method for training an eyelid margin segmentation model and a feature classification model, including:

[0129] A first data training set is constructed based on a preset first eye image data set, a second data training set is constructed based on a preset second eye image data set, and a third data training set is constructed based on a preset third eye image data set.

[0130] The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix.

[0131] Subsequently, all image data in the first data training set, the second data training set, and the third data training set were preprocessed, respectively. The preprocessing included normalization, resizing, and noise removal. Normalization helps to eliminate brightness and contrast differences between different images, resizing ensures that the images meet the requirements of the model input, and noise removal improves the image quality.

[0132] Finally, the eyelid margin segmentation model is trained according to the first data training set to obtain a trained eyelid margin segmentation model; the feature classification model is trained according to the second data training set to obtain a trained feature classification model; and the feature preliminary extraction model is trained according to the third data training set to obtain a trained feature preliminary extraction model.

[0133] Among them, the eyelid margin segmentation model is established based on the U2-Net model, including an encoder part, a jump connection part, a decoder part and a saliency fusion part, which is used for feature extraction and information fusion between channels. The encoder part and the decoder part are connected by a jump connection part.

[0134] When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is:

[0135] y1=ReLU(BN(K*x1+b));

[0136] Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization;

[0137] The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is:

[0138] y2=MaxPool(x2);

[0139] Among them, x2 is the third process image, and y2 is the output result after downsampling.

[0140] When any encoder receives input data, it upsamples the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is:

[0141]

[0142] Among them, (x3, y3) is the position where the difference is performed in the input data received by the encoder, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

[0143] In general, the eyelid margin segmentation model structure is as follows Figure 2 As shown, it includes 6 levels of first encoders and 5 first decoders; among them, the first 4 levels of first encoders (i.e., En_1, En_2, En_3 and En_4) are RSU-7, RSU-6, RSU-5 and RSU-4 structures respectively, and the first 4 levels of first decoders (i.e., De_1, De_2, De_3 and De_4) are RSU-7, RSU-6, RSU-5 and RSU-4 structures respectively; the 5th level first encoder, the 6th level first encoder and the 5th level first decoder (i.e., En_5, En_6 and De_5) are all RSU-4F structures; each RSU-L includes L second encoders and L-1 second decoders; RSU-4F is a symmetric encoder structure with a height of 4 and the same as RSU-L; L is a non-zero natural number.

[0144] Among them, the input of the first-level first encoder is the initial input of the eyelid margin segmentation model, and the input of any first encoder except the first-level first encoder is the output of the first encoder of the previous level of the first encoder; in the first four levels of first decoders, the input of any first decoder is the concatenation of the output of the first encoder of the same level corresponding to the first decoder and the output of the first decoder of the next level of the first decoder; the input of the fifth-level first decoder is the concatenation of the output of the fifth-level first encoder and the output of the sixth-level first encoder.

[0145] That is, the input of En_1 is the original feature image input to the eyelid margin segmentation model, the input of En_2 is the output of En_1, the input of En_3 is the output of En_2, the input of En_4 is the output of En_3, the input of En_5 is the output of En_4, and the input of En_6 is the output of En_5.

[0146] The input of De_1 is the concatenation of the output of En_1 and the output of De_2; the input of De_2 is the concatenation of the output of En_2 and the output of De_3; the input of De_3 is the concatenation of the output of En_3 and the output of De_4; the input of De_4 is the concatenation of the output of En_4 and the output of De_5; the input of De_5 is the concatenation of the output of En_5 and the output of En_6.

[0147] Adding an adaptive window mechanism between the feature classification model and the eyelid margin segmentation model can make the model more flexible in processing image features of different scales; the adaptive window mechanism is implemented by an additional feature preliminary extraction model and an adaptive window model. The feature preliminary extraction model is a basic SwinTransformer, which is used to extract features from the segmented image according to a preset first window size to obtain the initial image corresponding to the segmented image. The adaptive window model is used to initialize a set of basic windows (such as 8x8 or 16x16) according to the size of the input image, calculate the feature importance of each window based on the extracted features (for example, using a global pooling layer plus an attention mechanism), and dynamically adjust the size and position of the window according to the feature importance. For example, for areas with high feature importance, reduce the window size to obtain more details. Finally, re-divide the window.

[0148] The feature classification model is based on the improved model of Swin Transformer model, such as Figure 3 As shown, it includes: an image block splitting module, a 4th-order feature extraction module and a classification head; the image block splitting module divides the input image into non-overlapping image blocks according to the re-divided window size, and unifies the sizes of all image blocks based on the adaptive pooling layer.

[0149] The first-order feature extraction module includes a linear embedding submodule and a Swin Transformer Block submodule; the remaining third-order feature extraction modules include a Patch Merging submodule and a Swin Transformer Block submodule, as well as a feature fusion module and a classification head.

[0150] Each Swin Transformer Block submodule consists of two types of window attention modules (W-MSA and SW-MSA) alternating with each other, and two layers of multi-layer perceptrons (MLPs). These structures use layer normalization and residual connections to ensure smooth information flow and stable model training.

[0151] W-MSA calculates self-attention within each window, and the windows do not overlap to reduce computational complexity.

[0152] SW-MSA allows cross-window information exchange by being unique between adjacent windows, thereby increasing communication between image patches while maintaining locality.

[0153] MLP is generally a feedforward network consisting of two linear transformations and an activation function. The feedforward network is:

[0154] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0155] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias terms, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0156] The feature fusion module is used to fuse the four feature images output by the four-stage Swin Transformer Block submodule, more effectively integrating features of different scales and improving classification accuracy. Specifically, it obtains four feature images of different scales and aligns the sizes of all feature images according to a pre-set bilinear difference formula; the bilinear difference formula is:

[0157]

[0158] Among them, is the position where the difference is taken in the input data, is the pixel value of the four surrounding pixels, and is the preset difference weight; after all the size-aligned feature images are spliced ​​according to the preset target dimension, each feature image is linearly transformed based on the preset linear layer, and according to the preset attention mechanism, the attention score corresponding to each feature image is obtained; according to the attention score corresponding to each feature image and the preset Sotfmax function, the attention weight corresponding to each feature image is obtained, and the weighted sum is performed according to the attention weight corresponding to each feature image to obtain the fused feature image.

[0159] The present embodiment provides a training method for an eyelid margin segmentation model and a feature classification model, which provides a model basis for eyelid margin segmentation and feature classification. Through the double-layer nested U structure and RSU block design, multi-scale features are effectively extracted, providing a basis for achieving high-precision eyelid margin segmentation. The RSU block does not significantly increase the computational cost while maintaining high resolution, and is suitable for environments with limited computing resources. It has strong versatility, does not rely on training the backbone network, can be trained from scratch, and can adapt to different working environments. Through the Swin Transformer model, fine-grained feature classification can be performed on the segmented eyelid margin area to improve the accuracy and efficiency of classification. The introduction of the adaptive window mechanism allows the model to dynamically adjust the size and position of the window, thereby improving the flexibility of the model in processing targets of different sizes.

[0160] Example 3

[0161] This embodiment provides a feature classification method, which is based on a training method for an eyelid margin segmentation model and a feature classification model in Example 2, and includes: designing and implementing a control module and integrating it into the feature classification model. The control module is used to determine whether the weight needs to be adjusted by analyzing the statistics of the input features (such as mean, variance). Whether to update the weights is determined by evaluating the difficulty of the classification task (such as classification error). A lightweight neural network (such as MLP) is used to generate a control strategy, and the input of the network is the feature or activation value of the current layer. A sharing mechanism is implemented: the shared weights can be the same weight matrix, or the same weights can be generated through certain transformations.

[0162] A control module is inserted into each Swin Transformer Block to determine whether weights are shared or updated. In a multi-head self-attention mechanism, dynamic weight sharing can be applied to each attention head. In a feedforward network, the frequency of weight updates can be dynamically adjusted.

[0163] Weight sharing: The shared weights can be the same weight matrix, or the same weights can be generated through certain transformations.

[0164] Dynamic update: When the weights need to be updated, the weight values ​​are dynamically adjusted according to the decision of the control module. This can be achieved through the gradient information used in training.

[0165] That is, the control module is integrated with each layer of the feature segmentation model to ensure that the module can be dynamically adjusted during the training process.

[0166] After inserting the control module and sharing mechanism, compare the performance of static and dynamic weight sharing strategies on different tasks to verify their effectiveness. Specifically, this includes comparing static and dynamic weight sharing and multi-task testing. Comparing static and dynamic weight sharing involves training using static and dynamic weight sharing strategies on the same tasks and datasets. Evaluation metrics: Compare model performance metrics such as accuracy, loss, and training time. Multi-task testing: Test the effectiveness of the dynamic weight sharing strategy on different tasks (such as classification, detection, segmentation, etc.).

[0167] Bayesian optimization is used to perform hyperparameter tuning to tune the hyperparameters of the dynamic sharing strategy (such as the frequency of sharing and the intensity of updates) to find the optimal configuration.

[0168] Dynamic weight sharing improves the model's expressiveness and adaptability. Through flexible weight sharing and updating mechanisms, the model can adapt its weight structure to varying input features and task difficulty. This strategy not only reduces the number of parameters but also improves training efficiency and model generalization. This effectively integrates dynamic weight sharing into feature classification models.

[0169] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0170] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0171] In the present invention, unless otherwise expressly specified or limited, when a first feature is "above" or "below" a second feature, it may mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Furthermore, when a first feature is "above," "above," or "above" a second feature, it may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is at a higher level than the second feature. When a first feature is "below," "below," or "below" a second feature, it may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is at a lower level than the second feature.

[0172] In the description of this specification, the terms "one embodiment", "some embodiments", "embodiments", "examples", "specific examples" or "some examples" refer to the specific features, structures, materials or characteristics described in conjunction with the embodiment or example and included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0173] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for classifying eyelid margin features, characterized in that: include: S1. Inputting the original feature map into a preset eyelid margin segmentation model to obtain a segmented image corresponding to the original feature map, wherein the segmented image is an image in which the eyelid margin area is extracted; S2. Inputting the segmented image into a preset preliminary feature extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image; S3, dividing the segmented image into windows according to a preset first window size, and obtaining initialization position and size information corresponding to each window; Based on the initialization position and size information corresponding to each window, the initial image is globally pooled to generate a global feature description for each window; According to the pre-set self-attention layer and the global description of each window, the feature weight of each window is obtained; Obtaining a feature importance map corresponding to the segmented image according to the feature weight of each window, wherein the feature importance map includes the relative importance of each window area; Adjust the window size in the feature importance map according to a preset importance threshold, and re-window the segmented image based on the window size in the adjusted feature importance map; S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image; During this period, a feature image is obtained according to the segmented image after the window is re-divided and the pre-set first target dimension; The feature classification model includes: an image block splitting module, a 4-order feature extraction module, a feature fusion module and a classification head; wherein the first-order feature extraction module includes a linear embedding submodule and a Swin Transformer Block submodule, and the remaining three-order feature extraction modules all include a Patch Merging submodule and a Swin Transformer Block submodule; The feature fusion module is used to size-align the four feature images output by the Swin Transformer Block submodules of the four stages, splice all the size-aligned feature images according to the preset target dimension, perform a linear transformation on each feature image based on a preset linear layer, and obtain the attention score corresponding to each feature image according to the preset attention mechanism; obtain the attention weight corresponding to each feature image based on the attention score corresponding to each feature image and the preset Sotfmax function, and perform weighted summation based on the attention weight corresponding to each feature image to obtain the fused feature image; Each of the Swin Transformer Block submodules consists of a control module, a window attention module, and two layers of multi-layer perceptrons; Each control module is used to determine whether the weight needs to be adjusted based on the statistics of the input features, and to determine whether the weight needs to be updated based on the difficulty of the classification task; When weight adjustment and / or weight update are required, a corresponding control strategy is generated based on a pre-set lightweight neural network, and the adjusted and / or updated weights are shared to all Swin Transformer Block submodules; S5, inputting the feature image into a preset classification head to obtain a feature classification result; The eyelid margin segmentation model is based on U 2 -Net model, including an encoder and a decoder, and the encoder and decoder are connected by a skip connection.

2. The eyelid margin feature classification method according to claim 1, characterized in that: The S4 includes: S4-1. Dividing the segmented image into at least one image block according to the window position and window size of each window in the newly windowed segmented image, and adjusting the size of all the image blocks based on a preset adaptive pooling layer so that the size of each image block is the same; S4-2. After flattening any image block into a vector based on the local features of the image block, map the image block to the first target dimension using a preset linear transformation formula and a preset first target dimension; the linear transformation formula is: E=W patch ·patch; Among them, W patch is the preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block; S4-3, after all image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained according to a preset in-window self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block; At this time, the S5 includes: inputting the first feature image into a preset classification head to obtain a feature classification result.

3. The eyelid margin feature classification method according to claim 2, characterized in that: After S4-3 and before S5, the following is also included: S4-4, according to the preset downsampling multiple N, merge the image blocks in the first feature image to obtain the N downsampling multiple of the first feature image. 2 A first process graph, N is a non-zero even number; S4-5, after concatenating all the first process graphs according to the preset second target dimension, input the pre-set 1*1 convolution layer to obtain the second process graph; S4-6, according to the preset window self-attention mechanism and the second process map, obtain a second feature image; the size of the second feature image is 1 / 2 of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image; The self-attention within the window is: Where Q = W Q x, K = W K x,V=W V x, Q is the query vector, K is the key vector, V is the value vector, W Q 、W K and W V are all pre-set weight matrices, is a preset scaling factor, and T is a preset smoothness constant; At this time, the S5 includes: inputting the second feature image into a preset classification head to obtain a feature classification result.

4. The eyelid margin feature classification method according to claim 3, characterized in that: After S4-6 and before S5, the following also includes: S4-7, jump to S4-4, and replace the first feature image with the latest feature image, repeating the preset number of times; S4-8. Obtain M feature images of different scales, and align the sizes of all feature images according to a preset bilinear difference formula; the bilinear difference formula is: Among them, (x3, y3) is the position where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight; S4-9. After all the size-aligned feature images are spliced ​​according to a preset third target dimension, each feature image is linearly transformed based on a preset linear layer, and an attention score corresponding to each feature image is obtained according to a preset attention mechanism. S4-10, according to the attention score corresponding to each feature image and the preset Sotfmax function, obtain the attention weight corresponding to each feature image, and perform weighted summation according to the attention weight corresponding to each feature image to obtain the fused feature image; At this time, S5 includes inputting the fused feature image into a preset classification head to obtain a feature classification result.

5. The eyelid margin feature classification method according to claim 2, characterized in that: The S5 includes: The feature image is input into the classification head, and after the feature image is pooled based on a preset global average pooling layer, fine-grained feature classification is performed based on the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding level of eyelid margin opening or eyelid margin congestion.

6. The eyelid margin feature classification method according to claim 2, characterized in that: Said S4-3 also includes: Inputting the first feature image into a preset feedforward network for linear activation processing; The feedforward network is: FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2); Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias terms, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

7. The eyelid margin feature classification method according to claim 1, characterized in that: The eyelid margin segmentation model includes an L-level encoder and an L-1-level decoder, where L is a positive integer greater than 1; Then the S1 includes: S1-1. Input the original feature image into the eyelid margin segmentation model. The outputs of the last-level encoder and all decoders are respectively subjected to 3×3 convolution and Sigmoid function to obtain the saliency probability maps corresponding to the last-level encoder and all decoders. S1-2, upsample all output saliency probability maps to the same size as the original feature image and fuse them through a cascade operation; S1-3. Generate the segmentation image corresponding to the original feature image through a 1×1 convolution layer and a Sigmoid function.

8. The eyelid margin feature classification method according to claim 7, characterized in that: Said S1-1 also includes: When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is: y1=ReLU(BN(K*x1+b)); Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization; The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is: y2=MaxPool(x2); Among them, x2 is the third process image, and y2 is the output result after downsampling.

9. The eyelid margin feature classification method according to claim 7, characterized in that: The step S1-1 further includes: when any decoder receives input data, upsampling the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is: Among them, (x3, y3) is the position where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

10. The eyelid margin feature classification method according to claim 1, characterized in that: The S1 also includes: S0. constructing a first data training set based on a preset first eye image dataset, constructing a second data training set based on a preset second eye image dataset, and constructing a third data training set based on a preset third eye image dataset; The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix; Preprocessing all image data in the first data training set, the second data training set, and the third data training set respectively, wherein the preprocessing includes normalization, size scaling, and noise removal; Training a pre-built eyelid margin segmentation model according to the first data training set to obtain a trained eyelid margin segmentation model; Training a pre-built feature classification model according to the second data training set to obtain a trained feature classification model; The preliminary feature extraction model is trained according to the third data training set to obtain a trained preliminary feature extraction model.

Images