Eyelid edge feature classification method

Abstract

The invention relates to the technical field of image processing, in particular to an eyelid margin feature classification method, which comprises the following steps of: inputting an original feature map into an eyelid margin segmentation model to obtain a segmented image; performing window division on the segmented image to obtain initialization position and size information corresponding to each window, and inputting the segmented image into a feature preliminary extraction model to obtain an initial image; according to the initialization position and size information corresponding to each window and the initial image, performing window division on the segmented image again; the segmented image is input into a preset feature classification model, and the feature classification model obtains a feature image according to the segmented image, the image block size and the first target dimension; and inputting the feature image into a classification head to obtain a feature classification result. The method has the beneficial effects that feature classification is assisted by utilizing segmentation information, optimization is carried out according to the features of the eyelid margins, and the effect of processing the features of the eyelid margins is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an eyelid margin feature classification method. Background Art

[0002] The eyelid margin is the junction of the eyelid skin and the conjunctiva. Maintaining a healthy eyelid margin is essential for the health of the overall ocular surface. Blepharitis is one of the most common eye diseases, affecting approximately 47% of ophthalmology patients. Blepharitis often causes meibomian gland dysfunction, dry eyes, and blepharoconjunctivitis (BKC). BKC may cause astigmatism, keratopathy, amblyopia, vision loss, or corneal perforation. Therefore, early diagnosis and treatment of blepharitis are very important to prevent vision loss. According to the American Academy of Ophthalmology classification, blepharitis is divided into anterior blepharitis and posterior blepharitis according to anatomical location. Anterior blepharitis affects the eyelid skin, eyelash base, and eyelash follicles, while posterior blepharitis affects the meibomian glands. The main clinical feature of blepharitis is redness of the eyelid margin, while the hallmark of posterior blepharitis is blockage of the meibomian gland orifices.

[0003] In recent years, deep learning has shown significant potential in medical image segmentation and classification. One study proposed an automatic and objective assessment of eyelid morphology, while another study used deep learning technology to analyze the eyelid morphology of patients with thyroid-related eye disease, showing strong agreement between automatic and manual measurements.

[0004] However, these methods all regard segmentation and classification as independent tasks and fail to fully utilize segmentation information to assist feature classification, resulting in limited accuracy of classification results. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an eyelid margin feature classification method, which solves the technical problem that the prior art regards segmentation and classification as independent tasks, fails to fully utilize segmentation information to assist feature classification, resulting in limited accuracy of classification results.

[0007] (II) Technical solution

[0008] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0009] In a first aspect, an embodiment of the present invention provides a method for classifying eyelid margin features, comprising:

[0010] S1. Inputting the original feature map into a preset eyelid margin segmentation model to obtain a segmented image corresponding to the original feature map, wherein the segmented image is an image from which the eyelid margin area is extracted;

[0011] S2, inputting the segmented image into a preset preliminary feature extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image;

[0012] S3, dividing the segmented image into windows according to a preset first window size, and obtaining initialization position and size information corresponding to each window;

[0013] According to the initialization position and size information corresponding to each window, the initial image is globally pooled to generate a global feature description of each window;

[0014] According to the preset self-attention layer and the global description of each window, the feature weight of each window is obtained;

[0015] According to the feature weight of each window, a feature importance map corresponding to the segmented image is obtained, wherein the feature importance map includes the relative importance of each window area;

[0016] According to a preset importance threshold, the window size in the feature importance map is adjusted, and based on the adjusted window size in the feature importance map, the segmented image is re-windowed;

[0017] S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image;

[0018] During this period, a feature image is obtained according to the segmented image after the window is re-divided and the first target dimension set in advance;

[0019] S5, inputting the feature image into a preset classification head to obtain a feature classification result;

[0020] The eyelid margin segmentation model is based on a U2-Net model, including an encoder and a decoder, and the encoder and the decoder are connected via a jump connection.

[0021] Optionally, the S4 includes:

[0022] S4-1, dividing the segmented image into at least one image block according to the window position and window size of each window in the segmented image after the new window division, and adjusting the size of all the image blocks based on a preset adaptive pooling layer so that the size of each image block is the same;

[0023] S4-2, after flattening any image block into a vector based on the local features of the image block, map the image block to the first target dimension through a preset linear transformation formula and a preset first target dimension; the linear transformation formula is:

[0024] E=Wpatch ·patch;

[0025] Among them, W patch is the preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block;

[0026] S4-3, after all image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained according to a preset in-window self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block;

[0027] At this time, S5 includes: inputting the first feature image into a preset classification head to obtain a feature classification result.

[0028] Optionally, after S4-3 and before S5, the following steps are further included:

[0029] S4-4, according to the preset downsampling multiple N, the image blocks in the first feature image are merged to obtain the N downsampling multiple of the first feature image. 2 A first process diagram, N is a non-zero even number;

[0030] S4-5, after splicing all the first process graphs according to the preset second target dimension; input the preset 1*1 convolution layer to obtain the second process graph;

[0031] S4-6, according to the preset window self-attention mechanism and the second process map, obtain a second feature image; the size of the second feature image is half of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image;

[0032] The self-attention in the window is:

[0033]

[0034] Where Q = W Q x,K=W K x,V=W V x, Q is the query vector, K is the key vector, V is the value vector, W Q , W K and W V are all pre-set weight matrices, is a preset scaling factor, T is a preset smoothness constant;

[0035] At this time, S5 includes: inputting the second feature image into a preset classification head to obtain a feature classification result.

[0036] Optionally, after S4-6 and before S5, the following steps are further included:

[0037] S4-7, jump to S4-4, and replace the first feature image with the latest feature image, repeating a preset number of times;

[0038] S4-8, obtaining M feature images of different scales, and aligning the sizes of all feature images according to a preset bilinear difference formula; the bilinear difference formula is:

[0039]

[0040] Where (x3, y3) is the location where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight;

[0041] S4-9, after all the size-aligned feature images are spliced ​​according to a preset third target dimension, each feature image is linearly transformed based on a preset linear layer, and an attention score corresponding to each feature image is obtained according to a preset attention mechanism;

[0042] S4-10, according to the attention score corresponding to each feature image and the preset Sotfmax function, obtain the attention weight corresponding to each feature image, and perform weighted summation according to the attention weight corresponding to each feature image to obtain the fused feature image;

[0043] At this time, S5 includes inputting the fused feature image into a preset classification head to obtain a feature classification result.

[0044] Optionally, the S5 includes:

[0045] The feature image is input into the classification head, and after the feature image is pooled based on the preset global average pooling layer, fine-grained feature classification is performed according to the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding level of eyelid margin opening or eyelid margin congestion.

[0046] Optionally, the S4-3 further includes:

[0047] Inputting the first feature image into a preset feedforward network for linear activation processing;

[0048] The feedforward network is:

[0049] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0050] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias items, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0051] Optionally, the eyelid margin segmentation model includes an L-level encoder and an L-1-level decoder, where L is a positive integer greater than 1;

[0052] Then the S1 includes:

[0053] S1-1, input the original feature image into the eyelid margin segmentation model, and the output of the last level encoder and all decoders are respectively passed through 3×3 convolution and Sigmoid function to obtain the salient probability map corresponding to the last level encoder and all decoders;

[0054] S1-2, upsample all output saliency probability maps to the same size as the original feature image and fuse them through a cascade operation;

[0055] S1-3. Generate the segmentation image corresponding to the original feature image through a 1×1 convolution layer and a Sigmoid function.

[0056] Optionally, the S1-1 further includes:

[0057] When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is:

[0058] y1 = ReLU(BN(K*x1+b));

[0059] Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization;

[0060] The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is:

[0061] y2=MaxPool(x2);

[0062] Among them, x2 is the third process image, and y2 is the output result after downsampling.

[0063] Optionally, S1-1 further includes: when any decoder receives input data, upsampling the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is:

[0064]

[0065] Where (x3, y3) is the location where the difference is performed in the input data, Ii,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

[0066] Optionally, before S1, the step further includes:

[0067] S0, constructing a first data training set based on a preset first eye image data set, constructing a second data training set based on a preset second eye image data set, and constructing a third data training set based on a preset third eye image data set;

[0068] The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix;

[0069] Preprocessing all image data in the first data training set, the second data training set, and the third data training set respectively, wherein the preprocessing includes normalization, size scaling, and noise removal;

[0070] Training a pre-built eyelid margin segmentation model according to the first data training set to obtain a trained eyelid margin segmentation model;

[0071] Training a pre-built feature classification model according to the second data training set to obtain a trained feature classification model;

[0072] The preliminary feature extraction model is trained according to the third data training set to obtain a trained preliminary feature extraction model.

[0073] (III) Beneficial effects

[0074] The beneficial effects of the present invention are as follows: a method for classifying eyelid margin features of the present invention, while utilizing an adaptive window mechanism, through an eyelid margin segmentation model and a feature classification model, realizes the use of segmentation information to assist feature classification, and optimizes the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 is a flow chart of a method for classifying eyelid margin features;

[0076] Figure 2 This is the structure diagram of the eyelid margin segmentation model;

[0077] Figure 3 This is the structure diagram of the feature classification model. DETAILED DESCRIPTION

[0078] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0079] An eyelid margin feature classification method proposed in an embodiment of the present invention utilizes an adaptive window mechanism while using an eyelid margin segmentation model and a feature classification model to utilize segmentation information to assist feature classification, and optimizes the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features.

[0080] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0081] Example 1

[0082] This embodiment provides a method for classifying eyelid margin features, such as Figure 1 As shown, including:

[0083] S1. The original feature map is input into a preset eyelid margin segmentation model to obtain a segmentation image corresponding to the original feature map;

[0084] That is, when the preset eyelid margin segmentation model receives the original feature image, the outputs of the last level encoder and all decoders are respectively input into the preset 3×3 convolution and Sigmoid function to obtain the corresponding significant probability maps; all the output significant probability maps are upsampled to the same size as the original feature map and fused through the cascade operation box; through the 1×1 convolution layer and the Sigmoid function, the segmentation image corresponding to the original feature image is generated.

[0085] S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image;

[0086] During this period, a feature image is obtained according to the segmented image after the window is re-divided and the first target dimension set in advance;

[0087] That is, the segmented image is input into a preset feature classification model, and the feature classification model divides the segmented image into at least one image patch according to the size of the segmented image and a preset image patch size;

[0088] After flattening any image block into a vector based on the local features of the image block, the image block is mapped to the first target dimension through a preset linear transformation formula and a preset first target dimension; the linear transformation formula is:

[0089] E=W patch ·patch;

[0090] Among them, W patch is a preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block.

[0091] After all the image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained according to a preset in-window self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block. Specifically, the first feature image corresponding to the segmented image is obtained according to the first feature information corresponding to each image block, a preset window multi-head self-attention mechanism, and a preset moving window multi-head self-attention mechanism.

[0092] According to the preset downsampling multiple N, the image blocks in the first feature image are merged to obtain the N downsampling multiple of the first feature image. 2 first process graphs, where N is a non-zero even number; after all the first process graphs are spliced ​​according to a preset second target dimension, they are input into a preset 1×1 convolutional layer to obtain a second process graph;

[0093] According to the preset window self-attention mechanism and the second process map, a second feature image is obtained; the size of the second feature image is half of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image;

[0094] The self-attention in the window is:

[0095]

[0096] Where Q = W Q x,K=W K x,V=W V x, Q is the query vector, K is the key vector, V is the value vector, W Q , W K and W V are all pre-set weight matrices, is a preset scaling factor, and T is a preset smoothness constant. The self-attention mechanism within the window includes: a window multi-head self-attention mechanism and a moving window multi-head self-attention mechanism.

[0097] After the first feature image or the second feature image is generated, a linear activation process is performed on the first feature image or the second feature image through a pre-set feedforward network;

[0098] The feedforward network is:

[0099] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0100] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias items, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0101] The first feature image and the second feature image are size-aligned according to a preset bilinear difference formula; the bilinear difference formula is:

[0102]

[0103] Where (x3, y3) is the location where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight;

[0104] After all the feature images after size alignment are spliced ​​according to the preset third target dimension, each feature image is linearly transformed based on the preset linear layer, and the attention score corresponding to each feature image is obtained according to the preset attention mechanism;

[0105] According to the attention score corresponding to each feature image and the preset Sotfmax function, the attention weight corresponding to each feature image is obtained, and the weighted summation is performed according to the attention weight corresponding to each feature image to obtain the fused feature image.

[0106] S5, inputting the feature image into a preset classification head to obtain a feature classification result;

[0107] That is, when the preset classification head receives the fused feature image, it pools the feature image based on the preset global average pooling layer, and then performs fine-grained feature classification according to the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding levels of eyelid margin opening or eyelid margin congestion.

[0108] Before the segmented image is input into the feature classification model, an adaptive window mechanism can be added so that the model can dynamically adjust the size and position of the window, thereby improving the flexibility of the model in processing targets of different sizes, that is, adding step S2, inputting the segmented image into a preset feature preliminary extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image; and S3, dividing the segmented image into windows according to the preset first window size, and obtaining the initial position and size information corresponding to each window; performing global pooling on the initial image according to the initial position and size information corresponding to each window, and generating a global feature description of each window; obtaining the feature weight of each window according to the preset self-attention layer and the global description of each window; obtaining the feature importance map corresponding to the segmented image according to the feature weight of each window, and the feature importance map includes the relative importance of each window area; adjusting the window size in the feature importance map according to a preset importance threshold, and re-dividing the segmented image into windows based on the window size in the adjusted feature importance map.

[0109] The specific steps include:

[0110] Window initialization: Define the size of the basic window and initialize a set of basic windows according to the size of the input segmented image. For example, you can choose a smaller window such as 8×8 or 16×16. Each window will be evenly divided on the image in the initial stage. Initialize position and size Initialize the position and size information for each window, which will be used for subsequent adjustments.

[0111] Feature extraction: The traditional Swin Transformer is used for preliminary feature extraction. This stage uses a fixed-size window to process the image data and calculate the feature map (i.e., obtain the initial image).

[0112] Adaptive Adjustment: Feature Importance Calculation:

[0113] Global pooling: Global pooling is performed on the feature map to generate a global description of each window area. Global pooling can include global average pooling or global maximum pooling.

[0114] Attention Mechanism: Use an attention mechanism to calculate the importance of features. For example, use a self-attention layer to evaluate the feature weights for each window region.

[0115] Dynamically adjust the window:

[0116] Feature Importance Analysis: Based on the calculated feature importance, the feature content of each window is analyzed. For example, for areas with high feature importance, you can choose to reduce the window size to capture more details.

[0117] Adjustment strategy: Design an adjustment strategy, such as using specific rules or algorithms to dynamically modify the size and position of the window. You can decide whether to increase or decrease the window based on the feature importance distribution.

[0118] Local Adaptation: For each window region, an adaptive adjustment mechanism is applied to ensure that the window size and position can reflect the feature distribution of the current region.

[0119] Update window division: Adjust window: Update the size and position of the window according to the result of adaptive adjustment. Interpolation method or other algorithms can be used to accurately adjust the window division.

[0120] Apply New Window Partitioning: Apply the adjusted window partitioning to the feature classification model.

[0121] That is, the segmented image is divided into windows according to a preset first window size, the initialization position and size information corresponding to each window is obtained, and the segmented image is input into a preset SwinTransformer model, and the segmented image is subjected to feature extraction according to the first window size to obtain an initial image corresponding to the segmented image; according to the initialization position and size information corresponding to each window, the initial image is subjected to global pooling to generate a global feature description of each window;

[0122] According to the preset self-attention layer and the global description of each window, the feature weight of each window is obtained;

[0123] According to the feature weight of each window, a feature importance map corresponding to the segmented image is obtained, wherein the feature importance map includes the relative importance of each window area;

[0124] According to a preset importance threshold, the window size in the feature importance map is adjusted, and based on the adjusted window size in the feature importance map, the segmented image is re-windowed;

[0125] At this time, the feature classification model inputs the segmented image after the re-window division into the pre-set feature classification model, and then divides the segmented image into at least one image block according to the window position and window size of each window in the segmented image after the new window division, and adjusts the size of all image blocks based on the pre-set adaptive pooling layer to make the size of each image block the same, and then proceeds to the subsequent steps.

[0126] The present embodiment provides an eyelid margin feature classification method, which uses an eyelid margin segmentation model and a feature classification model to utilize segmentation information to assist feature classification, and optimizes the characteristics of the eyelid margin, thereby improving the effect of processing eyelid margin features.

[0127] Example 2

[0128] This embodiment provides a method for training an eyelid margin segmentation model and a feature classification model, including:

[0129] A first data training set is constructed based on a preset first eye image data set, a second data training set is constructed based on a preset second eye image data set, and a third data training set is constructed based on a preset third eye image data set.

[0130] The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix.

[0131] Subsequently, all image data in the first data training set, the second data training set and the third data training set are preprocessed respectively, and the preprocessing includes normalization, resizing and noise removal; normalization helps to eliminate the brightness and contrast differences between different images, resizing ensures that the image meets the requirements of the model input, and noise removal improves the quality of the image.

[0132] Finally, the eyelid margin segmentation model is trained according to the first data training set to obtain a trained eyelid margin segmentation model; the feature classification model is trained according to the second data training set to obtain a trained feature classification model; and the feature preliminary extraction model is trained according to the third data training set to obtain a trained feature preliminary extraction model.

[0133] Among them, the eyelid margin segmentation model is established based on the U2-Net model, including an encoder part, a jump connection part, a decoder part and a saliency fusion part, which is used for feature extraction and information fusion between channels. The encoder part and the decoder part are connected by a jump connection part.

[0134] When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is:

[0135] y1 = ReLU(BN(K*x1+b));

[0136] Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization;

[0137] The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is:

[0138] y2=MaxPool(x2);

[0139] Among them, x2 is the third process image, and y2 is the output result after downsampling.

[0140] When any encoder receives input data, it upsamples the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is:

[0141]

[0142] Where (x3, y3) is the position where the difference is made in the input data received by the encoder, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

[0143] In general, the eyelid margin segmentation model structure is as follows Figure 2 As shown, it includes 6 levels of first encoders and 5 first decoders; wherein, the first 4 levels of first encoders (i.e., En_1, En_2, En_3 and En_4) are RSU-7, RSU-6, RSU-5 and RSU-4 structures respectively, and the first 4 levels of first decoders (i.e., De_1, De_2, De_3 and De_4) are RSU-7, RSU-6, RSU-5 and RSU-4 structures respectively; the 5th level first encoder, the 6th level first encoder and the 5th level first decoder (i.e., En_5, En_6 and De_5) are all RSU-4F structures; each RSU-L includes L second encoders and L-1 second decoders; RSU-4F is a symmetric encoder structure with a height of 4 and the same as RSU-L; L is a non-zero natural number.

[0144] Among them, the input of the first-level first encoder is the initial input of the eyelid margin segmentation model, and the input of any first encoder except the first-level first encoder is the output of the first encoder of the previous level of the first encoder; in the first four levels of first decoders, the input of any first decoder is the concatenation result of the output of the first encoder of the same level corresponding to the first decoder and the output of the first decoder of the next level of the first decoder; the input of the fifth-level first decoder is the concatenation result of the output of the fifth-level first encoder and the output of the sixth-level first encoder.

[0145] That is, the input of En_1 is the original feature image input to the eyelid margin segmentation model, the input of En_2 is the output of En_1, the input of En_3 is the output of En_2, the input of En_4 is the output of En_3, the input of En_5 is the output of En_4, and the input of En_6 is the output of En_5.

[0146] The input of De_1 is the concatenation of the output of En_1 and the output of De_2; the input of De_2 is the concatenation of the output of En_2 and the output of De_3; the input of De_3 is the concatenation of the output of En_3 and the output of De_4; the input of De_4 is the concatenation of the output of En_4 and the output of De_5; the input of De_5 is the concatenation of the output of En_5 and the output of En_6.

[0147] Adding an adaptive window mechanism between the feature classification model and the eyelid margin segmentation model can make the model more flexible in processing image features of different scales; the adaptive window mechanism is implemented by an additional feature preliminary extraction model and an adaptive window model. The feature preliminary extraction model is a basic SwinTransformer, which is used to extract features from the segmented image according to a preset first window size to obtain an initial image corresponding to the segmented image. The adaptive window model is used to initialize a set of basic windows (e.g., 8x8 or 16x16) according to the size of the input image, calculate the feature importance of each window based on the extracted features (e.g., using a global pooling layer plus an attention mechanism), and dynamically adjust the size and position of the window according to the feature importance. For example, for areas with high feature importance, reduce the window size to obtain more details. Finally, re-divide the window.

[0148] The feature classification model is based on the improved model of Swin Transformer model, such as Figure 3 As shown, it includes: an image block splitting module, a 4th-order feature extraction module and a classification head; the image block splitting module divides the input image into non-overlapping image blocks according to the re-divided window size, and unifies the sizes of all image blocks based on the adaptive pooling layer.

[0149] The first-order feature extraction module includes a linear embedding submodule and a Swin Transformer Block submodule; the remaining third-order feature extraction modules include a Patch Merging submodule and a Swin Transformer Block submodule, as well as a feature fusion module and a classification head.

[0150] Each Swin Transformer Block submodule consists of two types of window attention modules (W-MSA and SW-MSA) used alternately, and two layers of multi-layer perceptrons (MLP). These structures use layer normalization and residual connections to ensure smooth information flow and stable model training.

[0151] W-MSA calculates self-attention in each window, and the windows do not overlap to reduce computational complexity.

[0152] SW-MSA allows cross-window information exchange by making it unique between adjacent windows, thereby increasing communication between image patches while maintaining locality.

[0153] MLP is generally a feedforward network consisting of two linear transformations and an activation function. The feedforward network is:

[0154] FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2);

[0155] Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias items, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

[0156] The feature fusion module is used to fuse the four feature images output by the Swin Transformer Block submodules in the four stages, integrate features of different scales more effectively, and improve classification accuracy. Specifically, it includes: obtaining four feature images of different scales, and aligning the sizes of all feature images according to the pre-set bilinear difference formula; the bilinear difference formula is:

[0157]

[0158] Among them, is the position where the difference is taken in the input data, is the pixel value of the surrounding four pixels, and is the preset difference weight; after all the size-aligned feature images are spliced ​​according to the preset target dimension, each feature image is linearly transformed based on the preset linear layer, and according to the preset attention mechanism, the attention score corresponding to each feature image is obtained; according to the attention score corresponding to each feature image and the preset Sotfmax function, the attention weight corresponding to each feature image is obtained, and the weighted sum is performed according to the attention weight corresponding to each feature image to obtain the fused feature image.

[0159] The training method of an eyelid margin segmentation model and a feature classification model provided in this embodiment provides a model basis for eyelid margin segmentation and feature classification. Through the double-layer nested U structure and RSU block design, multi-scale features are effectively extracted, which provides a basis for achieving high-precision eyelid margin segmentation. The RSU block does not significantly increase the computational cost while maintaining high resolution, and is suitable for environments with limited computing resources. It has strong versatility, does not rely on training the backbone network, can be trained from scratch, and can adapt to different working environments. Through the Swin Transformer model, fine-grained feature classification can be performed on the segmented eyelid margin area to improve the accuracy and efficiency of classification. The introduction of the adaptive window mechanism allows the model to dynamically adjust the size and position of the window, thereby improving the flexibility of the model in processing targets of different sizes.

[0160] Example 3

[0161] This embodiment provides a feature classification method, which is based on a method for training an eyelid margin segmentation model and a feature classification model in Embodiment 2, and includes: designing and implementing a control module and integrating it into the feature classification model. The control module is used to determine whether the weight needs to be adjusted by analyzing the statistics of the input features (such as mean, variance). Decide whether to update the weights by evaluating the difficulty of the classification task (such as classification error). Use a lightweight neural network (such as MLP) to generate a control strategy, and the input of the network is the features or activation values ​​of the current layer. Implement a sharing mechanism: the shared weights can be the same weight matrix, or the same weights can be generated through certain transformations.

[0162] Insert a control module in each Swin Transformer Block to decide weight sharing or updating. In the multi-head self-attention mechanism, dynamic weight sharing can be applied in each attention head. In the feedforward network, the update frequency of weights is dynamically adjusted.

[0163] Weight sharing: The shared weights can be the same weight matrix, or the same weights can be generated through certain transformations.

[0164] Dynamic update: When the weights need to be updated, the weight values ​​are adjusted dynamically based on the decision of the control module. This can be achieved through the gradient information used in training.

[0165] That is, the control module is integrated with each layer of the feature segmentation model to ensure that the module can be dynamically adjusted during the training process.

[0166] After inserting the control module and sharing mechanism, compare the performance of static and dynamic weight sharing strategies on different tasks to verify their effectiveness. Specifically, it includes comparing static and dynamic weight sharing and multi-task testing. Comparing static and dynamic weight sharing is to use static weight sharing and dynamic weight sharing strategies for training on the same tasks and datasets. Evaluation indicators: Compare the performance indicators of the model, such as accuracy, loss value, training time, etc. Multi-task testing: Test the effect of dynamic weight sharing strategy on different tasks (such as classification, detection, segmentation, etc.).

[0167] Bayesian optimization is used to perform hyperparameter tuning to tune the hyperparameters of the dynamic sharing strategy (such as sharing frequency and update intensity) to find the optimal configuration.

[0168] The dynamic weight sharing strategy can improve the expressiveness and adaptability of the model. Through the flexible weight sharing and updating mechanism, the model can adjust its weight structure under different input features and task difficulties. This strategy not only reduces the number of parameters, but also improves the training efficiency and the generalization ability of the model. The dynamic weight sharing strategy is effectively integrated into the feature classification model.

[0169] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0170] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0171] In the present invention, unless otherwise clearly specified and limited, when a first feature is “on” or “below” a second feature, it may be that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. Moreover, when a first feature is “above”, “above” or “above” a second feature, it may be that the first feature is directly above or obliquely above the second feature, or it may simply mean that the first feature is higher in level than the second feature. When a first feature is “below”, “below” or “below” a second feature, it may be that the first feature is directly below or obliquely below the second feature, or it may simply mean that the first feature is lower in level than the second feature.

[0172] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0173] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may alter, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for classifying eyelid margin features, characterized in that: include: S1. Inputting the original feature map into a preset eyelid margin segmentation model to obtain a segmented image corresponding to the original feature map, wherein the segmented image is an image from which the eyelid margin area is extracted; S2, inputting the segmented image into a preset preliminary feature extraction model, performing feature extraction on the segmented image according to a preset first window size, and obtaining an initial image corresponding to the segmented image; S3, dividing the segmented image into windows according to a preset first window size, and obtaining initialization position and size information corresponding to each window; According to the initialization position and size information corresponding to each window, the initial image is globally pooled to generate a global feature description of each window; According to the preset self-attention layer and the global description of each window, the feature weight of each window is obtained; According to the feature weight of each window, a feature importance map corresponding to the segmented image is obtained, wherein the feature importance map includes the relative importance of each window area; According to a preset importance threshold, the window size in the feature importance map is adjusted, and based on the adjusted window size in the feature importance map, the segmented image is re-windowed; S4, inputting the segmented image after re-window division into a pre-set feature classification model to obtain a feature image; During this period, a feature image is obtained according to the segmented image after the window is re-divided and the first target dimension set in advance; S5, inputting the feature image into a preset classification head to obtain a feature classification result; The eyelid margin segmentation model is based on a U2-Net model, including an encoder and a decoder, and the encoder and the decoder are connected via a jump connection.

2. The eyelid margin feature classification method according to claim 1, characterized in that: The S4 includes: S4-1, dividing the segmented image into at least one image block according to the window position and window size of each window in the segmented image after the new window division, and adjusting the size of all the image blocks based on a preset adaptive pooling layer so that the size of each image block is the same; S4-2, after flattening any image block into a vector based on the local features of the image block, map the image block to the first target dimension through a preset linear transformation formula and a preset first target dimension; the linear transformation formula is: E=W patch ·patch; Among them, W patch is the preset linear transformation matrix, E is the flattened image block vector, and patch is the original image block; S4-3, after all image blocks are mapped to the first target dimension, first feature information corresponding to each image block is obtained according to a preset in-window self-attention mechanism, and a first feature image corresponding to the segmented image is obtained based on the first feature information corresponding to each image block; At this time, S5 includes: inputting the first feature image into a preset classification head to obtain a feature classification result.

3. The eyelid margin feature classification method according to claim 2, characterized in that: After S4-3 and before S5, the following further includes: S4-4, according to the preset downsampling multiple N, the image blocks in the first feature image are merged to obtain the N downsampling multiple of the first feature image. 2 A first process diagram, N is a non-zero even number; S4-5, after splicing all the first process graphs according to the preset second target dimension; input the preset 1*1 convolution layer to obtain the second process graph; S4-6, according to the preset window self-attention mechanism and the second process map, obtain a second feature image; the size of the second feature image is half of the size of the first feature image The number of channels of the second feature image is N times the number of channels of the first feature image; The self-attention in the window is: in, Q is the query vector, K is the key vector, V is the value vector, W Q , W K and W V are all pre-set weight matrices, is a preset scaling factor, T is a preset smoothness constant; At this time, S5 includes: inputting the second feature image into a preset classification head to obtain a feature classification result.

4. The eyelid margin feature classification method according to claim 3, characterized in that: After S4-6 and before S5, the following also includes: S4-7, jump to S4-4, and replace the first feature image with the latest feature image, repeating a preset number of times; S4-8, obtaining M feature images of different scales, and aligning the sizes of all feature images according to a preset bilinear difference formula; the bilinear difference formula is: Where (x3, y3) is the location where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight; S4-9, after all the size-aligned feature images are spliced ​​according to a preset third target dimension, each feature image is linearly transformed based on a preset linear layer, and an attention score corresponding to each feature image is obtained according to a preset attention mechanism; S4-10, according to the attention score corresponding to each feature image and the preset Sotfmax function, obtain the attention weight corresponding to each feature image, and perform weighted summation according to the attention weight corresponding to each feature image to obtain the fused feature image; At this time, S5 includes inputting the fused feature image into a preset classification head to obtain a feature classification result.

5. The eyelid margin feature classification method according to claim 2, characterized in that: The S5 includes: The feature image is input into the classification head, and after the feature image is pooled based on the preset global average pooling layer, fine-grained feature classification is performed according to the feature information of the pooled feature image to obtain a feature classification result; the feature classification result includes eyelid margin opening or eyelid margin congestion, and the corresponding level of eyelid margin opening or eyelid margin congestion.

6. The eyelid margin feature classification method according to claim 2, characterized in that: The S4-3 also includes: Inputting the first feature image into a preset feedforward network for linear activation processing; The feedforward network is: FFN(x)=ReLU(W2·ReLU(W1·x+b1)+b2); Among them, W1 and W2 are pre-set weights, b1 and b2 are pre-set bias items, x is the feature information of the feedforward network input image, and FFN(x) is the output result of the feedforward network.

7. The eyelid margin feature classification method according to claim 1, characterized in that: The eyelid margin segmentation model includes an L-level encoder and an L-1-level decoder, where L is a positive integer greater than 1; Then the S1 includes: S1-1, input the original feature image into the eyelid margin segmentation model, and the output of the last level encoder and all decoders are respectively passed through 3×3 convolution and Sigmoid function to obtain the salient probability map corresponding to the last level encoder and all decoders; S1-2, upsample all output saliency probability maps to the same size as the original feature image and fuse them through a cascade operation; S1-3. Generate the segmentation image corresponding to the original feature image through a 1×1 convolution layer and a Sigmoid function.

8. The eyelid margin feature classification method according to claim 7, characterized in that: The S1-1 also includes: When any encoder receives input data, a third process image is obtained according to a preset formula 1; the formula 1 is: y1 = ReLU(BN(K*x1+b)); Among them, K is the convolution kernel, x1 is the input data, b is the bias term, * is the convolution operation, and BN is batch normalization; The third process image is downsampled by a preset maximum pooling formula to obtain a downsampling result; the maximum pooling formula is: y2=MaxPool(x2); Among them, x2 is the third process image, and y2 is the output result after downsampling.

9. The eyelid margin feature classification method according to claim 7, characterized in that: The S1-1 further includes: when any decoder receives input data, upsampling the input data according to a preset bilinear difference formula to obtain an upsampling result; the bilinear difference formula is: Where (x3, y3) is the location where the difference is performed in the input data, I i,j is the pixel value of the four pixels around (x3, y3), W ij is the preset difference weight.

10. The eyelid margin feature classification method according to claim 1, characterized in that: The S1 also includes: S0, constructing a first data training set based on a preset first eye image data set, constructing a second data training set based on a preset second eye image data set, and constructing a third data training set based on a preset third eye image data set; The first data training set includes at least one eye image data and a mask corresponding to each eye image data; the second data training set includes at least one eyelid margin segmentation image and a classification result corresponding to each eyelid margin segmentation image; the third data training set includes at least one eyelid margin segmentation image and a corresponding feature matrix; Preprocessing all image data in the first data training set, the second data training set, and the third data training set respectively, wherein the preprocessing includes normalization, size scaling, and noise removal; Training a pre-built eyelid margin segmentation model according to the first data training set to obtain a trained eyelid margin segmentation model; Training a pre-built feature classification model according to the second data training set to obtain a trained feature classification model; The preliminary feature extraction model is trained according to the third data training set to obtain a trained preliminary feature extraction model.

Images