Multi-label myopia image classification method based on deformable self-attention and transformer
By introducing deformable self-attention and Transformer's multi-label myopia image classification method in image classification, using offset network and multi-head attention module, the problem of low retinal state recognition accuracy is solved, and the accurate recognition of high myopia is achieved and Information capture.
Patent Information
- Application Number
- CN202411013885.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
Existing image classification methods cannot accurately identify retinal states, especially due to the low recognition accuracy caused by restriction of receptive field and irrelevant partial effects.
Using a multi-label myopia image classification method based on deformable self-attention and Transformer, the offset is calculated through the pre-trained Resnet18 network and the offset network, combined with the multi-head attention module and linear interpolation, the attention is limited on the left and right sides, and the Flexibility and efficiency of self-attention modules.
It achieves higher retinal state recognition accuracy, can accurately identify high myopia, and improves attention focus and information capture capabilities in myopia-related areas.
Smart Images

Figure CN119006897A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image classification, and in particular to a multi-label close-up image classification method based on deformable self-attention and transformer. Background Art
[0002] Detecting high myopia is important for correctly identifying retinal conditions. High myopia is generally defined as spherical equivalent (SE) ≤ -6.0 diopters or axial length (AL) ≥ 26.0 mm. At present, myopia is generally detected by measuring refractive error, but in many cases, due to scene restrictions, personnel cannot be present, relevant tests cannot be conducted, and relevant historical test records are lacking. This results in a lack of basis for judging the retinal status. With the improvement of technical means, deep learning image classification methods have gradually become one of the effective means to solve this problem.
[0003] Existing image classification methods can be roughly divided into two categories:
[0004] The image classification method based on convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculations. This method is designed based on the translation invariance of image tasks and is good at being applied to tasks such as image processing. CNN can automatically extract features from images and achieve image classification through a combination of convolutional layers, pooling layers, and fully connected layers. However, the receptive field of CNN is limited. If the pixels are far apart, many layers of convolution are required.
[0005] Transformer-based image classification methods, through their unique self-attention mechanism, are able to capture the dependencies between different regions in an image. Larger and even global receptive fields give Transformer models higher representation capabilities than CNN models. However, using dense attention may result in excessive memory and computational costs, and features may be affected by irrelevant parts beyond the region of interest. On the other hand, the sparse attention employed is data agnostic and may limit the ability to model long-range relationships.
[0006] Therefore, current methods are unable to accurately identify the retinal state. Summary of the invention
[0007] The present application provides a multi-label near-vision image classification method based on deformable self-attention and transformer, which can be used to solve the technical problem of low accuracy in retinal state recognition.
[0008] The present application provides a multi-label near-view image classification method based on deformable self-attention and transformer, the method comprising:
[0009] Step 1: Input retinal OCT image;
[0010] Step 2: Put the input image into the pre-trained Resnet18 network to obtain the feature map, and perform linear projection on the feature map as the key k and value v;
[0011] Step 3: Patchembed the input image and then perform linear projection on the sliced image as the query tag q;
[0012] Step 4: Input q into the offset network, calculate the offset, and obtain the deformation sampling point;
[0013] Step 5: Perform linear interpolation on the deformed sampling points to obtain the sampling features, and obtain the left query through linear projection and the right query
[0014] Step 6: q, and Calculate the average value to get the final average query vector
[0015] Step 7: k and v are input into the multi-head attention, and the classification head outputs the classification result.
[0016] Furthermore, the input image is put into the pre-trained Resnet18 network to obtain a feature map, and the feature map is linearly projected as the key k and value v; including:
[0017] The obtained feature map x f Perform linear projection to get the keys and values:
[0018] k = x f W k ,v=x f W v ,
[0019] Among them, W k , W v ∈R C×C Represents a linear projection.
[0020] Furthermore, in step 3, the input image is patchbed sliced, and then the sliced image is linearly projected as the query tag q, including:
[0021] Patchembed the image to obtain the feature map x∈R H×W×C (H, W, C represent the height, width, and number of channels of the feature map, respectively);
[0022] Generate a uniform reference grid on the feature map, denoted as The size of the reference grid is obtained by downsampling the input feature map, and the downsampling factor is r, that is, H G =H / r,W G =W / r; the points on the reference grid constitute the reference points. In order to obtain the offset of each reference point, the feature map is linearly projected to obtain the query mark q=xW q , where W q ∈R C×C Represents a linear projection.
[0023] Furthermore, q is input into the offset network, the offset is calculated, and the deformation sampling point is obtained; including:
[0024] Step 4-1: Use the query feature through the offset network and output the offset values of the reference points respectively:
[0025] The horizontal axis is shifted to the left and right by one unit to form a horizontal axis offset, and the offset network is used to generate the offsets in the left and right vertical axes. The offset network uses the query features and outputs the offset values of the left and right reference points respectively. The area covered by each reference point is a local s×s area. Among them, s is the maximum value of the offset, so the offset generation network can also perceive the local features to learn a reasonable offset. Therefore, the offset network is set as two convolution modules with nonlinear activation. The input features are first captured by a 5×5 depth convolution. Then, GELU activation and 1×1 convolution are used to obtain the one-dimensional longitudinal offset.
[0026] Step 4-2: Input the query tag q into the offset network to obtain the deformation sampling point:
[0027] The query token q is passed through a lightweight offset network θ offset (·) is processed to generate the offset Δp=θ in the vertical axis direction offset (q), in the horizontal direction, the offset to the left and right is fixed by one unit, and the offsets in these two directions are combined to obtain the left and right offsets, that is, the horizontal direction is fixed to the left and right, and the corresponding vertical direction is deformable; in order to stabilize the training process, a predefined factor s is used to limit the scaling amplitude Δp of q, that is, Δp←stanh(Δp), and finally the offset values Δp on the left and right are generated respectively l and Δp r .
[0028] Furthermore, the deformation sampling points are linearly interpolated to obtain the sampling features, and the left query is obtained through linear projection and the right query include:
[0029] Step 5-1: Sample features from the positions of the left and right deformation points respectively to obtain the sampling feature map and Perform linear projection on the sampled feature map to obtain the deformed left query and the right query Expressed as:
[0030]
[0031] The sampling function Set it as linear interpolation, differentiable:
[0032] For all r ly Establishment
[0033] For all r ry Establishment
[0034] where g(a,b)=max(0,1-|ab|), r ly 、r ry Represents each point on the left and right vertical axes respectively. , r r Represents the left and right sides of each point on the horizontal axis, index z∈R H×W×C All positions on p l 、p r Represents the horizontal axis of the reference point after shifting to the left and right, respectively. g is only l , p r The integral points of are non-zero;
[0035] Step 5-2: Linearly project the deformation feature map to obtain the deformation left and right query:
[0036]
[0037] Where W q ∈R C×C Represents a linear projection.
[0038] Furthermore, the average query vector Determined by:
[0039]
[0040] Furthermore, the obtained k,v are input into the multi-head attention, and the classification head outputs the classification result; including:
[0041] Step 7-1: Get k,v are input into the multi-head attention module in the vision transformer, and finally the output of each head is obtained:
[0042]
[0043] Where σ(·) represents the softmax function, d = C / M is the dimension of each head, C is the number of channels, M is the number of attention heads; Z (m) represents the embedding output of the mth attention head, k (m) 、v (m) ∈R N×d represent query, key, and value embeddings respectively;
[0044] z=Concat(z (1) ,…,z (M) )W o
[0045] The features of each head are connected together and passed through W o ∈R C×C to obtain the final output.
[0046] An MLP (Multi-layer Perceptron) block with two linear transformations and GELU (Gaussian Error Linear Unit) activation is used to provide nonlinearity; with LN (layer normalization) and multi-head deformation attention (DAT), the l-th layer transformer block is expressed as:
[0047] (z' l =DAT(LN(z l-1 ))+z l-1
[0048] z l =MLP(LN(z' l ))+z' l
[0049] Among them, z' l represents the intermediate features of the lth transformer block, that is, the result after DAT processing; z l-1 represents the input features of the lth transformer block; z l Represents the final output features of the lth transformer block.
[0050] This application proposes a classification method for multi-label myopia images. Using a bilateral deformable attention module, this method can model the characteristics of myopia images in a targeted manner, learn sparse attention patterns in a data-dependent manner, and limit the deformed attention to the left and right sides of each specific position, so that the model can focus on myopia-related areas. This method can learn several sets of query-agnostic offsets to shift left and right queries to important areas, thereby enhancing the original self-attention module with higher flexibility and efficiency to capture more effective information, and ultimately be able to more accurately identify high myopia. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 This is a flowchart of the multi-label myopia OCT image classification method based on deformable self-attention and transformer provided in this application.
[0052] Figure 2 This is a diagram showing the offset process of the reference point provided in this application.
[0053] Figure 3 This is the attention map provided by this application that visualizes the area of attention of the response model. DETAILED DESCRIPTION
[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0055] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.
[0056] Combination Figure 1 The multi-label close-up image classification method based on deformable self-attention and transformer of the present invention comprises the following steps:
[0057] Step 1: Input retinal OCT image.
[0058] Step 2: Put the input image into the pre-trained Resnet18 network to obtain the feature map, and perform linear projection on the feature map as the key k and value v, specifically:
[0059] The obtained feature map x f Perform linear projection to get the keys and values:
[0060] k = x f W k ,v=x f W v ,
[0061] Among them, W k , W v ∈R C×C Represents a linear projection.
[0062] Step 3: Patchembed the input image and then perform linear projection on the sliced image as the query tag q, specifically:
[0063] Patchembed the image to obtain the feature map x∈R H×W×C (H, W, C represent the height, width, and number of channels of the feature map, respectively);
[0064] Generate a uniform reference grid on the feature map, denoted as The size of the reference grid is obtained by downsampling the input feature map, and the downsampling factor is r, that is, H G =H / r,W G =W / r; the points on the reference grid constitute the reference points. In order to obtain the offset of each reference point, the feature map is linearly projected to obtain the query mark q=xW q , where W q ∈R C×C Represents a linear projection.
[0065] Step 4: Input q into the offset network, calculate the offset, and obtain the deformation sampling point, specifically:
[0066] Step 4-1: Use the offset network to query features and output the offset values of the reference points respectively. Specifically:
[0067] The horizontal axis is moved one unit to the left and one unit to the right to form a horizontal offset, and the offset network is used to generate the offsets in the left and right vertical axes. The offset network uses the query features and outputs the offset values of the left and right reference points respectively. The area covered by each reference point is a local 1×s area, where s is the maximum value of the offset, so the offset generation network can also perceive the local features to learn a reasonable offset. Therefore, the offset network is set as two convolution modules with nonlinear activation. The input features are first captured through a 5×5 depth convolution. Then, GELU activation and 1×1 convolution are used to obtain the one-dimensional longitudinal offset;
[0068] Step 4-2: Input the query tag q into the offset network to obtain the deformation sampling point; specifically:
[0069] The query token q is passed through a lightweight offset network θ offset (·) is processed to generate the offset Δp=θ in the vertical axis direction offset(q), in the horizontal direction, the offset to the left and right is fixed by one unit, and the offsets in these two directions are combined to obtain the left and right offsets, that is, the horizontal direction is fixed to the left and right, and the corresponding vertical direction is deformable; in order to stabilize the training process, a predefined factor s is used to limit the scaling amplitude Δp of q, that is, Δp←stanh(Δp), and finally the offset values Δp on the left and right are generated respectively l and Δp r .
[0070] Step 5: Perform linear interpolation on the deformed sampling points to obtain sampling features, and obtain and Specifically:
[0071] Step 5-1: Apply the offset value to the reference point to get the deformation point. Figure 2 The offset of the reference points is shown. For clarity, only two reference points are shown. There are more points in the actual implementation. Sample features from the positions of the left and right deformation points to obtain the sampled feature map and Perform linear projection on the sampled feature map to obtain the deformed left query and the right query Expressed as:
[0072]
[0073] The sampling function Set it as linear interpolation, differentiable:
[0074] For all r ly Established;
[0075] For all r ry Established.
[0076] where g(a,b)=max(0,1-|ab|), r ly 、r ry Represents each point on the left and right vertical axes. l , r r Represents the left and right sides of each point on the horizontal axis, index z∈R H×W×C All positions on p l 、p r Represents the horizontal axis of the reference point after shifting to the left and right, respectively. g is only l , p r The integral points of are non-zero;
[0077] Step 5-2: Linearly project the deformation feature map to obtain the deformation left and right query:
[0078]
[0079] Where W q ∈R C×C Represents a linear projection.
[0080] Step 6: q, and Calculate the average value to get the final average query vector
[0081] An average query vector is generated based on the original input and the information of the sampled left and right positions. By calculating the average query vector, the model can simultaneously consider information at different positions instead of being limited to a fixed query.
[0082] Average query vector Determined by:
[0083]
[0084] Step 7: k,v are input into the multi-head attention, and the classification head outputs the classification result, which is:
[0085] Step 7-1: k,v are input into the multi-head attention module in the vision transformer, and finally the output of each head is obtained:
[0086]
[0087] where σ(·) represents the softmax function, d = C / M (C: number of channels, M: number of attention heads) is the dimension of each head; Z (m) represents the embedding output of the mth attention head, k (m) 、v (m) ∈R N×d represent query, key, and value embeddings respectively;
[0088] z=Concat(z( 1) ,…,z (M) )W o
[0089] The features of each head are connected together and passed through W o ∈R C×C to obtain the final output.
[0090] To construct the transformer block, an MLP (Multi-layer Perceptron) block with two linear transformations and GELU (Gaussian Error Linear Unit) activation is usually used to provide non-linearity; with LN (layer normalization) and multi-head deformation attention (DAT), the l-th layer transformer block is represented as:
[0091] z' l =DAT(LN(z l-1 ))+z l-1
[0092] z l =MLP(LN(z' l ))+z' l
[0093] Among them, z' l represents the intermediate features of the lth transformer block, that is, the result after DAT processing; z l-1 represents the input features of the lth transformer block; z l Represents the final output features of the lth transformer block.
[0094] To illustrate the interpretability of the model, Figure 3 It demonstrates that the algorithm can focus on important areas related to high myopia, such as the choroid and sclera.
[0095] The present invention discloses a classification method for multi-label myopia images. Using a bilateral deformable attention module, the method can model the characteristics of myopia images in a targeted manner, learn sparse attention patterns in a data-dependent manner, and limit the deformable attention to the left and right sides of each specific position, so that the model can focus on myopia-related areas. The method can learn several sets of query-agnostic offsets to shift the left and right queries to important areas, thereby enhancing the original self-attention module with higher flexibility and efficiency to capture more effective information, and ultimately be able to screen for high myopia more accurately.
[0096] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.
Claims
1. A multi-label close-up image classification method based on deformable self-attention and transformer, characterized in that: The method comprises: Step 1: Input retinal OCT image; Step 2: Put the input image into the pre-trained Resnet18 network to obtain the feature map, and perform linear projection on the feature map as the key k and value v; Step 3: Patchembed the input image and then perform linear projection on the sliced image as the query tag q; Step 4: Input q into the offset network, calculate the offset, and obtain the deformation sampling point; Step 5: Perform linear interpolation on the deformed sampling points to obtain the sampling features, and obtain the left query through linear projection and the right query Step 6: q, and Calculate the average value to get the final average query vector Step 7: k and v are input into the multi-head attention, and the classification head outputs the classification result.
2. The method according to claim 1, characterized in that Put the input image into the pre-trained Resnet18 network to get the feature map, and perform linear projection on the feature map as the key k and value v; include: The obtained feature map x f Linear projection of rows to get keys and values: k=x f W k ,v=x f W v , Among them, W k , W v ∈R C×C Represents a linear projection.
3. The method according to claim 1, characterized in that Step 3: Patchembed the input image and then perform linear projection on the sliced image as the query tag q, including: Patchembed the image to obtain the feature map x∈R H×W×C , H, W, and C represent the height, width, and number of channels of the feature map respectively; Generate a uniform reference grid on the feature map, denoted as The size of the reference grid is obtained by downsampling the input feature map, and the downsampling factor is r, that is, H G =H / r,W G =W / r; the points on the reference grid constitute the reference points. In order to obtain the offset of each reference point, the feature map is linearly projected to obtain the query mark q=xW q , where W q ∈R C×C Represents a linear projection.
4. The method according to claim 1, characterized in that: Input q into the offset network, calculate the offset, and get the deformation sampling point; including: Step 4-1: Use the query features through the offset network to output the offset values of the reference points respectively: The horizontal axis is shifted to the left and right by one unit to form a horizontal axis offset, and the offset network is used to generate the offsets in the left and right vertical axes. The offset network uses the query features and outputs the offset values of the left and right reference points respectively. The area covered by each reference point is a local s×s area. Among them, s is the maximum value of the offset, so the offset generation network can also perceive the local features to learn a reasonable offset. Therefore, the offset network is set as two convolution modules with nonlinear activation. The input features are first captured by a 5×5 depth convolution. Then, GELU activation and 1×1 convolution are used to obtain the one-dimensional longitudinal offset. Step 4-2: Input the query tag q into the offset network to obtain the deformation sampling point: The query token q is passed through a lightweight offset network θ offset (·) is processed to generate the offset Δp=θ in the vertical axis direction offset (q), in the horizontal direction, the offset to the left and right is fixed by one unit, and the offsets in these two directions are combined to obtain the left and right offsets, that is, the horizontal direction is fixed to the left and right, and the corresponding vertical direction is deformable; in order to stabilize the training process, a predefined factor s is used to limit the scaling amplitude Δp of q, that is, Δp←stanh(Δp), and finally the offset values Δp on the left and right are generated respectively l and Δp r .
5. The method according to claim 1, characterized in that Linear interpolation is performed on the deformed sampling points to obtain the sampling features, and the left query is obtained through linear projection. and the right query include: Step 5-1: Sample features from the positions of the left and right deformation points respectively to obtain the sampling feature map and Perform linear projection on the sampled feature map to obtain the deformed left query and the right query Expressed as: The sampling function Set it as linear interpolation, differentiable: For all r ly Establishment For all r ry Establishment where g(a,b)=max(0,1-|ab|), r ly 、r ry Represents each point on the left and right vertical axes. l , r r Represents the left and right sides of each point on the horizontal axis, index z∈R H×W×G All positions on p l 、p r Represents the horizontal axis of the reference point after shifting to the left and right, respectively. g is only l , p r The integral points of are non-zero; Step 5-2: Linearly project the deformation feature map to obtain the deformation left and right query: Where W q ∈R C×C Represents a linear projection.
6. The method according to claim 1, characterized in that Average query vector Determined by:
7. The method according to claim 1, characterized in that Will get k,v are input into the multi-head attention, and the classification head outputs the classification result; including: Step 7-1: Get k,v are input into the multi-head attention module in the vision transformer, and finally the output of each head is obtained: Where σ(·) represents the softmax function, d = C / M is the dimension of each head, C represents the number of channels, and M represents the number of attention heads; Z (m) represents the embedding output of the mth attention head, k (m) 、v (m) ∈R N×d represent query, key, and value embeddings respectively; from=Concat(from (1) ,…,With (M) )IN o The features of each head are connected together and passed through W o ∈R C×C to obtain the final output. An MLP block with two linear transformations and GELU activation is used to provide nonlinearity; with LN and multi-head deformation attention, the l-th layer transformer block is expressed as: z' l =DAT(LN(z l-1 ))+z l-1 With l =MLP(LN(z' l ))+z' l Among them, z' l represents the intermediate features of the lth transformer block, that is, the result after DAT processing; z l-1 represents the input features of the lth transformer block; z l Represents the final output features of the lth transformer block.
