A Transformer-based corneal neural image segmentation method and system

Abstract

This invention discloses a corneal neural image segmentation method and system based on Transformer. The method combines local feature extraction from CNNs with global feature extraction from Transformers, and constructs a corneal neural image segmentation network based on the ResUnet framework. This network includes a multi-scale deformation attention module, an external attention module, and a depthwise separable convolutional module. The multi-scale deformation attention module and the external attention module form a new cross-attention module. This invention combines the advantages of CNNs and Transformers to extract corneal neural features more comprehensively, solving the problems of fragmentation and noise cell interference in corneal neural image segmentation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image segmentation, and particularly relates to a corneal nerve image segmentation method and system based on a Transformer. BACKGROUND

[0002] In recent years, the rapid development of medical imaging technology has brought a revolution to the field of clinical medicine. Medical images usually provide clinicians with a wealth of information about biological or anatomical tissues; for example, morphological changes in the nerve fibers in the cornea are closely related to various systemic diseases and eye diseases (dry eye disease, corneal neuropathy, conjunctivitis, etc.). Among them, the extraction of features from corneal nerve images is conducive to further assisting in disease diagnosis, which plays a crucial role in accurate diagnosis and treatment. However, medical images often exhibit great quality differences, regardless of whether the same or different devices are used - exhibiting defects such as intensity inhomogeneity, low contrast, significant blurring or noise, etc., all of which can occur during image acquisition, such as Figure 1 Several low-quality corneal nerve images are shown in the above. These obstacles of low-quality corneal nerve images also directly affect the feature extraction of corneal nerves, further affecting the performance of many subsequent image analysis tasks, such as related disease diagnosis. Therefore, fully automatic and reliable corneal nerve image segmentation technology is of great importance to clinical diagnosis and intelligent medical diagnosis.

[0003] Prior to this, there have been some traditional image segmentation methods proposed. Early corneal nerve segmentation methods use manually crafted features to measure the tubularity of pixels and detect linear structures in an unsupervised setting, such as linear operators based on neighborhood difference, Frangi filtering based on eigenvalues of Hessian matrix, flux along different axes using gradient projection (OOF) and its measurement on the boundary of local spherical regions, etc.

[0004] Although the traditional methods have achieved certain results, there are still problems such as slow segmentation speed and low accuracy. In recent years, deep learning models represented by convolutional neural networks (CNN) have been developed in the field of medical image segmentation, and have made great progress and development. Typical representatives include the Unet network architecture.

[0005] Convolutional neural networks (CNN) extract features based on convolution kernels, so they have strong local feature extraction capabilities, but their modeling capabilities for global context dependencies are limited. In addition, Transformers have been continuously developed in the field of natural language processing (NLP) and extended to the field of computer vision, and because of their strong global context feature extraction capabilities, Transformers have made a big splash in the field of computer vision (CV).

[0006] How can we provide a corneal neural image segmentation scheme to solve the problems of fragmentation and interference from noisy cells in corneal neural image segmentation? Summary of the Invention

[0007] The main objective of this invention is to provide a corneal neural image segmentation method and system based on Transformer, thereby overcoming the shortcomings of the prior art.

[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: a corneal neural image segmentation method based on Transformer, comprising combining local feature extraction of CNN and global feature extraction of Transformer, and constructing a corneal neural image segmentation network based on the ResUnet framework. The corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module, and a depthwise separable convolution module, wherein the multi-scale deformation attention module and the external attention module constitute a new cross-attention module.

[0009] In a preferred embodiment, each head of the deformation attention module separately characterizes the attention weighting between queries and keys, and then the final output is obtained by linear weighting of the weights.

[0010] In a preferred embodiment, the calculation formula for the deformation attention module is:

[0011]

[0012] Among them, W′ m x k It is the encoding of the key element, A mqk It is the weight of the k-th key element, and there are a total of |Ω k | key value, Z q It is the feature vector of the query.

[0013] In a preferred embodiment, the formula for calculating the deformation attention is:

[0014]

[0015] Where m is the index of the attention head, k is the index of the sample key value, K is the total number of samples, and ΔP mqk andA mqk Let A represent the sample offset and the attention weight of the m-th head in the k-th sample, respectively. mqk The value range is [0, 1], after normalization

[0016] In a preferred embodiment, the calculation formula for the external attention module is:

[0017]

[0018] Among them, M k and M v There are two memory units, and F represents the input feature map.

[0019] In a preferred embodiment, the depth-separable convolution module employs a DW convolution module inserted between two 1*1 convolutions to perform convolution operations in each channel.

[0020] In a preferred embodiment, the method further includes label constraints for reducing the centerline-based label to the actual thickness of corneal nerves.

[0021] In a preferred embodiment, the label constraint adopts a coarse-to-fine training strategy. Specifically, the training strategy involves repeatedly inputting the first prediction result of the model's first output into the same model to obtain a second prediction result, constraining the first prediction result with 3*3 pixel labels to obtain a first constraint loss, constraining the second prediction result with 2*2 pixel labels to obtain a second constraint loss, and finally obtaining the total constraint loss by using the first constraint loss and the second constraint loss in a 6:4 ratio.

[0022] On the other hand, embodiments of the present invention also provide a corneal neural image segmentation system based on Transformer, including: a corneal neural image segmentation network, wherein the corneal neural image segmentation network is constructed based on the ResUnet framework by combining local feature extraction of CNN and global feature extraction of Transformer, and the corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module and a depthwise separable convolution module, wherein the multi-scale deformation attention module and the external attention module form a new cross-attention module.

[0023] In a preferred embodiment, the system further includes a training strategy module, used to repeatedly input the first prediction result (the output of the model's first prediction) into the same model to obtain a second prediction result, and to constrain the first prediction result with 3*3 pixel labels to obtain a first constraint loss.

[0024] The second constraint loss is obtained by constraining the second prediction result with 2*2 pixel labels. Finally, the total constraint loss is obtained by using the first constraint loss and the second constraint loss in a 6:4 ratio.

[0025] Compared with the prior art, the beneficial effects of the present invention are at least as follows: the present invention combines the advantages of CNN and Transformer to extract corneal neural features more comprehensively, which can solve the problems of fragmentation and noise cell interference in corneal neural image segmentation. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of existing low-quality corneal nerve images;

[0028] Figure 2 This is a schematic diagram of the overall model architecture of the corneal neural image segmentation network in one embodiment of the present invention;

[0029] Figure 3 This is the training strategy for different label constraints in this invention. Detailed Implementation

[0030] The invention will be more fully understood through the following detailed description, which should be read in conjunction with the accompanying drawings. Detailed embodiments of the invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. Therefore, the specific functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as intended to teach those skilled in the art to employ the representative basis of the invention in different ways in any suitable detailed embodiment.

[0031] The present invention discloses a corneal nerve image segmentation method and system based on Transformer. It addresses the problems of low contrast and background noise interference in the corneal nerve image segmentation process, which lead to segmentation breaks and discontinuities, and incorrect segmentation. It combines the advantages of CNN and Transformer to extract corneal nerve features more comprehensively, and can be used for accurate extraction of corneal nerves and intelligent medical diagnosis.

[0032] Combination Figure 2As shown in the embodiment of the present invention, a corneal neural image segmentation method based on Transformer is disclosed, which specifically includes: combining local feature extraction of CNN and global feature extraction of Transformer, and constructing a corneal neural image segmentation network based on the ResUnet framework. The corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module and a depthwise separable convolution module. The multi-scale deformation attention module and the external attention module form a new cross-attention module.

[0033] Specifically, this invention combines the powerful local feature extraction capabilities of CNNs and the powerful global feature extraction capabilities of Transformers, and constructs a corneal neural image segmentation network based on the ResUnet framework, such as... Figure 2 As shown, the corneal neural image segmentation network consists of three parts: a multi-scale deformable attention module, an external attention module, and a depth-wise block module. Furthermore, the deformable attention and external attention modules form a new cross-attention module.

[0034] For the multi-scale deformation attention module, the multi-head attention mechanism in Transformers separately characterizes the attention weighting between queries and keys, and then obtains the final output through linear weighting of the weights, as calculated below:

[0035]

[0036] Among them, W′ m x k It is the encoding of the key element, A mqk It is the weight of the k-th key element, and there are a total of |Ω k | key values, so A mqk It is a normalized weight vector, proportional to Z q That is, the feature vector of the query. To distinguish different positions, Z... q X k Generally, it is an aggregation of content features and location encoding. In this attention method, the number of queries and keys significantly impacts the computation and storage of the multi-head attention module; the computational complexity in formula (1) is... For images, N q N kThese can be considered as image pixels, so the computational complexity of the multi-head attention mechanism is approximately the square of the feature map size.

[0037] Inspired by deformable convolution, the deformable attention module focuses only on a small set of key sampling points around the reference point, without considering the spatial size of the feature map. This reduces the complexity of the original Transformer module and directs attention to extracting relatively important information locations through learnable offsets.

[0038] Given the input feature map x∈R C×H×W q is the content feature Z q An index, p q It is a two-dimensional reference point, and the deformation attention is calculated as shown in formula (2):

[0039]

[0040] Where m is the index of the attention head, k is the index of the sample key value, and K is the total number of samples. ΔP mqk andA mqk Let Z represent the sample offset and the attention weight of the m-th head in the k-th sample, respectively, and both have a feature Z. q A is obtained through linear mapping. mqk The value range is [0, 1], after normalization This attention complexity, which combines deformation convolution, is O(2N). q C 2 Its computational complexity is reduced from the original O(N) 2 The complexity is reduced to O(N).

[0041] For the external attention module, the original Transformer self-attention mechanism only learns the features of a single sample, ignoring the mutual influence of features between samples in the global dataset. To overcome the shortcomings of the original attention module, this invention introduces an external attention module. This module learns the features between different samples through learnable external units (key and value), thus simplifying the time complexity of the original Transformer to a linear relationship with the number of pixels. Furthermore, since the key and value are two learnable external units shared across the entire dataset, the correlation between different samples can also be implicitly considered. These two units are implemented as two linear layers, allowing for direct end-to-end optimization. This invention utilizes two such memory units as M... k and M v This increases the network's capacity. Therefore, the entire external attention calculation formula is as follows:

[0042]

[0043] F out =AM v (3)

[0044] This invention, by incorporating an external attention module, not only reduces time complexity but also enhances the model's ability to resist background cell interference by mining similar features of corneal nerves in the entire dataset and the mutual constraints of these similar features, thereby improving the accuracy of corneal nerve image segmentation.

[0045] For the depthwise separable convolutional module, this invention adds two fully connected linear layers after self-attention to further enhance the model's expressive power. However, this dimensionality increase and decrease operation is computationally expensive. To reduce computation, a 1x1 convolution is considered, allowing for dimensionality increase and decrease without increasing computation. Furthermore, a DW module is added between the two 1x1 convolutions, performing convolution operations in each channel to further extract local features. DW is also a lightweight feature extraction module, further reducing computation. Therefore, replacing the original fully connected linear layers in the Transformer with 1x1 convolutions combined with DW convolutions effectively compensates for the Transformer's insufficient local feature extraction capability.

[0046] In summary, the deformation attention module can learn the features of a single sample, while the external attention module can integrate the influence of similar features among samples in the entire dataset. The features learned from a single sample are input into the external attention module as Q (query), and the outputs of these two modules are concatenated as input to the subsequent depthwise separable convolutional module. This achieves better fusion of features within a single sample and features between different samples. This allows the model to learn the corneal nerve features in the samples more comprehensively, thus achieving better segmentation results.

[0047] Preferably, the present invention also includes label constraints for reducing the difference between centerline-based labels and the actual thickness of corneal nerves. Specifically, due to the difficulty and time-consuming nature of corneal nerve labeling, existing datasets only have centerline-based labels, which differ significantly from the actual thickness of corneal nerves. To reduce this error, existing methods expand the labels to 2*2 pixels or 3*3 pixels, which reduces the error to some extent. This invention employs a coarse-to-fine training strategy, repeatedly inputting the model's initial output, pred1 (i.e., the first prediction result), into the same model to obtain pred2 (i.e., the second prediction result). Pred1 (i.e., the first prediction result) is constrained with 3*3 pixel labels to obtain loss1 (i.e., the first constraint loss), and pred2 (i.e., the second prediction result) is constrained with 2*2 pixel labels to obtain loss2 (i.e., the second constraint loss). Finally, the total loss is obtained by combining loss1 and loss2 in a 6:4 ratio. In other embodiments, the 6:4 ratio can be extended to other ranges, requiring adjustment based on the training process to select the optimal ratio. Figure 3 As shown, this coarse-to-fine label constraint training strategy can better reduce the defects of the labels themselves, thereby improving the performance of the model.

[0048] Corresponding to the above-mentioned Transformer-based corneal neural image segmentation method, this invention also provides another Transformer-based corneal neural image segmentation system, including: a corneal neural image segmentation network, which combines local feature extraction of CNN and global feature extraction of Transformer, and is constructed based on the ResUnet framework. The corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module, and a depthwise separable convolution module. The multi-scale deformation attention module and the external attention module form a new cross-attention module.

[0049] In addition, the system also includes a training strategy module, which is used to repeatedly input the first prediction result from the model's first output into the same model to obtain a second prediction result. The first prediction result is constrained with 3*3 pixel labels to obtain a first constraint loss, and the second prediction result is constrained with 2*2 pixel labels to obtain a second constraint loss. Finally, the total constraint loss is obtained by combining the first and second constraint losses in a 6:4 ratio. The specific principles of each module can be found in the corresponding descriptions in the above methods, and will not be elaborated here.

[0050] All aspects, embodiments, features, and examples of this invention are to be regarded as illustrative in all respects and are not intended to limit the invention, the scope of which is defined only by the claims. Other embodiments, modifications, and uses will become apparent to those skilled in the art without departing from the spirit and scope of the invention as claimed.

[0051] The use of headings and sections in this invention is not intended to limit the invention; each section can be applied to any aspect, embodiment or feature of the invention.

Claims

1. A corneal neural image segmentation method based on Transformer, characterized in that, The method includes: combining local feature extraction from CNNs and global feature extraction from Transformers, and constructing a corneal neural image segmentation network based on the ResUnet framework. The corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module, and a depthwise separable convolutional module. The multi-scale deformation attention module and the external attention module form a new cross-attention module. Each head of the deformation attention module individually characterizes the attention weighting between queries and keys, and then the final output is obtained through linear weighting. The calculation formula for the deformation attention module is: Among them, W m 'x k It is the encoding of the key element, A mqk It is the weight of the k-th key element, and there are a total of |Ω k | key value, Z q It is the feature vector of the query; The formula for calculating the deformation attention is: Where m is the index of the attention head, k is the index of the sample key value, K is the total number of samples, and ΔP mqk andA mqk Let A represent the sample offset and the attention weight of the m-th head in the k-th sample, respectively. mqk The value range is [0,1], after normalization The calculation formula for the external attention module is as follows: Among them, M k and M v There are two memory units, where F represents the input feature map; The depth-separable convolution module employs a DW convolution module inserted between two 1*1 convolutions to perform convolution operations in each channel.

2. The corneal neural image segmentation method based on Transformer according to claim 1, characterized in that: The method also includes label constraints for reducing the centerline-based label from the actual thickness of corneal nerves.

3. The corneal neural image segmentation method based on Transformer according to claim 2, characterized in that: The label constraint adopts a coarse-to-fine training strategy. Specifically, the first prediction result of the model's first output is repeatedly input into the same model to obtain a second prediction result. The first prediction result is constrained with 3*3 pixel labels to obtain a first constraint loss. The second prediction result is constrained with 2*2 pixel labels to obtain a second constraint loss. Finally, the total constraint loss is obtained by combining the first constraint loss and the second constraint loss in a 6:4 ratio.

4. A corneal neural image segmentation system based on Transformer, characterized in that, The system includes a corneal neural image segmentation network, which combines local feature extraction from CNNs and global feature extraction from Transformers, and is constructed based on the ResUnet framework. The corneal neural image segmentation network includes a multi-scale deformation attention module, an external attention module, and a depthwise separable convolutional module. The multi-scale deformation attention module and the external attention module form a new cross-attention module. Each head of the deformation attention module individually characterizes the attention weighting between queries and keys, and then the final output is obtained through linear weighting. The calculation formula for the deformation attention module is: Among them, W m 'x k It is the encoding of the key element, A mqk It is the weight of the k-th key element, and there are a total of |Ω k | key value, Z q It is the feature vector of the query; The formula for calculating the deformation attention is: Where m is the index of the attention head, k is the index of the sample key value, K is the total number of samples, and ΔP mqk andA mqk Let A represent the sample offset and the attention weight of the m-th head in the k-th sample, respectively. mqk The value range is [0,1], after normalization The calculation formula for the external attention module is as follows: Among them, M k and M v There are two memory units, where F represents the input feature map; The depth-separable convolution module employs a DW convolution module inserted between two 1*1 convolutions to perform convolution operations in each channel.

5. A corneal neural image segmentation system based on Transformer according to claim 4, characterized in that: The system also includes a training strategy module, which is used to repeatedly input the first prediction result of the model's first output as input into the same model to obtain a second prediction result, constrain the first prediction result with 3*3 pixel labels to obtain a first constraint loss, constrain the second prediction result with 2*2 pixel labels to obtain a second constraint loss, and finally obtain the total constraint loss by using the first constraint loss and the second constraint loss in a 6:4 ratio.

Images