A method for constructing a category-based attention network for retinitis pigmentosa grading

Abstract

The present invention relates to the field of image processing technology, and in particular to a method for constructing a category attention network for grading retinitis pigmentosa. The present invention constructs a category attention network for retinitis pigmentosa, and the category attention network for retinitis pigmentosa includes: two parallel branches, a category weighted attention module and a classifier; the two parallel branches include: an efficient self-attention Transformer branch and a local feature extraction branch; the fundus color photo image is respectively input into the efficient self-attention Transformer branch and the local feature extraction branch, the output features of the efficient self-attention Transformer branch and the output features of the local feature extraction branch are extracted and spliced ​​along the channel dimension to obtain fused features; the fused features are passed through the category weighted attention module to assign different weights to different categories to obtain weighted output features; the weighted output features are passed through the classifier to output the grading results of retinitis pigmentosa. The present invention makes the network more lightweight and more accurate.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for constructing a category attention network for grading retinitis pigmentosa. Background Art

[0002] Retinitis Pigmentosa (RP) is a rare hereditary ophthalmic disease. RP is mainly caused by gene mutations related to photoreceptors. The affected photoreceptor cells gradually degenerate, leading to visual impairment or even blindness. With the increasing aging of society and the deepening of genetic disease research, the number of blindness cases related to RP has increased significantly worldwide. In the field of ophthalmology, color fundus photography (CFP) is a widely used diagnostic technology. Fundus color photography technology helps clinicians observe the detailed changes of the retina, optic disc, macula and blood vessels by non-invasively taking high-resolution color images of the fundus, facilitating early detection and management of various ophthalmic diseases, especially in the grading diagnosis of RP. Fundus color photography technology is cost-effective and easy to operate, and has been widely used in the screening and diagnosis of ophthalmic diseases. Doctors determine the stage and progression of RP by observing the changes in retinal pigment deposition, vascular structure and optic disc, but it only relies on the subjective judgment of doctors, which makes the judgment of the grade of RP have certain errors and uncertainties.

[0003] In recent years, methods based on deep learning (DL) have been widely used in the field of medical image processing and analysis. Convolutional Neural Network (CNN) and Transformer have shown great application potential in many medical image processing and analysis tasks. With the development of these technologies, the accuracy and efficiency of medical image processing and analysis have been significantly improved. They have shown great potential in improving the accuracy and efficiency of tasks such as image processing, classification, and segmentation, and have promoted the development of medical image analysis towards automation and intelligence. Through these technologies, researchers can use large-scale medical image data for analysis, greatly improving the screening and diagnosis efficiency of ophthalmic diseases.

[0004] Although deep learning technology has made great progress in many medical fields, its application in the automatic grading of retinitis pigmentosa is still in a relatively preliminary stage. On the one hand, the existing automated models are mostly trained based on large-scale data sets. However, since retinitis pigmentosa is a rare disease, relevant high-quality data sets are relatively scarce. Due to the limitation of data volume, the existing deep learning models are prone to overfitting during the training process, which in turn affects the generalization ability of the model. This makes the model unstable when facing different characteristics and different types of patients, resulting in challenges in the development and optimization of the model. On the other hand, the different grades of retinitis pigmentosa show obvious differences in features on fundus color photos, but the category distribution is extremely unbalanced. Due to the small number of images of patients with middle and late stage retinitis pigmentosa, the existing models often over-learn the features of categories with large sample sizes during training and ignore the categories with fewer samples. This unbalanced data distribution problem greatly affects the accuracy of the model when dealing with grading tasks in actual scenarios, making it difficult to take into account the performance of each grade, resulting in low grading accuracy and difficulty in fully adapting to patients with retinitis pigmentosa at different stages. In addition, the field of medical image analysis has high requirements for the real-time performance and deployment flexibility of the model, especially in medical equipment with limited hardware resources. Existing deep learning models, especially large convolutional neural networks (CNN) or Transformer models, have excellent accuracy, but due to their high computational complexity, they cannot be efficiently run directly on resource-constrained devices. Therefore, how to achieve lightweight while maintaining model accuracy is a major challenge facing existing technologies. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the defects of the existing retinitis pigmentosa dataset, which is scarce, unbalanced in categories, and the inability of the retinitis pigmentosa model to run efficiently directly on resource-constrained devices.

[0006] In order to solve the above technical problems, the present invention provides a method for constructing a category attention network for grading retinitis pigmentosa, comprising the following steps:

[0007] Constructing a retinitis pigmentosa category attention network, the retinitis pigmentosa category attention network comprising: two parallel branches, a category weighted attention module and a classifier; the two parallel branches comprising: an efficient self-attention Transformer branch and a local feature extraction branch;

[0008] Input the fundus color image into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch;

[0009] Inputting the fundus color photo image into the local feature extraction branch, extracting local information, and obtaining output features of the local feature extraction branch;

[0010] The output features of the efficient self-attention Transformer branch are concatenated with the output features of the local feature extraction branch along the channel dimension to obtain the fused features;

[0011] The fusion features are passed through the category-weighted attention module to assign different weights to different categories, and the weighted output features are obtained, including:

[0012] The fused features are passed through the global feature extraction component to extract channel weights and spatial attention weights to obtain global features;

[0013] The global features are passed through the category weighting component to assign independent weights to each category, and the weighted output features are obtained, including:

[0014] The global features are processed by a 1×1 convolution layer and the number of channels of the global features is adjusted to ;

[0015] The feature map after the number of channels is adjusted is divided into category feature maps, and calculate the weight score of each category feature map, where is the number of retinitis pigmentosa categories, The number of channels for each category feature map;

[0016] right Each category feature map is average pooled to obtain the semantic feature map corresponding to each category feature map;

[0017] The weight score of each category feature map is weighted averaged with its corresponding semantic feature map to obtain the category weighted attention matrix;

[0018] Multiply the global feature by the category-weighted attention matrix element by element to obtain the weighted output feature;

[0019] The weighted output features are passed through a classifier to output the prediction results of retinitis pigmentosa grading.

[0020] Preferably, the efficient self-attention Transformer branch comprises: a patch block module and a plurality of feature extraction modules connected in series along the forward propagation direction, wherein the first feature extraction module comprises: a linear embedding layer and an efficient Transformer module connected in sequence, and the other feature extraction modules except the first feature extraction module all comprise a linear merging layer and an efficient Transformer module connected in sequence;

[0021] Among them, by replacing the global multi-head attention mechanism in the visual converter ViT with an efficient self-attention mechanism, an efficient Transformer module is obtained, including: deep convolutional layers, layer normalization, efficient self-attention mechanisms, layer normalization, and fully connected layers connected in sequence along the forward propagation direction, and the features output by the deep convolutional layer are residually connected with the output features of the efficient self-attention mechanism.

[0022] Preferably, the step of inputting the fundus color photograph image into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch includes:

[0023] The fundus color photo image is divided into blocks through the patch block module to obtain multiple feature maps of the same size;

[0024] After each feature map is converted into its corresponding one-dimensional sequence, all the one-dimensional sequences are concatenated to obtain the input sequence of the first feature extraction module;

[0025] After the input sequence of the first feature extraction module is dimensionally increased by the linear embedding layer of the first feature extraction module, the first output feature is extracted by the efficient Transformer module of the first feature extraction module;

[0026] After the first output feature is linearly transformed by the linear merging layer of the second feature extraction module, the global information is captured by the efficient Transformer module of the second feature extraction module to obtain the second output feature, and the second output feature is used as the input of the next feature extraction module;

[0027] The output features of the last feature extraction module are used as the output features of the efficient self-attention Transformer branch.

[0028] Preferably, the input features of the efficient Transformer module are passed through the efficient Transformer module, and the attention weights are adjusted to obtain the output features of the efficient Transformer module, including:

[0029] After the input features of the efficient Transformer module are processed by deep convolution layer and layer normalization, they are processed by the efficient self-attention mechanism to obtain the output features of the efficient self-attention mechanism, including:

[0030] The feature map after layer normalization is reshaped through the reshape operation to obtain the reshaped features, where the reshaped features include Patch blocks, The number of patches that the feature map after layer normalization is divided into in height or width;

[0031] Multiply the reshaped features with three different linear projection layers to obtain the query vector, key vector and value vector;

[0032] Perform average operations on each patch block in the query vector and key vector respectively to obtain the average features of the query vector and the average features of the key vector;

[0033] Calculate the attention score of the average features of the query vector and the average features of the key vector to obtain the feature association graph;

[0034] The feature association graph is used to select features through the Topk algorithm to obtain a sparse index matrix;

[0035] Merge the key vector and the value vector with the sparse index matrix respectively to obtain the merged key vector and value vector;

[0036] After calculating the similarity by performing a dot product between the query vector and the merged key vector, the similarity is divided by the square root of the dimension of the key vector to obtain the attention weight;

[0037] Normalize the attention weights through the Softmax operation to obtain the attention weight matrix;

[0038] The value vector is weighted and summed according to the attention weight matrix to obtain the output features of the efficient self-attention mechanism;

[0039] After residual connection is performed on the output features of the efficient self-attention mechanism and the features output by the deep convolutional layer, they are processed by layer normalization and fully connected layers in turn to obtain the output features of the efficient Transformer module.

[0040] Preferably, the fusion feature is passed through a global feature extraction component to extract channel weights and spatial attention weights to obtain global features, including:

[0041] Pass the fused features through the channel weighting unit, extract the channel weights, and obtain the intermediate feature map;

[0042] The intermediate feature map is passed through the spatial weighted unit to extract the spatial attention weights and obtain the global features.

[0043] Preferably, the fusion feature is passed through a global feature extraction component to extract channel weights and spatial attention weights to obtain global features, including:

[0044] The fused features are channel compressed through the global average pooling operation to obtain the pooled features;

[0045] After the pooled features pass through the convolution layer with a 1×1 convolution kernel, the channel weight is obtained through the Sigmoid activation function;

[0046] Multiply the channel weight and the fusion feature element by element to obtain the intermediate feature map;

[0047] After average pooling the intermediate feature map along the channel direction, the spatial attention weight is obtained through the Sigmoid activation function;

[0048] The spatial attention weights are element-wise multiplied with the intermediate feature map to obtain the global features.

[0049] Preferably, the feature map after the number of channels is adjusted is divided into Class feature maps are generated, and the weight score of each class feature map is calculated using the following formula:

[0050] ,

[0051] The pair The category feature maps are averaged and pooled respectively to obtain the semantic feature map corresponding to each category feature map. The calculation formula is:

[0052] ,

[0053] The weight score of each category feature map is weighted averaged with its corresponding semantic feature map to obtain a category weighted attention matrix, and the calculation formula is:

[0054] ,

[0055] The global feature is multiplied element by element with the category weighted attention matrix to obtain the weighted output feature, and the calculation formula is:

[0056] ,

[0057] in, Indicates The weight score of the class feature map, Indicates the number of channels of each category feature map, represents the global maximum pooling, Indicates In the category feature map The value of the channel, Indicates The semantic feature map corresponding to the category feature map, represents the category-weighted attention matrix, represents the weighted output features, Represents global features, Represents element-wise multiplication.

[0058] Preferably, the local feature extraction branch is any one of a MobileNet convolution branch, a GoogleNet convolution branch, and a DenseNet121 convolution branch.

[0059] Preferably, the fundus color photograph image is input into the convolution branch of DenseNet121, local information is extracted, and output features of the convolution branch of DenseNet121 are obtained;

[0060] Among them, the convolution branch of DenseNet121 includes: the initial convolution layer and dense blocks of different convolutional layers, There are feature connection layers, each of which includes a 1x1 convolution and a 2x2 average pooling layer connected in sequence.

[0061] Preferably, the training process of the retinitis pigmentosa class attention network is:

[0062] Obtaining a fundus color photo dataset, amplifying the data of the fundus color photo dataset, and then preprocessing it, and dividing the preprocessed fundus color photo dataset into a training set, a validation set, and a test set;

[0063] The training set is used to train the retinitis pigmentosa category attention network, and the progressive coding strategy is used to construct the total loss function to optimize the retinitis pigmentosa category attention network. The total loss function includes: the loss function of the standard coding and the progressive loss function used in the progressive coding. The trained retinitis pigmentosa category attention network is verified by the validation set to obtain the target retinitis pigmentosa category attention network.

[0064] The formula for the total loss function is:

[0065] ,

[0066] ,

[0067] ,

[0068] in, represents the total loss function, represents the loss function of standard encoding, represents the progressive loss function used by progressive coding, Indicates the total number of samples in the current batch, Indicates One-Hot encoding of the true label of the sample, if the The samples belong to retinitis pigmentosa type, is 1, otherwise it is 0. Represents the retinitis pigmentosa category attention network prediction The samples belong to The probability of the retinitis pigmentosa category, Indicates The progressively encoded labels of samples.

[0069] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0070] The method for constructing a category attention network for grading retinitis pigmentosa described in the present invention comprises the following steps: splicing the output features of the efficient self-attention Transformer branch and the output features of the local feature extraction branch along the channel dimension to obtain fused features; extracting channel weights and spatial attention weights from the fused features through the global feature extraction component of the category weighted attention module to obtain global features; passing the global features through the category weighted component of the category weighted attention module to assign independent weights to each category to obtain weighted output features, and outputting the weighted output features through the classifier to output the prediction results of grading retinitis pigmentosa. The category weighted attention module can integrate the global information of the fused features by extracting the channel weights and spatial attention weights, and by assigning different weights to each category in the global features, it is ensured that the category attention network of retinitis pigmentosa can pay balanced attention to samples of different categories without being affected by the number of samples, thereby improving the recognition ability of the category attention network for category features with a small number of samples.

[0071] In addition, the efficient self-attention Transformer branch divides the fundus color photo images into blocks through the patch block module, which is convenient for subsequent feature extraction and learning, and reduces the spatial resolution of the image through multiple feature extraction modules, while extracting richer feature information, gradually improving the feature extraction ability of the model, and obtaining the output characteristics of the efficient self-attention Transformer branch; the efficient self-attention Transformer branch uses an improved efficient self-attention mechanism to significantly reduce the number of parameters and computational complexity of the retinitis pigmentosa category attention network while retaining the high accuracy of the retinitis pigmentosa category attention network, thereby ensuring the lightweight and high performance of the retinitis pigmentosa category attention network. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0073] Figure 1 This is the attention network structure diagram of the retinitis pigmentosa category.

[0074] Figure 2This is a flowchart of the steps of a method for constructing a category attention network for grading retinitis pigmentosa according to the present invention.

[0075] Figure 3 This is the structural diagram of the efficient Transformer module.

[0076] Figure 4 This is the structural diagram of the efficient self-attention mechanism in the efficient Transformer module.

[0077] Figure 5 This is the structural diagram of the dense block in the convolution branch of DenseNet121.

[0078] Figure 6 It is the structural diagram of the category weight attention module.

[0079] Figure 7 It is a structural diagram of the global feature extraction component in the category weighted attention module.

[0080] Figure 8 It is a structural diagram of the category weighted component in the category weighted attention module. DETAILED DESCRIPTION

[0081] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0082] Reference Figure 2 As shown, the first embodiment of the present invention provides a method for constructing a category attention network for grading retinitis pigmentosa, comprising the following steps:

[0083] Step S1: Construct the attention network for the retinitis pigmentosa category, such as Figure 1 As shown, Figure 1 The structure diagram of the attention network of the retinitis pigmentosa category includes: two parallel branches, a category weighted attention module and a classifier; the two parallel branches include: an efficient self-attention Transformer branch and a local feature extraction branch;

[0084] Step S2: Input the fundus color photo image into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch;

[0085] In this embodiment, preferably, the efficient self-attention Transformer branch includes: a patch block module and multiple feature extraction modules connected in series along the forward propagation direction, wherein the first feature extraction module includes: a linear embedding layer and an efficient Transformer module connected in sequence, and the other feature extraction modules except the first feature extraction module include a linear merging layer and an efficient Transformer module connected in sequence;

[0086] like Figure 3 As shown, Figure 3 : It is a structural diagram of the efficient Transformer module, wherein the efficient Transformer module is obtained by replacing the global multi-head attention mechanism in the visual converter ViT with an efficient self-attention mechanism, including: deep convolutional layers, layer normalization, efficient self-attention mechanisms, layer normalization, and fully connected layers connected in sequence along the forward propagation direction, and the features output by the deep convolutional layers are residually connected with the output features of the efficient self-attention mechanism.

[0087] The global multi-head attention mechanism in the traditional visual converter ViT is replaced by efficient self-attention to achieve a lightweight effect. For the input features of a given efficient Transformer module, a deep convolutional layer is first used to implicitly encode the position information, followed by layer normalization, efficient attention mechanism, layer normalization, and fully connected layers; among them, the features output by the deep convolutional layer are residually connected with the output features of the efficient self-attention mechanism, so that the network can obtain more information to prevent overfitting.

[0088] In this embodiment, preferably, the fundus color photo image is input into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch, including:

[0089] The fundus color photo image is divided into blocks through the patch block module to obtain multiple feature maps of the same size;

[0090] After each feature map is converted into its corresponding one-dimensional sequence, the input sequence of the first feature extraction module is obtained by concatenating them;

[0091] The input sequence of the first feature extraction module is projected to a higher dimension through the linear embedding layer of the first feature extraction module, and then the first output feature is extracted through the efficient Transformer module of the first feature extraction module;

[0092] After the first output feature is linearly transformed by the linear merging layer of the second feature extraction module, the global information is captured by the efficient Transformer module of the second feature extraction module to obtain the second output feature, and the second output feature is used as the input of the next feature extraction module;

[0093] The output features of the last feature extraction module are used as the output features of the efficient self-attention Transformer branch.

[0094] like Figure 4 As shown, Figure 4 : is a structural diagram of the efficient self-attention mechanism in the efficient Transformer module. In this embodiment, preferably, the input features of the efficient Transformer module are passed through the efficient Transformer module, the attention weight is adjusted, and the output features of the efficient Transformer module are obtained, including:

[0095] After the input features of the efficient Transformer module are processed by deep convolution layer and layer normalization, they are processed by the efficient self-attention mechanism to obtain the output features of the efficient self-attention mechanism, including:

[0096] The feature map after layer normalization Reshape through the reshape operation to obtain the reshaped features , where the reshaped features include patch blocks, each patchtoken contains feature vectors, is the number of patches that the feature map after layer normalization is divided into in height or width. Represents the height of the feature map after layer normalization. The width of the feature map after normalization of the representation layer, The number of channels of the feature map after layer normalization;

[0097] The reshaped features are respectively combined with three different linear projection layers Multiply them together to get the query vector , key vector With value vector , and its characteristic dimensions are ;

[0098] Perform average operations on each patch block in the query vector and key vector respectively to obtain the average features of the query vector , the average characteristics of the key vector , ;

[0099] Calculate the attention score of the average feature of the query vector and the average feature of the key vector to obtain the feature association graph , the calculation formula is:

[0100] ,

[0101] in, represents the feature association graph, represents the average feature of the query vector, represents the average characteristics of the key vector, Indicates transpose.

[0102] The feature association graph is used to select features through the Topk algorithm to obtain a sparse index matrix ;

[0103] The calculation formula of the sparse index matrix is:

[0104] ,

[0105] in, represents a sparse index matrix, represents the Topk algorithm, Represents a feature association graph.

[0106] Merge the key vector and value vector with the sparse index matrix respectively to obtain the merged key vector and value vector. The formula is:

[0107] ,

[0108] in, Represents a merge operation on vectors.

[0109] The query vector With the merged key vector After performing the dot product to calculate the similarity, the similarity is divided by the square root of the dimension of the key vector to obtain the attention weight;

[0110] Normalize the attention weights through the Softmax operation to obtain the attention weight matrix;

[0111] According to the attention weight matrix Weighted summation, the attention weights are normalized through the Softmax operation to obtain the attention weight matrix;

[0112] The value vector is weighted and summed according to the attention weight matrix to obtain the output features of the efficient self-attention mechanism , the formula is:

[0113] ,

[0114] in, represents the output features of the efficient self-attention mechanism, represents the Softmax operation to normalize it. is the square root of the key vector dimension. In order to prevent the dot product result from being too large, the similarity is usually divided by the square root of the dimension of each attention head to avoid the gradient explosion problem caused by too large attention weights.

[0115] After residual connection is performed on the output features of the efficient self-attention mechanism and the features output by the deep convolutional layer, they are processed by layer normalization and fully connected layers in turn to obtain the output features of the efficient Transformer module.

[0116] The dimension size is Fundus color images Input the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch, where and Represents fundus color images The height and width of 3 represents the fundus color image The number of channels.

[0117] Fundus color image The image is divided into multiple non-overlapping small blocks (patches) through the patch block module. Each patch block is regarded as an independent "token" that represents the pixel information of the area. This process converts the spatial information of the two-dimensional image into a one-dimensional sequence that can be processed by the efficient Transformer module, facilitating subsequent feature extraction and learning. This embodiment uses a 4×4 patch block size, and the feature dimension of each block is 4 × 4 ×3 = 48. By splicing these one-dimensional sequences, the input sequence of the first feature extraction module is obtained for subsequent processing. The input sequence of the first feature extraction module The dimension is .

[0118] The input sequence of the first feature extraction module Through the linear embedding layer of the first feature extraction module, the dimension is projected to a higher dimension and the dimension size is Output features of the linear embedding layer , Represents the output features of the linear embedding layer The number of channels;

[0119] The output features of the linear embedding layer Through the efficient Transformer module of the first feature extraction module, the dimension size is The first output feature ;

[0120] The first output feature The linear transformation is performed through the linear merging layer of the second feature extraction module to reduce the number of tokens, reduce the spatial resolution of the image, and extract richer feature information, resulting in a dimension size of The output features of the linear merging layer of the second feature extraction module ;

[0121] The output features of the linear merging layer of the second feature extraction module Through the efficient Transformer module of the second feature extraction module, the dimension size is The second output feature ;

[0122] The second output feature The linear merging layer of the third feature extraction module is used for linear transformation to reduce the number of tokens, reduce the spatial resolution of the image, and extract richer feature information, resulting in a dimension size of The output features of the linear merging layer of the third feature extraction module ;

[0123] The output features of the linear merging layer of the third feature extraction module Through the efficient Transformer module of the third feature extraction module, the dimension size is The third output feature of ;

[0124] The third output feature The linear merging layer of the fourth feature extraction module is used for linear transformation to reduce the number of tokens, reduce the spatial resolution of the image, and extract richer feature information, resulting in a dimension size of The output features of the linear merging layer of the fourth feature extraction module ;

[0125] The output features of the linear merging layer of the fourth feature extraction module Through the efficient Transformer module of the fourth feature extraction module, the dimension size is The fourth output feature , the fourth output feature Serves as the output features of an efficient self-attention Transformer branch.

[0126] By designing an efficient self-attention mechanism, the number of parameters and computations are significantly reduced while retaining the accuracy of the attention network for the retinitis pigmentosa category. This efficient design makes the attention network for the retinitis pigmentosa category suitable for deployment in environments with limited hardware resources, such as embedded medical devices. Compared with traditional deep learning architectures, this efficient self-attention can reduce computational complexity while ensuring the accuracy of feature extraction, taking into account both the lightweight and high performance of the model.

[0127] Step S3: inputting the fundus color photo image into the local feature extraction branch, extracting local information, and obtaining output features of the local feature extraction branch;

[0128] In this embodiment, specifically, the local feature extraction branch is any one of the MobileNet convolution branch, the GoogleNet convolution branch, and the DenseNet121 convolution branch.

[0129] In this embodiment, preferably, the fundus color photograph image is input into the convolution branch of DenseNet121, local information is extracted, and output features of the convolution branch of DenseNet121 are obtained;

[0130] Among them, the convolution branch of DenseNet121 includes: the initial convolution layer and dense blocks of different convolutional layers, There are feature connection layers, each of which includes a 1x1 convolution and a 2x2 average pooling layer connected in sequence.

[0131] The fundus color image is input into the DenseNet121 convolution branch to extract local information and obtain the output features of the DenseNet121 convolution branch, including:

[0132] The fundus color image is passed through the initial convolution layer to extract the initial edge and texture features to obtain image features;

[0133] The image features are passed through the first dense block, and the dense connection mechanism is used to enhance the feature reuse capability to obtain the output features of the first dense block. The output features of the first dense block are passed through the first feature connection layer, which performs channel compression through 1x1 convolution and further reduces the spatial resolution through 2x2 average pooling to obtain the output features of the first feature connection layer;

[0134] The first The output features of the feature connection layer are connected through dense blocks, using the dense connection mechanism to enhance the feature reuse capability, and obtain the The output features of dense blocks are ;

[0135] The first The output features of the dense block are The feature connection layer performs channel compression through 1x1 convolution and further reduces the spatial resolution through 2x2 average pooling to obtain the first The output features of the feature connection layer are The output features of the dense blocks are used as the output features of the convolutional branches of DenseNet121.

[0136] The DenseNet121 convolution branch extracts the output features of the DenseNet121 convolution branch from the fundus color image, gradually extracts the deep features in the image through dense blocks and feature connection layers, and performs dimensionality reduction processing to enhance the learning ability of complex features and the capture of global semantic information.

[0137] In this embodiment, the convolution branch of DenseNet121 includes: an initial convolution layer and dense blocks of 4 different convolution layers, and 3 feature connection layers, each of which includes a 1x1 convolution and a 2x2 average pooling layer connected in sequence.

[0138] In this embodiment, the dimension size is Fundus color images Input the DenseNet121 convolution branch, extract the output features of the DenseNet121 convolution branch, and obtain the output features of the DenseNet121 convolution branch, including:

[0139] Fundus color images After an initial convolution layer consisting of convolution and maximum pooling, the convolution kernel size is 7×7 and the step size is 2, the image features are obtained. , whose dimensions are , which extracts preliminary edge and texture features and halves the spatial resolution, where is the number of channels of image features;

[0140] Image features Entering the first dense block, the first dense block consists of 6 convolutional layers, and the dense connection mechanism enhances the feature reuse capability. The output of each layer is not only used as the input of the next layer, but also shares information with all subsequent layers to obtain the output features of the first dense block. , whose dimensions are .

[0141] Output features of the first dense block Through the first feature connection layer, the first feature connection layer performs channel compression through 1x1 convolution and further reduces the spatial resolution through 2x2 average pooling to obtain the output features of the first feature connection layer. , whose dimensions are ;

[0142] Output features of the first feature connection layer Through the second dense block, the second dense block has a total of 12 convolutional layers, and continues to extract higher-level features through the dense connection mechanism to obtain the output features of the second dense block , whose dimensions are .

[0143] Output features of the second dense block After the second feature connection layer, the spatial resolution is further reduced through 1x1 convolution and 2x2 average pooling to obtain the output features of the second feature connection layer. , whose dimensions are .

[0144] Output features of the second feature connection layer Through the third dense block, in which the network extracts deeper abstract features, the third dense block consists of 24 convolutional layers, and the output features of the third dense block are obtained , whose dimensions are .

[0145] Output features of the third dense block The output features of the third feature connection layer are obtained through the third feature connection layer , whose dimensions are .

[0146] Output features of the third feature connection layer Through the fourth dense block, the fourth dense block has 16 convolutional layers, and the output features of the convolution branch of DenseNet121 are obtained. , whose dimensions are .

[0147] The dense block is a key module in the convolution branch of DenseNet121. It uses the dense connection mechanism to achieve efficient feature extraction and reuse. The output of each layer is not only used as the input of the next layer, but also shares information with all subsequent layers. , the dense block extracts features layer by layer through multiple convolutional layers, where represents the height of the dense block input feature map, represents the width of the dense block input feature map, Indicates the number of channels of the dense block input feature map.

[0148] like Figure 5 As shown, Figure 5 It is a structural diagram of a dense block in the convolution branch of DenseNet121. In this embodiment, three layers of convolution are taken as an example to describe the operation steps and feature splicing process of the dense block.

[0149] Each convolution layer in a dense block includes one 1×1 convolution and one 3×3 convolution. The 1×1 convolution is used to compress the channels first, and then the 3×3 convolution is used to extract spatial features. After After the convolution layer, the generated Features The output of each layer is concatenated with the output of the previous layers in the channel dimension as the input of the next layer, where Indicates the number of channels of the output feature map.

[0150] Input dense blocks into feature maps The first feature is generated after the first layer of convolution , the first feature Input feature map with dense block Splice to get the second layer of convolution input features ,in, Represents the number of channels of the first feature, and the concatenation formula is: ,

[0151] in, represents the second layer convolution input features, Indicates the number of channels of the second layer convolution input feature, represents the dense block input feature map, Indicates the first feature.

[0152] The second layer of convolution input features The second feature is generated through the second layer of convolution , the second feature With the second layer of convolution input features Splice to get the third layer of convolution input features ,in, Represents the number of channels of the second feature, and the concatenation formula is:

[0153] ,

[0154] in, represents the third layer convolution input feature, Indicates the number of channels of the third layer convolution input features, Indicates the second feature.

[0155] Input the third layer of convolution into the feature The third feature is generated through the third layer of convolution , the third feature With the third layer convolution input features Splice to get the final output features ,in, Represents the number of channels of the third feature, and the concatenation formula is: ,

[0156] in, represents the final output features, Indicates the number of channels of the final output feature, Indicates the third feature.

[0157] After multiple layers of convolution and feature concatenation, the final output feature map of the dense block is , contains rich feature representations. These features will be input into the next feature connection layer.

[0158] The above operation can be extended to the general case, and the general formula of dense blocks is:

[0159] No. The convolution layer generates feature It will be concatenated with the feature map of the previous layer For splicing, the formula is as follows:

[0160] ,

[0161] The number of channels of the final output feature map is the number of channels of the dense block input feature map Add the sum of the number of output channels of each layer Therefore, the calculation formula for the final output feature map is:

[0162] ,

[0163] in, represents the final output feature map, Indicates Layer convolution input features, Indicates Layer convolution input features After The convolution layer generates feature, denotes the total number of convolutional layers of a dense block, Represents a dense block input feature map After After the convolution layer, the generated The number of channels of the feature, Indicates the number of channels of the final output feature map.

[0164] Through multi-layer convolution and dense connections, the DenseNet121 convolution branch provides rich output features of the DenseNet121 convolution branch, enhancing the model's ability to capture complex structures and details. The combination of the DenseNet121 convolution branch and the efficient self-attention Transformer branch further optimizes the network's feature extraction ability for fundus color photos, making the model more expressive when processing multi-level features.

[0165] By utilizing the dense connection mechanism of DenseNet121, multi-level convolutional features are extracted to enhance the model's ability to capture local features and details, and integrated with the self-attention mechanism to improve the overall classification effect.

[0166] Step S4: concatenate the output features of the efficient self-attention Transformer branch and the output features of the local feature extraction branch along the channel dimension to obtain fused features;

[0167] In this embodiment, the dimension is Output features of the efficient self-attention Transformer branch With dimension Output features of the convolutional branch of DenseNet121 Splicing along the channel dimension, the feature dimension is Fusion features , the formula for obtaining fusion features is:

[0168] ,

[0169] in, represents the fusion feature, represents the output features of the efficient self-attention Transformer branch, Represents the output features of the DenseNet121 convolutional branch, Indicates that a Contact operation is performed.

[0170] In order to solve the problem of uneven distribution of data sets, this paper designs a category-weighted attention module. Figure 6 As shown, Figure 6 The structural diagram of the class weighted attention module. This module contains a global feature extraction component and a class weighting component. The global feature extraction component of the class weighted attention module includes a channel weighting unit and a spatial weighting unit.

[0171] Step S4: The fused features are passed through the category weighted attention module to assign different weights to different categories to obtain weighted output features, including:

[0172] The fused features are passed through the global feature extraction component to extract channel weights and spatial attention weights to obtain global features;

[0173] like Figure 7 As shown, Figure 7 The structural diagram of the global feature extraction component in the category weighted attention module. The category weighted attention module includes: channel weighted unit and spatial weighted unit.

[0174] The fusion feature is passed through the global feature extraction component to extract the channel weight and the spatial attention weight to obtain the global feature, including:

[0175] The fusion features Through the channel weighting unit, the channel weight is extracted to obtain the intermediate feature map ,include:

[0176] The fusion features are processed by global average pooling operation. Perform channel compression and obtain a size of The pooled features of represents the height of the fused feature, represents the width of the fused feature, Indicates the number of channels of fused features;

[0177] After the pooled features pass through the convolution layer with a 1×1 convolution kernel, the channel weight is obtained through the Sigmoid activation function;

[0178] The channel weights and fusion features Multiply element by element to get the intermediate feature map , the calculation formula is as follows:

[0179] ,

[0180] in, represents the intermediate feature map, represents the fusion feature, represents the Sigmoid function, represents a convolutional layer consisting of a 1×1 convolution kernel, represents the global average pooling operation, Represents element-wise multiplication.

[0181] The intermediate feature map Through the spatial weighted unit, the spatial attention weight is extracted to obtain the global feature ,include:

[0182] For the intermediate feature map After average pooling along the channel direction, the spatial attention weight is obtained through the Sigmoid activation function;

[0183] Combine the spatial attention weights with the intermediate feature maps Multiply element by element to get the global features , the calculation formula is as follows:

[0184] ，

[0185] in, Represents global features, Represents the average pooling operation along the channel direction.

[0186] like Figure 8 As shown, Figure 8 The structural diagram of the category weighting component in the category weight attention module. In the category weighting component, equal treatment of all categories is ensured by assigning independent weights to each category. In particular, for the four grades in the retinitis pigmentosa (RP) grading task, the category weighting module improves the classification accuracy by learning weights for different categories.

[0187] The global features are passed through the category weighting component to assign independent weights to each category, and the weighted output features are obtained, including:

[0188] Global Features After 1×1 convolutional layer processing, the number of channels of the global feature is adjusted to ;

[0189] The feature map after adjusting the number of channels Divide into category feature maps, and calculate the weight score of each category feature map, where is the number of retinitis pigmentosa categories, The number of channels for each category feature map;

[0190] The formula for calculating the weight score of each category feature map is:

[0191] ,

[0192] in, Indicates The weight score of the class feature map, Indicates the number of channels of each category feature map, represents the global maximum pooling, Indicates In the category feature map The value of each channel.

[0193] right The category feature maps are average pooled separately to obtain the semantic feature map corresponding to each category feature map , the calculation formula is:

[0194] ,

[0195] in, Indicates The semantic feature map corresponding to the category feature map.

[0196] The weight score of each category feature map The corresponding semantic feature map Perform weighted averaging to obtain the category-weighted attention matrix , the calculation formula is:

[0197] ,

[0198] in, represents the class-weighted attention matrix.

[0199] Global Features and the class-weighted attention matrix Multiply element by element to get the weighted output features , the calculation formula is:

[0200] ,

[0201] in, represents the weighted output features, Represents global features.

[0202] Step S6: The weighted output features are passed through a classifier to output a prediction result of the retinitis pigmentosa grade.

[0203] The classifier consists of a fully connected layer and an activation function Softmax. The specific calculation formula is:

[0204] ,

[0205] in, is the fully connected layer, is the Softmax activation function, is the weighted output feature, It is the prediction result of retinitis pigmentosa grading, which corresponds to the four grades of retinitis pigmentosa: normal, early, middle and late.

[0206] In this embodiment, preferably, the training process of the retinitis pigmentosa category attention network is:

[0207] Obtaining a fundus color photo dataset, amplifying the data of the fundus color photo dataset, and then preprocessing it, and dividing the preprocessed fundus color photo dataset into a training set, a validation set, and a test set;

[0208] The training set is used to train the retinitis pigmentosa category attention network, and the progressive coding strategy is used to construct the total loss function to optimize the retinitis pigmentosa category attention network. The total loss function includes: the loss function of the standard coding and the progressive loss function used in the progressive coding. The trained retinitis pigmentosa category attention network is verified by the validation set to obtain the target retinitis pigmentosa category attention network.

[0209] The formula for the total loss function is:

[0210] ,

[0211] ,

[0212] ,

[0213] in, represents the total loss function, represents the loss function of standard encoding, represents the progressive loss function used by progressive coding, Indicates the total number of samples in the current batch, Indicates One-Hot encoding of the true label of the sample, if the The samples belong to retinitis pigmentosa type, is 1, otherwise it is 0. Represents the retinitis pigmentosa category attention network prediction The samples belong to The probability of the retinitis pigmentosa category, Indicates The progressively encoded labels of samples.

[0214] The clinical classification of retinitis pigmentosa includes four levels: normal, early, middle and late stages. As the disease develops, it gradually progresses from the normal stage to the late stage, showing obvious gradualness, and there is a sequential relationship between the categories. Traditional classification coding methods often cannot effectively capture these sequential features, and therefore, the present invention adopts a progressive coding strategy to better reflect the hierarchical progressive relationship between the classifications.

[0215] The introduction of progressive coding aims to directly express the order of categories through coding, further widening the feature differences between different levels. Compared with the standard classification method, progressive coding is more suitable for ordered classification tasks, because in such tasks, the impact of misclassifying samples into adjacent categories is much less than that of misclassifying samples into categories with large differences.

[0216] When progressive coding is used, the coding of each category is no longer conventional One-Hot coding, but all elements from the first element to the position of the category are set to 1, and all elements after the category are set to 0. This coding method not only indicates the label of the category, but also contains the information of the previous category, which can better reflect the progressive relationship between levels. For the fundus color photo dataset used in the present invention, there are four levels, and their One-Hot coding labels and progressive coding labels are shown in Table 1.

[0217] Table 1

[0218] level One-Hot Encoding Progressive Coding normal [1,0,0,0] [1,0,0,0] Early [0,1,0,0] [1,1,0,0] Mid-term [0,0,1,0] [1,1,1,0] Late stage [0,0,0,1] [1,1,1,1]

[0219] Since progressive coding requires the normalization of the model output through the Sigmoid activation function, the present invention uses a binary cross-entropy loss function (BCE) to replace the traditional cross entropy loss function (Cross Entropy). BCE can compare the model output with the target value of progressive coding element by element. This method not only takes into account the independence of each category, but also adapts to the multi-label characteristics of progressive coding. Therefore, the model can not only accurately classify samples, but also follow the order between categories in sequence, so as to better capture the differences between different categories.

[0220] The present invention effectively addresses the problems caused by the scarcity of data sets and class imbalance in the process of retinitis pigmentosa grading by introducing a class-weighted attention module and a progressive coding loss function. The class-weighted attention module ensures that the model can pay balanced attention to samples of different categories, improves the classification effect of a small number of samples, and enhances the recognition ability of difficult-to-classify samples. The label progressive coding loss function utilizes the progressiveness of retinitis pigmentosa grading to further improve the learning efficiency of the model on class-imbalanced data sets, so that the model can better distinguish patients with retinitis pigmentosa at different stages.

[0221] The category weighted attention module and progressive coding loss function in the present invention have strong modular characteristics. They can be easily integrated into different deep learning architectures to deal with category imbalance problems and improve lightweight effects. This flexibility enables the technology of the present invention to have a wide range of application scenarios, suitable for different data sets and task requirements, and has high scalability and versatility. By optimizing the self-attention mechanism, the present invention ensures a high classification accuracy while maintaining the efficiency of the model. This balance characteristic enables the present invention to achieve reasonable control of computing resources while improving the classification performance, and is suitable for application in actual medical scenarios.

[0222] In summary, the present invention demonstrates significant technical advantages and practicality in addressing key issues in the task of grading retinitis pigmentosa. In particular, while solving the problems of class imbalance and data scarcity, it achieves effective control of parameter quantity and computing resources through lightweight design and efficient feature extraction, and has high application prospects.

[0223] In order to verify the effectiveness of the method of the present invention, this embodiment 2 uses the fundus color photo dataset provided by Shenyang He's Ophthalmology for experimental verification. The details are as follows:

[0224] The fundus color photo dataset of the present invention contains a total of 596 fundus color photos from 298 patients, and the four levels of images include 300 normal, 70 early, 138 mid-term and 88 late-term images. The image resolutions include 2048*1536, 3100*2848 and 1920*1088. In order to unify the input image size and accelerate network training, all fundus color photos are resampled to 336*336. The graded labels are annotated by senior ophthalmologists based on the size, shape, position of pigmentation, choroidal atrophy, optic disc color, vascular tortuosity and other factors.

[0225] Since the number of images in the original fundus color photo dataset is small, in order to prevent overfitting and enhance the generalization ability of the model, two data augmentation strategies are adopted: preprocessing augmentation and online augmentation.

[0226] (a) Preprocessing Data Augmentation

[0227] First, the 596 images in the original dataset were preprocessed and augmented. The main operations included random rotation and random flipping. Through these operations, the number of images in the dataset was significantly expanded. The amount of data at each level after expansion is as follows: 600 normal images, 210 early images, 414 mid-term images, and 264 late images. Finally, the entire dataset was expanded from 596 images to 1488 images.

[0228] On the expanded dataset, the data was divided into training set, validation set, and test set in a ratio of 7:1.5:1.5. To prevent data leakage, the left and right eye images of the same patient (including the expanded samples) were divided into the same dataset. In addition, the proportion of each category in each dataset was kept consistent to ensure the comparability of the experiments and the stability of the results. In all experiments, the same random seed and resolution were used to ensure fairness in training and testing.

[0229] (b) Online data augmentation

[0230] In order to improve the generalization ability of the network and reduce overfitting, the present invention adopts an online random data augmentation strategy. In each training batch, the images are randomly rotated, sharpened, adjusted in brightness, and colored to achieve dynamic data expansion, ensuring that each batch of images generates diverse variants during the training process, reducing the risk of overfitting, and improving the efficiency and robustness of the model by compressing the search space.

[0231] In order to verify the performance of the proposed attention network for the retinitis pigmentosa category, it was compared with several current advanced deep learning classification networks, including ResNet-50, ResNext-50, InceptionV3, DenseNet121, DenseNet161, Swin Transformer, DTMFormer, CF-ViT and LF-ViT. To ensure the fairness of the experiment, all models used pre-trained weights on ImageNet-1K and were fully fine-tuned in the retinitis pigmentosa classification task, that is, all layers participated in the training and no parameters were frozen.

[0232] As shown in Table 2, Table 2 shows the classification performance of each model on the same test set.

[0233] Table 2

[0234]

[0235] The experimental results show that the proposed retinitis pigmentosa category attention network (EfficientRP-CATNet) performs well in multiple key indicators, especially in 、 、 and These four aspects reached 84.21%, 84.28%, 84.66% and 84.21% respectively. These results prove that the proposed retinitis pigmentosa category attention network (EfficientRP-CATNet) can not only provide excellent classification performance with a relatively small number of parameters, but also achieve a good balance between precision and recall, especially leading other models in F1 value, further confirming its superiority in the retinitis pigmentosa grading task.

[0236] Although the proposed retinitis pigmentosa category attention network (EfficientRP-CATNet) is not the model with the smallest number of parameters, its outstanding performance proves that it has high parameter efficiency. Compared with models with larger parameters such as Swin Transformer and DenseNet161, the proposed retinitis pigmentosa category attention network achieves better results with fewer parameters, demonstrating the advantages of its lightweight design and efficient network structure design.

[0237] DenseNet121 and DenseNet161 use dense connections to allow features to be reused more efficiently between network layers, thereby improving the model's feature extraction capabilities. DenseNet161 outperforms DenseNet121 due to its larger number of parameters, which shows that increasing the network depth can further improve classification performance while keeping the architecture unchanged.

[0238] Swin Transformer is a visual model based on the Transformer architecture. By introducing a hierarchical structure, it reduces the number of parameters while improving the ability to capture local and global features. Despite the large number of parameters, Swin Transformer performs better than traditional convolutional neural networks such as ResNet50, InceptionV3, ResNext50, and DenseNet161 in the task of grading retinitis pigmentosa.

[0239] DTMFormer is a method designed to address the attention collapse problem encountered by Transformer models in medical image segmentation. This problem arises mainly because Transformer models often suffer from poor model convergence when dealing with tasks with higher complexity, especially when the labeled data is limited. To address this issue, DTMFormer introduces a novel, plug-and-play Transformer module that dynamically merges redundant and similar tokens and aggregates them into fewer semantic tokens, thereby reducing model complexity and improving performance. In the automatic grading of retinitis pigmentosa, DTMFormer outperforms Swin Transformer with a reduced number of parameters.

[0240] CF-ViT is a model that improves the efficiency and effectiveness of visual transformers by introducing a coarse-to-fine strategy. This method enables the model to focus on important areas in the image, thereby improving the overall performance of the network. CF-ViT adopts a hierarchical structure, first making rough predictions, and then fine-tuning the preliminary results through more detailed predictions. LF-ViT aims to reduce spatial redundancy in visual transformers by introducing a novel token sparsification method. This method ensures that only the most important tokens are processed, thereby reducing computational complexity and improving the efficiency of the model. LF-ViT significantly reduces the number of parameters while maintaining high performance by selectively focusing on task-related areas in the image. Although the number of parameters of the CF-ViT and LF-ViT models is smaller than that of the retinitis pigmentosa category attention network proposed in this invention, the performance is poor in the retinitis pigmentosa automatic classification task. The reason is that although CF-ViT and LF-ViT have improved efficiency, they cannot capture important detailed features in the retinitis pigmentosa classification task, which relies on the accurate identification of small lesions and retinal structural changes. The combination of the self-attention mechanism of the retinitis pigmentosa category attention network proposed in the present invention and DenseNet121 ensures the comprehensive extraction of global and local features, and has more advantages in processing the details of medical images.

[0241] The proposed retinitis pigmentosa category attention network (EfficientRP-CATNet) shows excellent performance in comparison with other models. Through the lightweight improvement of the Transformer structure, not only the powerful global feature extraction capability of the Transformer is maintained, but also its computational efficiency is significantly improved. The introduced category-weighted attention mechanism and progressive coding loss function further enhance the performance of the model in dealing with data category imbalance problems, while improving the ability to capture detailed features in fundus color photos. While maintaining excellent performance, the model significantly reduces the number of parameters and successfully achieves lightweight and efficient design.

[0242] In practical applications, the proposed retinitis pigmentosa category attention network (EfficientRP-CATNet) performs well in multiple key indicators, especially in balancing precision and recall, which shows the practical application value of the model in the automatic classification of retinitis pigmentosa. By effectively solving the problems of category imbalance and data scarcity, the model can maintain high accuracy and stability under different classification conditions, showing strong generalization ability.

[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0244] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0245] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0247] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for constructing a category attention network for grading retinitis pigmentosa, characterized in that: The following steps are involved: Constructing a retinitis pigmentosa category attention network, the retinitis pigmentosa category attention network comprising: two parallel branches, a category weighted attention module and a classifier; the two parallel branches comprising: an efficient self-attention Transformer branch and a local feature extraction branch; Input the fundus color image into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch; Inputting the fundus color photo image into the local feature extraction branch, extracting local information, and obtaining output features of the local feature extraction branch; The output features of the efficient self-attention Transformer branch are concatenated with the output features of the local feature extraction branch along the channel dimension to obtain the fused features; The fusion features are passed through the category-weighted attention module to assign different weights to different categories, and the weighted output features are obtained, including: The fused features are passed through the global feature extraction component to extract channel weights and spatial attention weights to obtain global features; The global features are passed through the category weighting component to assign independent weights to each category, and the weighted output features are obtained, including: The global features are processed by a 1×1 convolution layer and the number of channels of the global features is adjusted to ; The feature map after the number of channels is adjusted is divided into Class feature maps are generated, and the weight score of each class feature map is calculated using the following formula: ， in, is the number of retinitis pigmentosa categories, is the number of channels of each category feature map, Indicates The weight score of the category feature map, represents the global maximum pooling, Indicates In the category feature map The value of each channel; right Each category feature map is average pooled to obtain the semantic feature map corresponding to each category feature map; The weight score of each category feature map is weighted averaged with its corresponding semantic feature map to obtain the category weighted attention matrix; Multiply the global feature by the category-weighted attention matrix element by element to obtain the weighted output feature; The weighted output features are passed through a classifier to output the prediction results of retinitis pigmentosa grading.

2. A method for constructing a category attention network for grading retinitis pigmentosa according to claim 1, characterized in that: The efficient self-attention Transformer branch includes: a patch block module and multiple feature extraction modules connected in series along the forward propagation direction, wherein the first feature extraction module includes: a linear embedding layer and an efficient Transformer module connected in sequence, and the other feature extraction modules except the first feature extraction module include a linear merging layer and an efficient Transformer module connected in sequence; Among them, by replacing the global multi-head attention mechanism in the visual converter ViT with an efficient self-attention mechanism, an efficient Transformer module is obtained, including: deep convolutional layers, layer normalization, efficient self-attention mechanisms, layer normalization, and fully connected layers connected in sequence along the forward propagation direction, and the features output by the deep convolutional layer are residually connected with the output features of the efficient self-attention mechanism.

3. A method for constructing a category attention network for grading retinitis pigmentosa according to claim 2, characterized in that: The fundus color image is input into the efficient self-attention Transformer branch to capture global information and obtain the output features of the efficient self-attention Transformer branch, including: The fundus color photo image is divided into blocks through the patch block module to obtain multiple feature maps of the same size; After each feature map is converted into its corresponding one-dimensional sequence, all the one-dimensional sequences are concatenated to obtain the input sequence of the first feature extraction module; After the input sequence of the first feature extraction module is dimensionally increased by the linear embedding layer of the first feature extraction module, the first output feature is extracted by the efficient Transformer module of the first feature extraction module; After the first output feature is linearly transformed by the linear merging layer of the second feature extraction module, the global information is captured by the efficient Transformer module of the second feature extraction module to obtain the second output feature, and the second output feature is used as the input of the next feature extraction module; The output features of the last feature extraction module are used as the output features of the efficient self-attention Transformer branch.

4. A method for constructing a category attention network for grading retinitis pigmentosa according to claim 2, characterized in that: The input features of the efficient Transformer module are passed through the efficient Transformer module, and the attention weights are adjusted to obtain the output features of the efficient Transformer module, including: After the input features of the efficient Transformer module are processed by deep convolution layer and layer normalization, they are processed by the efficient self-attention mechanism to obtain the output features of the efficient self-attention mechanism, including: The feature map after layer normalization is reshaped through the reshape operation to obtain the reshaped features, where the reshaped features include Patch blocks, The number of patches that the feature map after layer normalization is divided into in height or width; Multiply the reshaped features with three different linear projection layers to obtain the query vector, key vector and value vector; Perform average operations on each patch block in the query vector and key vector respectively to obtain the average features of the query vector and the average features of the key vector; Calculate the attention score of the average features of the query vector and the average features of the key vector to obtain the feature association graph; The feature association graph is used to select features through the Topk algorithm to obtain a sparse index matrix; Merge the key vector and the value vector with the sparse index matrix respectively to obtain the merged key vector and value vector; After calculating the similarity by performing a dot product between the query vector and the merged key vector, the similarity is divided by the square root of the dimension of the key vector to obtain the attention weight; Normalize the attention weights through the Softmax operation to obtain the attention weight matrix; The value vector is weighted and summed according to the attention weight matrix to obtain the output features of the efficient self-attention mechanism; After residual connection is performed on the output features of the efficient self-attention mechanism and the features output by the deep convolutional layer, they are processed by layer normalization and fully connected layers in turn to obtain the output features of the efficient Transformer module.

5. The method for constructing a category attention network for grading retinitis pigmentosa according to claim 1, characterized in that: The fusion feature is passed through the global feature extraction component to extract the channel weight and the spatial attention weight to obtain the global feature, including: Pass the fused features through the channel weighting unit, extract the channel weights, and obtain the intermediate feature map; The intermediate feature map is passed through the spatial weighted unit to extract the spatial attention weights and obtain the global features.

6. A method for constructing a category attention network for grading retinitis pigmentosa according to claim 5, characterized in that: The fusion feature is passed through the global feature extraction component to extract the channel weight and the spatial attention weight to obtain the global feature, including: The fused features are channel compressed through the global average pooling operation to obtain the pooled features; After the pooled features pass through the convolution layer with a 1×1 convolution kernel, the channel weight is obtained through the Sigmoid activation function; Multiply the channel weight and the fusion feature element by element to obtain the intermediate feature map; After average pooling the intermediate feature map along the channel direction, the spatial attention weight is obtained through the Sigmoid activation function; The spatial attention weights are element-wise multiplied with the intermediate feature map to obtain the global features.

7. The method for constructing a category attention network for grading retinitis pigmentosa according to claim 1, characterized in that: The pair The category feature maps are averaged and pooled respectively to obtain the semantic feature map corresponding to each category feature map. The calculation formula is: ， The weight score of each category feature map is weighted averaged with its corresponding semantic feature map to obtain a category weighted attention matrix, and the calculation formula is: ， The global feature is multiplied element by element with the category weighted attention matrix to obtain the weighted output feature, and the calculation formula is: ， in, Indicates The weight score of the category feature map, Indicates the number of channels of each category feature map, Indicates In the category feature map The value of the channel, Indicates The semantic feature map corresponding to the category feature map, represents the category-weighted attention matrix, represents the weighted output features, Represents global features, Represents element-wise multiplication.

8. The method for constructing a category attention network for grading retinitis pigmentosa according to claim 1, characterized in that: The local feature extraction branch is any one of the MobileNet convolution branch, the GoogleNet convolution branch, and the DenseNet121 convolution branch.

9. A method for constructing a category attention network for grading retinitis pigmentosa according to claim 7, characterized in that: Input the fundus color image into the DenseNet121 convolution branch, extract local information, and obtain the output features of the DenseNet121 convolution branch; Among them, the convolution branch of DenseNet121 includes: the initial convolution layer and dense blocks of different convolutional layers, There are feature connection layers, each of which includes a 1x1 convolution and a 2x2 average pooling layer connected in sequence.

10. The method for constructing a category attention network for grading retinitis pigmentosa according to claim 1, characterized in that: The training process of the attention network for the retinitis pigmentosa category is: Obtaining a fundus color photo dataset, amplifying the data of the fundus color photo dataset, and then preprocessing it, and dividing the preprocessed fundus color photo dataset into a training set, a validation set, and a test set; The training set is used to train the retinitis pigmentosa category attention network, and the progressive coding strategy is used to construct the total loss function to optimize the retinitis pigmentosa category attention network. The total loss function includes: the loss function of the standard coding and the progressive loss function used in the progressive coding. The trained retinitis pigmentosa category attention network is verified by the validation set to obtain the target retinitis pigmentosa category attention network. The formula for the total loss function is: ， ， ， in, represents the total loss function, represents the loss function of standard encoding, represents the progressive loss function used by progressive coding, Indicates the total number of samples in the current batch, Indicates One-Hot encoding of the true label of the sample, if The samples belong to retinitis pigmentosa type, is 1, otherwise it is 0. Represents the retinitis pigmentosa category attention network prediction The samples belong to The probability of the retinitis pigmentosa category, Indicates The progressively encoded labels of samples.

Images