A multi-label classification system and method for retinal OCT image lesions

Through the multi-stage self-attention fusion, local target perception and graph convolution label inline modules of the IAF-MCNet network, the problem of insufficient global connection and disease association in retinal OCT image classification is solved, and the classification accuracy of retinal diseases is improved and efficiency.

CN116503639BActive Publication Date: 2024-12-06SUZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310240656.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-14
Publication Date
2024-12-06
Estimated Expiration
2043-03-14

AI Technical Summary

Technical Problem

In the prior art, the global connection of retinal OCT images is not established, and the correlation between diseases with smaller lesions and different retinal diseases is ignored, resulting in inaccurate classification.

Method used

A multi-label classification system for retinal OCT image lesions is designed, and multi-stage features are fused through multi-stage self-attention fusion module (MSFM), local object perception module (LOAM) splits shallow and deep features stitching, and graph convolution labels are graphically convolutional. Inline module (GLAM) constructs the intrinsic connection between disease tags and forms the IAF-MCNet network.

Benefits of technology

Effectively extracting characteristic information of retinal OCT images improves the accuracy and efficiency of disease classification, can serve as a reference for doctors' diagnosis, reduces the burden on doctors to read films, and provides assistance for grassroots fundus disease screening.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116503639B_ABST
    Figure CN116503639B_ABST
Patent Text Reader

Abstract

The present invention relates to a multi-label classification system and method for retinal OCT image lesions. The system includes: an acquisition module: used for acquiring retinal OCT images; a first fusion module: used for extracting deep feature maps of retinal OCT images, and fusing features outputted in the deep feature extraction process with the deep feature map to obtain an image feature map; a splicing module: used for extracting shallow features of two retinal OCT images, and splicing shallow features including local target information with the image feature map to obtain splicing features; a second fusion module: used for constructing input data and inputting it into a graph convolutional neural network to obtain a vector matrix, and then fusing the vector matrix and the splicing features to obtain an output result; a classification module: used for classifying various diseases of retinal OCT images according to the output results. The present invention can effectively extract features of retinal OCT images and classify various retinal diseases.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multi-label classification system and method for retinal OCT image lesions. Background Art

[0002] Retinal diseases are currently the most common blinding diseases in my country, such as diabetic retinopathy, macular degeneration or holes, retinitis pigmentosa, etc. Multiple retinal diseases may occur at the same time. Unlike common eye diseases, retinal diseases are irreversible. Optical coherence tomography (OCT) has gradually become the gold standard for the diagnosis of fundus diseases due to its high resolution and non-destructive characteristics. Classification of retinal OCT images based on traditional machine learning is one of the commonly used methods, such as logistic regression, support vector machine, and random forest, but this requires a lot of prior knowledge to manually design generalizable features. With the development of deep neural networks and hardware technology, convolutional neural networks have become the mainstream of image classification with their advantages of automatically learning deep image features. Among them, VGGNet deepens the depth of deep learning networks, and GoogLeNet thinks from the perspective of model breadth. The Inception structure includes convolution kernels of different sizes and integrates multi-scale features of images. The milestone ResNet proposed a residual block structure to solve the problem of performance degradation due to excessive network depth. On this basis, DenseNet proposed a densely connected block with better performance.

[0003] In the prior art, the literature Gulshan V, Peng L, Coram M, et al. "Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs" Jama (2016) uses a convolutional neural network to classify fundus color photographs of diabetic retinopathy, and the literature Cecilia, S, Lee, et al. "Deep Learning Is Effective for Classifying Normal versus Age-Related Macular Degeneration OCT Images" Ophthalmology Retina (2017) uses a convolutional neural network to distinguish OCT images of AMD patients, but these networks only perform binary classification on some diseases, which makes it difficult to achieve the goal of fundus disease screening. Reference Junjun He, Cheng Li, Jin Ye, Yu Qiao, Lixu Gu, "Multi-label ocular disease classification with a dense correlation deep neural network", Biomedical Signal Processing and Control, 63, 102167 (2021) used a convolutional neural network with an attention mechanism to perform multi-label classification of fundus color photos for multiple diseases, but lacked the ability to build correlations for long-distance information. With the development of the Transformer structure in the field of computer vision, convolutional neural networks combined with self-attention modules can better extract image features, thereby improving classification accuracy. However, these methods do not take into account the correlation between labels.

[0004] Although ordinary convolutional neural networks can achieve relatively advanced classification results, they have the following problems: (1) The receptive field can be expanded through convolutional pooling operations, but global connections cannot be established, which is not enough to fully utilize contextual information to achieve accurate predictions; (2) The connection between deep and shallow information is not close enough. Shallow information contains more local information, while deep information is often more abstract and complex. Only inputting deep features into the classifier often ignores diseases with smaller lesions; (3) Some retinal diseases may have complications. For example, patients with age-related macular degeneration may have both drusen and exudates. Existing convolutional neural networks cannot build disease label correlations. These problems lead to inaccurate classification of various diseases in retinal OCT images. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art of failing to establish a global connection for retinal OCT images, ignoring diseases with smaller lesions, and ignoring the correlation between different retinal diseases, thereby leading to inaccurate classification of diseases in retinal OCT images.

[0006] In order to solve the above technical problems, the present invention provides a multi-label classification system for retinal OCT image lesions, comprising:

[0007] Acquisition module: used to acquire two retinal OCT images of the same subject;

[0008] The first fusion module is used to extract the deep features of the two retinal OCT images to obtain a deep feature map, and at the same time, the features output at each stage in the deep feature extraction process are fused with the deep feature map to obtain an image feature map;

[0009] A stitching module is used to extract shallow features of the two retinal OCT images, wherein the shallow features include local target information, and stitch the shallow features including the local target information with the image feature map to obtain stitching features;

[0010] The second fusion module is used to construct input data based on different retinal disease labels, and input the input data into the graph convolutional neural network to obtain a vector matrix, and then fuse the vector matrix with the splicing features to obtain an output result;

[0011] Classification module: used for classifying various diseases of retinal OCT images according to the output results.

[0012] In one embodiment of the present invention, the first fusion module extracts the deep features of the two retinal OCT images to obtain a deep feature map, specifically: extracting the deep features of the two retinal OCT images through sequentially connected convolution units, first densely connected blocks, second densely connected blocks, third densely connected blocks, and fourth densely connected blocks to obtain a deep feature map;

[0013] Among them, the convolution unit includes a convolution layer, a batch normalization layer and a RELU activation layer connected in sequence; the first dense connection block includes 1×1 convolution and 3×3 convolution performed 6 times; the second dense connection block includes 1×1 convolution and 3×3 convolution performed 12 times; the third dense connection block includes 1×1 convolution and 3×3 convolution performed 24 times; the fourth dense connection block includes 1×1 convolution and 3×3 convolution performed 16 times.

[0014] In one embodiment of the present invention, the first fusion module fuses the features output at each stage in the deep feature extraction process with the deep feature map to obtain an image feature map, specifically:

[0015] Performing matrix addition on the outputs of the convolution unit, the first dense connection block, and the second dense connection block, respectively, through downsampling and 1×1 convolution, and the output of the third dense connection block through 1×1 convolution to obtain a first matrix addition result, then performing 1×1 convolution on the first matrix addition result to obtain a Query matrix, and modifying the size of the Query matrix;

[0016] The output of the fourth densely connected block is dimensionally upgraded, and the result after dimension upgrading is split by channel to obtain the initial Key matrix, the initial Value matrix and the original visual feature matrix F split , performing 1×1 convolution on the initial Key matrix and the initial Value matrix to obtain a Key matrix and a Value matrix respectively, and modifying the size of the Key matrix;

[0017] The modified size Query matrix and the modified size Key matrix are matrix multiplied to obtain a first matrix multiplication result, and the weight feature matrix W of the first matrix multiplication result is extracted through softmax activation, and the weight feature matrix W is matrix multiplied with the Value matrix to obtain a second matrix multiplication result, and the second matrix multiplication result is resized and 1×1 convolved before being combined with the original visual feature matrix F split Perform matrix addition to obtain a second matrix addition result, wherein the second matrix addition result is an image feature map.

[0018] In one embodiment of the present invention, the shallow features of the two retinal OCT images are extracted in the stitching module, specifically:

[0019] Extracting features output by the first densely connected block, the second densely connected block, and the third densely connected block respectively, and dividing the three output features into non-overlapping image blocks respectively;

[0020] The image blocks corresponding to the three features are stacked along the channel dimension, and the three stacked image blocks are scaled into one-dimensional features through 1×1 convolution. Then, the results of global average pooling and global maximum pooling of the three one-dimensional features are matrix-added to obtain the third matrix addition result, the fourth matrix addition result, and the fifth matrix addition result;

[0021] The third matrix addition result, the fourth matrix addition result and the fifth matrix addition result are fused into three sub-vectors through two fully connected layers and ReLU activation layers respectively, and finally the three sub-vectors are concatenated into a vector output by channel, wherein the vector output is a shallow feature including local target information.

[0022] In one embodiment of the present invention, the second fusion module constructs input data based on different retinal disease labels, and inputs the input data into a graph convolutional neural network to obtain a vector matrix, and then fuses the vector matrix with the splicing feature to obtain an output result, specifically:

[0023] Different retinal disease labels are input into the word vector model to construct the initial graph structure, and an adjacency matrix is ​​constructed based on the frequency of occurrence of different retinal disease labels in pairs.

[0024] Inputting the initial graph structure and the adjacency matrix as input data into the graph convolutional neural network at the same time to obtain a vector matrix;

[0025] The splicing features are input into the fully connected layer to obtain a fully connected result, and then the fully connected result is matrix multiplied with the transposed vector matrix to obtain a third matrix multiplication result, and then the third matrix multiplication result is spliced ​​with the fully connected result by channel to obtain an output result.

[0026] In one embodiment of the present invention, the element A in the adjacency matrix A ij Satisfy the formula:

[0027] A ij =M ij / N i

[0028] Among them, the adjacency matrix A satisfies represents a matrix that sets the diagonal elements of the adjacency matrix A to 1, I represents the identity matrix, and M ij Indicates disease label i and disease labels j The number of simultaneous occurrences, N i Represents the total number of occurrences of each disease label.

[0029] In one embodiment of the present invention, it also includes an optimization module: used to optimize the process of the first fusion module, the splicing module and the second fusion module according to the classification result obtained by the classification module, and the formula is:

[0030]

[0031] Among them, C represents the number of disease labels, p i Indicates that disease label l appears in retinal OCT imagesi The probability of y i ∈{0,1} is the true category label, when y i = 0, indicating that there is no disease label in the retinal OCT image. i When y i =1, indicating that there is a disease label l in the retinal OCT image i .

[0032] In order to solve the above technical problems, the present invention provides a multi-label classification method for retinal OCT image lesions, comprising:

[0033] Step S1: acquiring two retinal OCT images of the same subject;

[0034] Step S2: extracting deep features of the two retinal OCT images to obtain a deep feature map, and fusing the features output at each stage in the deep feature extraction process with the deep feature map to obtain an image feature map;

[0035] Step S3: extracting shallow features of the two retinal OCT images, wherein the shallow features include local target information, and splicing the shallow features including the local target information with the image feature map to obtain spliced ​​features;

[0036] Step S4: constructing input data based on different retinal disease labels, and inputting the input data into a graph convolutional neural network to obtain a vector matrix, and then fusing the vector matrix with the splicing features to obtain an output result;

[0037] Step S5: Classify various diseases of the retinal OCT image according to the output results.

[0038] To solve the above technical problems, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned multi-label classification method for retinal OCT image lesions are implemented.

[0039] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multi-label classification method for retinal OCT image lesions are implemented.

[0040] The above technical solution of the present invention has the following advantages compared with the prior art:

[0041] The IAF-MCNet network of the present invention is designed with three modules: a multi-stage self-attention fusion module (MSFM) fuses multi-stage features at the bottom of the feature extractor to perceive the dependency between different pixels; a local target perception module (LOAM) segments shallow feature vectors, identifies shallow local targets and splices and fuses them with deep features to enhance the ability to perceive small target features; a graph convolution label inline module (GLAM) uses the co-occurrence probability matrix of retinal disease labels as an adjacency matrix to construct the intrinsic connection between different disease labels; the design of the three modules can effectively extract various feature information in retinal OCT images, providing reliable support for the subsequent classification of retinal diseases;

[0042] The present invention has strong practicality and versatility, and its classification results can also be used as a reference for doctors to diagnose fundus diseases, reducing the burden of doctors reading films, improving the accuracy and efficiency of lesion classification, and also can provide assistance for fundus disease screening in grassroots communities. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0044] Figure 1 is a flow chart of the present invention;

[0045] Figure 2 Schematic diagram of the overall structure of the IAF-MCNet network in an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of the structure of a multi-stage self-attention fusion module (MSFM) in an embodiment of the present invention;

[0047] Figure 4 1 is a schematic diagram of the structure of a local object awareness module (LOAM) in an embodiment of the present invention;

[0048] Figure 5 It is a schematic diagram of the structure of a graph convolutional label inline module (GLAM) in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0050] Embodiment 1

[0051] Reference Figure 1 As shown, the present invention provides a multi-label classification system for retinal OCT image lesions, comprising:

[0052] Acquisition module: used to acquire two retinal OCT images of the same subject;

[0053] The first fusion module is used to extract the deep features of the two retinal OCT images to obtain a deep feature map, and at the same time, the features output at each stage in the deep feature extraction process are fused with the deep feature map to obtain an image feature map;

[0054] A stitching module is used to extract shallow features of the two retinal OCT images, wherein the shallow features include local target information, and stitch the shallow features including the local target information with the image feature map to obtain stitching features;

[0055] The second fusion module is used to construct input data based on different retinal disease labels, and input the input data into the graph convolutional neural network to obtain a vector matrix, and then fuse the vector matrix with the splicing features to obtain an output result;

[0056] Classification module: used for classifying various diseases of retinal OCT images according to the output results.

[0057] See also Figure 2 , This embodiment designs an information associated fusion multi-label classification network (IAF-MCNet) based on convolutional neural networks, attention mechanisms, and graph convolution structures. It uses two retinal OCT images (i.e., B-scan images) of the same subject as input to classify 11 retinal disease labels (if all labels are 0, it is a normal image). For ease of understanding and description, the IAF-MCNet network has designed three network modules. The multi-stage self-attention fusion module (MSFM) fuses multi-stage features at the bottom of the feature extractor to perceive the dependencies between different pixels. MSFM corresponds to the first fusion module; the local target perception module (LOAM) divides the shallow feature vector, identifies the shallow local target and splices and fuses it with the deep feature to enhance the ability to perceive small target features. LOAM corresponds to the splicing module; the graph convolution label inline module (GLAM) uses the co-occurrence probability matrix of the retinal disease label as the adjacency matrix to construct the intrinsic connection between different disease labels. GLAM corresponds to the second fusion module. Finally, the fully connected layer is used as a classifier to classify the 11 disease labels of retinal OCT images.

[0058] The following is a detailed introduction to this embodiment:

[0059] In this embodiment, the IAF-MCNet network is used for disease classification of retinal OCT images. Two retinal OCT images (i.e., B-scan images) are obtained as network input through the acquisition module to predict 11 disease labels (if all labels are 0, it is a normal image). This embodiment starts from long-distance correlation, deep and shallow feature fusion, and the intrinsic relationship between labels, and strengthens the ability to integrate the intrinsic connection of feature and label information. Specifically, the IAF-MCNet network structure is as follows: Figure 2 As shown in the figure, in the first fusion module, the feature extractor uses Densenet121 pre-trained based on ImageNet as the backbone network, and removes the last global pooling and fully connected layers. The dense connection block used by Densenet121 reduces network parameters and can improve feature propagation and utilization efficiency, and alleviate the problem of gradient disappearance. It includes a total of 5 stages: the 0th stage is a convolution unit (including a 7×7 convolution with a step size of 2, a batch normalization layer, and a Relu activation layer); the dense connection blocks of the 1st to 4th stages contain 6, 12, 24, and 16 bottleneck structures (1×1 convolution, 3×3 convolution), respectively, that is, the first dense connection block includes 1×1 convolution and 3×3 convolution executed 6 times, the second dense connection block includes 1×1 convolution and 3×3 convolution executed 12 times, the third dense connection block includes 1×1 convolution and 3×3 convolution executed 24 times, and the fourth dense connection block includes 1×1 convolution and 3×3 convolution executed 16 times. Each stage downsamples once and doubles the number of channels. Densenet121 downsamples 5 times in total, and the output channels are 64, 128, 256, 512, and 1024 respectively. They are marked as 0-4 levels from high to low resolution, with a total of 5 levels of feature maps.

[0060] (a) Multi-stage self-attention fusion module (MSFM)

[0061] See also Figure 3 In the first fusion module, since the deeper the convolutional neural network, the more important the extracted features are and the richer the semantics, the feature map of stage 4 (i.e., the output of the fourth densely connected block) is upgraded through a multi-layer perceptron consisting of two fully connected layers and a ReLU activation layer. The number of channels becomes three times the original number. The upgraded new feature map is split by channel to obtain the initial Key matrix and the initial Value matrix, and 1×1 convolution operations are performed to construct the Key matrix and the Value matrix respectively. The Key matrix is ​​resized to facilitate matrix multiplication and calculation of correlation. At the same time, the upgraded new feature map is split by channel to obtain the original visual feature matrix F split. The feature maps of stages 0-2 (i.e., the outputs of the convolutional unit, the first densely connected block, and the second densely connected block) are all downsampled and 1×1 convolved. The feature maps of stage 3 (i.e., the output of the third densely connected block) are convolved 1×1. The results obtained from the above stages 0-2 and 3 are added element by element and subjected to 1×1 convolution to obtain the Query matrix (query matrix), and the size of the Query matrix is ​​modified. Here, shallow information is integrated to enhance the attention and understanding of detailed features. The modified-size Query matrix and the modified-size Key matrix are matrix multiplied, and the weight feature matrix W after matrix multiplication is extracted through softmax activation. The weight feature matrix W is multiplied with the Value matrix, and the multiplication result is resized and 1×1 convolved before being combined with the original visual feature matrix F. split The output is obtained by adding the two OCT images after MSFM. Figure 2 This part is similar to the residual connection in ResNet, which makes the network easier to optimize and is formulated as follows:

[0062] W = softmax(KQ T )

[0063]

[0064] in, Represents the output result of MSFM. The output results of two OCT images after MSFM are added together. Figure 2 The image feature map in , T represents transpose, and conv represents convolution.

[0065] (b) Local Object Awareness Module (LOAM)

[0066] See also Figure 4 In the splicing module (i.e., the LOAM module), mainly for the shallow information output by stages 1-3, the multi-scale feature maps (i.e., the shallow information output by the first densely connected block, the second densely connected block, and the third densely connected block) are divided into non-overlapping image blocks, and the image blocks are stacked along the channel dimension, and scaled into one-dimensional features by 1×1 convolution to fuse the information of each image block, and then the results of global average pooling of the three one-dimensional features are matrix-added with the results of global maximum pooling, and then fused into three sub-vectors by a multi-layer perceptron composed of two fully connected layers and a ReLU activation layer, and finally the three sub-vectors are spliced ​​into a vector output by channel, wherein the vector output is a shallow feature including local target information. The LOAM module of this embodiment can be regarded as an attention-based image block extractor, which can perceive the feature blocks corresponding to small lesions, fuse local information and global information, so as to improve the accuracy and recall rate of disease diagnosis.

[0067] The output of LOAM (i.e., the local target of the shallow features output by the first densely connected block, the second densely connected block, and the third densely connected block) is concatenated (CAT) with the image feature map output by MSFM to obtain the concatenated features. The above two modules (MSFM and LOAM) can improve the representation learning ability of the network and integrate the deep and shallow feature information.

[0068] (c) Graph Convolutional Label Inline Module (GLAM)

[0069] See also Figure 5 By observing the data set, it is found that some disease labels appear in pairs and are related to complications of some retinal diseases, while some disease labels are less correlated. Therefore, a graph convolutional label inline module (GLAM) is constructed to find the relationship between labels. The GLAM module is the second fusion module. The graph convolutional neural network needs to construct an initial graph structure and an adjacency matrix. This embodiment constructs the initial graph structure in a label vectorized manner, using the GloVe model (a typical word vector model) trained based on Wikipedia data to vectorize the disease label words and construct distance relationships.

[0070] Specifically, in the second fusion module, different retinal disease labels are input into the GloVe model to construct an initial graph structure, and an adjacency matrix is ​​constructed based on the frequency of occurrence of different retinal disease labels in pairs; the constructed initial graph structure and the adjacency matrix are input into the graph convolutional neural network (GCN) to obtain a vector matrix; the splicing matrix obtained by the above splicing (CAT) is input into the fully connected layer to obtain a fully connected result, and then the fully connected result is matrix multiplied with the transposed vector matrix, and then the multiplication result and the fully connected result are spliced ​​by channel to obtain the output result.

[0071] In the classification module, the output of the second fusion module passes through the fully connected layer (FC) to classify various diseases of the retinal OCT images (classified as normal or abnormal).

[0072] Furthermore, this embodiment uses the co-occurrence probability matrix of retinal disease labels to represent the adjacency matrix. First, the number of times labels appear simultaneously in pairs in the training set is counted, and the matrix M is: ij ∈R C×C , R indicates that the matrix elements are real numbers, C indicates the number of disease labels, M ij Indicates disease label i and disease labels j The number of times the categories appear at the same time. Then calculate the total number of times each label appears N i , then the adjacency matrix A satisfies:

[0073] A ij =M ij / N i

[0074] A ij The diagonal elements of the adjacency matrix A are set to 1, because their own probability of occurrence is 1, which prevents the node's own characteristics from being ignored during calculation and also alleviates the numerical instability caused by continuous multiplication. In addition, it should be noted that the constructed graph is a directed graph, A ij With A ji The calculation results are inconsistent. Since the co-occurrence probability matrix of retinal disease labels is only the global prior knowledge of the data set, there may be special cases in some samples. Therefore, this embodiment sets the adjacency matrix to dynamically update the parameters according to the back propagation of the deep learning network. The formula is as follows:

[0075]

[0076] in, It represents a matrix with the diagonal elements of the adjacency matrix A set to 1, and I is the identity matrix.

[0077] Furthermore, in this embodiment, the propagation mode between the graph convolutional neural network (GCN) layers is:

[0078]

[0079] Among them, H is the feature of each layer, l represents the current layer, W is the parameter to be learned, σ is the nonlinear activation function, and ReLU is used. is through The calculated degree matrix is ​​calculated as follows:

[0080]

[0081] Furthermore, the IAF-MCNet network first performs representation learning to obtain abstract visual features, then fuses the label association results and visual features through the graph convolutional label inline module (GLAM), and finally outputs the category probability P = [p1, p2, ..., p M ], where p i Indicates that disease label l appears in retinal OCT images i This embodiment also includes an optimization module, which uses a binary cross entropy function to calculate the loss between the prediction result of each category and the true label to back-propagate and optimize the model parameters of the entire IAF-MCNet network (i.e., the process of optimizing the first fusion module, the splicing module, and the second fusion module), and the calculation method is as follows:

[0082]

[0083] Among them, C represents the number of disease labels, yi ∈{0,1} is the true category label, when y i = 0, indicating that there is no disease label l in the image i When y i =1, it means that there is a disease label l in the image i .

[0084] Experimental analysis

[0085] The multi-label classification data of this embodiment was collected from multiple hospitals, including the First Affiliated Hospital of Soochow University, Henan Provincial Eye Hospital, Changsha Aier Eye Hospital, etc., and was taken by the regional scanning mode of the SDOCT equipment built in the laboratory. A total of 1,700 eyes of retinal OCT image data were collected, with an image size of 1024×512, and the lesions of each eye were annotated and diagnosed by professional ophthalmologists. In order to fully reflect the condition of the patient's retina, this study, under the guidance of an ophthalmologist, selected two high-quality single-channel two-dimensional B-scan images that can contain all lesion information for each eye for model training and testing. The data samples are randomly divided into 1020 eyes in the training set, 340 eyes in the validation set, and 340 eyes in the test set in a ratio of 3:1:1 in terms of eyes. There are 11 disease labels in total, including: posterior vitreous detachment, epiretinal membrane, subretinal fluid, choroidal neovascularization, drusen, retinal schisis, cystoid edema, exudation, ellipsoid band loss, choroidal atrophy and others. When all disease labels are marked as 0, it means that the current retinal image is a normal image. To facilitate the display of results, the disease labels are numbered. The label numbers (1)-(11) in the experimental result table represent the above lesions respectively. The number of times each disease label appears in pairs in the training set is shown in Table 1. The diagonal value is set to 0 (it is meaningless for a label to appear with itself). This table is used to calculate the adjacency matrix.

[0086] Table 1. The number of times each retinal disease label appears in pairs in the training set

[0087] Tag Number (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (1) 0 14 18 8 81 17 19 35 37 26 26 (2) 14 0 11 9 47 34 29 25 25 30 26 (3) 18 11 0 10 26 10 15 21 4 5 19 (4) 8 9 10 0 23 9 6 13 13 6 9 (5) 81 47 26 23 0 24 25 40 68 69 44 (6) 17 34 10 9 24 0 35 25 20 13 15 (7) 19 29 15 6 25 35 0 37 14 10 15 (8) 35 25 21 13 40 25 37 0 32 10 24 (9) 37 25 4 13 68 20 14 32 0 15 14 (10) 26 30 5 6 69 13 10 10 15 0 7 (11) 26 26 19 9 44 15 15 24 14 7 0

[0088] For computational and memory considerations, all images are downsampled to 320×320. To prevent network overfitting, a variety of online data augmentation methods are used to fully utilize the training set, including random up-down and left-right flipping, random rotation within ±30 degrees, and random contrast, brightness, and sharpness transformation. In addition, the input image pixel values ​​are normalized to [0,1] to accelerate network convergence.

[0089] The area under the curve (AUC) and average precision (AP) were used as classification evaluation indicators.

[0090] (a) Ablation experiment

[0091] In this embodiment, an IAF-MCNet network for classifying various diseases of retinal OCT images is designed through a multi-stage self-attention fusion module (MSFM), a local target perception module (LOAM) and a graph convolution label inline module (GLAM). Among them, the MSFM module extracts pixel spatial similarity information and captures long-distance feature dependencies; the LOAM module splits the encoder shallow local target and the underlying feature splicing fusion to enhance the perception of small lesions; the GLAM module counts the label co-occurrence probability matrix as an adjacency matrix, and captures the intrinsic connection of different labels through the graph convolution layer. In order to verify and analyze the effectiveness of the three modules, the following ablation experiments were performed: the baseline network used is a feature extractor with DenseNet121 as the backbone. The MSFM module, LOAM module and GLAM module are all pluggable structures. The three modules can be embedded in the baseline network separately or in groups to form different independent network structures for ablation experiments.

[0092] The ablation results of the average AUC and AP for each type of disease are shown in Tables 2 and 3. Different types of diseases present different morphological features in the image, and the classification effect of the classifier on different diseases is also different. For example, the disease labels (6) and (10) correspond to the diseases retinoschisis and choroidal atrophy, whose lesion areas are large and the morphological features are obvious. Their AUC evaluation index can reach about 95%, and the AP evaluation index can also reach 76% and 87%; while the disease labels (5) and (9) correspond to the diseases drusen and ellipsoidal band loss, whose lesion areas are small, the morphological features are not obvious, and they are easily disturbed by noise. The AUC evaluation index is only about 80%, and the AP evaluation index is only 72% and 48%, which are lower than other diseases. Different models may perform well on a certain disease label due to different design focuses and the randomness of model training. However, the task is to assist in the diagnosis and screening of retinal diseases, and the average index results of all disease categories are more important. The average results of the network model indicators composed of the three designed modules embedded in the baseline network are better than the average results of the baseline network indicators. The average results of the network model indicators composed of the three modules embedded in the baseline network in pairs are better than the network model composed of a single module embedded in the baseline network. The network model composed of the three modules embedded in the baseline network (i.e., the IAF-MCNet network) has achieved the best results in both AUC and AP, which shows the effectiveness of the three designed modules. In addition, this embodiment has four disease labels (4, 6, 7, and 11) that achieve the best results under the AUC indicator, and three disease labels (2, 3, and 6) that achieve the best results under the AP indicator, which is the most compared with all ablation experimental networks. And no disease label indicator is significantly lower than other networks, such as the Baseline+MSFM network has a significantly lower indicator than other networks in disease label (9), and the Baseline+LOAM network has an AUC indicator on disease label (8) that is the only one below 90% among all ablation experimental networks. In summary, this embodiment achieved the best classification performance in the ablation experiment. Compared with the Baseline network method, the average AUC index and the average AP index were improved by 2.08% and 3.4% respectively.

[0093] Table 2 Multi-label classification network ablation experiment AUC (%) results

[0094]

[0095] Table 3 Multi-label classification network ablation experiment AP (%) results

[0096]

[0097] (b) Comparative experiment

[0098] In order to test the superiority of the performance of the proposed multi-label classification network IAF-MCNet based on information inline fusion, it is compared and analyzed with other advanced deep learning classification networks. Table 4 shows the average AUC and AP evaluation indicators of all label categories of each network. It can be seen from Table 4 that compared with 10 advanced classification networks, this embodiment achieves the best performance in terms of average AUC and AP indicators. Compared with InceptionV4, which achieves the second best performance, the average AUC indicator and average AP indicator are improved by 1.51% and 1.74% respectively.

[0099] Table 4 Comparative experimental results of different networks on retinal OCT multi-label classification tasks

[0100] method mAUC(%) mAP(%) DPN68 87.68 61.08 ResNet50 89.25 62.91 EfficientNet-b3 87.46 64.80 SENet 88.80 65.10 CBAM 88.92 66.23 ResNeXt50 88.48 66.51 MobileNet 89.00 66.69 ResNeSt14d 89.87 65.53 InceptionV4 89.89 69.28 IAF-MCNet 91.40 71.02

[0101] Embodiment 2

[0102] This embodiment provides a multi-label classification method for retinal OCT image lesions, including:

[0103] Step S1: acquiring two retinal OCT images of the same subject;

[0104] Step S2: extracting deep features of the two retinal OCT images to obtain a deep feature map, and fusing the features output at each stage in the deep feature extraction process with the deep feature map to obtain an image feature map;

[0105] Step S3: extracting shallow features of the two retinal OCT images, wherein the shallow features include local target information, and splicing the shallow features including the local target information with the image feature map to obtain spliced ​​features;

[0106] Step S4: constructing input data based on different retinal disease labels, and inputting the input data into a graph convolutional neural network to obtain a vector matrix, and then fusing the vector matrix with the splicing features to obtain an output result;

[0107] Step S5: Classify various diseases of the retinal OCT image according to the output results.

[0108] Embodiment 3

[0109] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multi-label classification method for retinal OCT image lesions described in Embodiment 2 are implemented.

[0110] Embodiment 4

[0111] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the multi-label classification method for retinal OCT image lesions described in Embodiment 2 are implemented.

[0112] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0117] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A multi-label classification system for retinal OCT image lesions, characterized by: include: Acquisition module: used to acquire two retinal OCT images of the same subject; The first fusion module is used to extract the deep features of the two retinal OCT images to obtain a deep feature map, and at the same time, the features output at each stage in the deep feature extraction process are fused with the deep feature map to obtain an image feature map; A stitching module is used to extract shallow features of the two retinal OCT images, wherein the shallow features include local target information, and stitch the shallow features including the local target information with the image feature map to obtain stitching features; The second fusion module is used to construct input data based on different retinal disease labels, and input the input data into the graph convolutional neural network to obtain a vector matrix, and then fuse the vector matrix with the splicing features to obtain an output result; Classification module: used for classifying various diseases of retinal OCT images according to the output results; The first fusion module extracts the deep features of the two retinal OCT images to obtain a deep feature map, specifically: extracting the deep features of the two retinal OCT images through sequentially connected convolution units, first densely connected blocks, second densely connected blocks, third densely connected blocks, and fourth densely connected blocks to obtain a deep feature map; The convolution unit includes a convolution layer, a batch normalization layer, and a RELU activation layer connected in sequence; the first dense connection block includes 1×1 convolution and 3×3 convolution performed 6 times; the second dense connection block includes 1×1 convolution and 3×3 convolution performed 12 times; the third dense connection block includes 1×1 convolution and 3×3 convolution performed 24 times; the fourth dense connection block includes 1×1 convolution and 3×3 convolution performed 16 times; The first fusion module fuses the features output at each stage in the deep feature extraction process with the deep feature map to obtain an image feature map, specifically: Performing matrix addition on the outputs of the convolution unit, the first dense connection block, and the second dense connection block, respectively, through downsampling and 1×1 convolution, and the output of the third dense connection block through 1×1 convolution to obtain a first matrix addition result, then performing 1×1 convolution on the first matrix addition result to obtain a Query matrix, and modifying the size of the Query matrix; The output of the fourth densely connected block is dimensionally upgraded, and the result after dimension upgrading is split by channel to obtain the initial Key matrix, the initial Value matrix and the original visual feature matrix F split , performing 1×1 convolution on the initial Key matrix and the initial Value matrix to obtain a Key matrix and a Value matrix respectively, and modifying the size of the Key matrix; The modified size Query matrix and the modified size Key matrix are matrix multiplied to obtain a first matrix multiplication result, and the weight feature matrix W of the first matrix multiplication result is extracted through softmax activation, and the weight feature matrix W is matrix multiplied with the Value matrix to obtain a second matrix multiplication result, and the second matrix multiplication result is resized and 1×1 convolved before being combined with the original visual feature matrix F split Performing matrix addition to obtain a second matrix addition result, wherein the second matrix addition result is an image feature map; The shallow features of the two retinal OCT images are extracted in the stitching module, specifically: Extracting features output by the first densely connected block, the second densely connected block, and the third densely connected block respectively, and dividing the three output features into non-overlapping image blocks respectively; The image blocks corresponding to the three features are stacked along the channel dimension, and the three stacked image blocks are scaled into one-dimensional features through 1×1 convolution. Then, the results of global average pooling and global maximum pooling of the three one-dimensional features are matrix-added to obtain the third matrix addition result, the fourth matrix addition result, and the fifth matrix addition result; The third matrix addition result, the fourth matrix addition result and the fifth matrix addition result are respectively fused into three sub-vectors through two fully connected layers and ReLU activation layers, and finally the three sub-vectors are spliced ​​into a vector output according to channels, wherein the vector output is a shallow feature including local target information; In the second fusion module, input data is constructed based on different retinal disease labels, and the input data is input into the graph convolutional neural network to obtain a vector matrix, and then the vector matrix and the splicing feature are fused to obtain an output result, specifically: Different retinal disease labels are input into the word vector model to construct the initial graph structure, and an adjacency matrix is ​​constructed based on the frequency of occurrence of different retinal disease labels in pairs. Inputting the initial graph structure and the adjacency matrix as input data into the graph convolutional neural network at the same time to obtain a vector matrix; The splicing features are input into the fully connected layer to obtain a fully connected result, and then the fully connected result is matrix multiplied with the transposed vector matrix to obtain a third matrix multiplication result, and then the third matrix multiplication result is spliced ​​with the fully connected result by channel to obtain an output result.

2. The multi-label classification system for retinal OCT image lesions according to claim 1, characterized in that: The element A in the adjacency matrix A ij Satisfy the formula: A ij =M ij / N i Among them, the adjacency matrix A satisfies represents a matrix that sets the diagonal elements of the adjacency matrix A to 1, I represents the identity matrix, and M ij Indicates disease label i and disease labels j The number of simultaneous occurrences, N i Represents the total number of occurrences of each disease label.

3. The multi-label classification system for retinal OCT image lesions according to claim 1, characterized in that: It also includes an optimization module: used to optimize the process of the first fusion module, the splicing module and the second fusion module according to the classification results obtained by the classification module. The formula is: Among them, C represents the number of disease labels, p i Indicates that disease label l appears in retinal OCT images i The probability of y i ∈{0,1} is the true category label, when y i = 0, indicating that there is no disease label in the retinal OCT image. i When y i =1, indicating that there is a disease label l in the retinal OCT image i .

4. A multi-label classification method for retinal OCT image lesions, characterized by: include: Step S1: acquiring two retinal OCT images of the same subject; Step S2: extracting deep features of the two retinal OCT images to obtain a deep feature map, and fusing the features output at each stage in the deep feature extraction process with the deep feature map to obtain an image feature map; Step S3: extracting shallow features of the two retinal OCT images, wherein the shallow features include local target information, and splicing the shallow features including the local target information with the image feature map to obtain spliced ​​features; Step S4: constructing input data based on different retinal disease labels, and inputting the input data into a graph convolutional neural network to obtain a vector matrix, and then fusing the vector matrix with the splicing features to obtain an output result; Step S5: classifying various diseases of the retinal OCT image according to the output result; In step S2, the deep features of the two retinal OCT images are extracted to obtain a deep feature map, specifically: the deep features of the two retinal OCT images are extracted by sequentially connecting a convolution unit, a first densely connected block, a second densely connected block, a third densely connected block, and a fourth densely connected block to obtain a deep feature map; The convolution unit includes a convolution layer, a batch normalization layer, and a RELU activation layer connected in sequence; the first dense connection block includes 1×1 convolution and 3×3 convolution performed 6 times; the second dense connection block includes 1×1 convolution and 3×3 convolution performed 12 times; the third dense connection block includes 1×1 convolution and 3×3 convolution performed 24 times; the fourth dense connection block includes 1×1 convolution and 3×3 convolution performed 16 times; In step S2, the features output at each stage in the deep feature extraction process are fused with the deep feature map to obtain an image feature map, specifically: Performing matrix addition on the outputs of the convolution unit, the first dense connection block, and the second dense connection block, respectively, through downsampling and 1×1 convolution, and the output of the third dense connection block through 1×1 convolution to obtain a first matrix addition result, then performing 1×1 convolution on the first matrix addition result to obtain a Query matrix, and modifying the size of the Query matrix; The output of the fourth densely connected block is dimensionally upgraded, and the result after dimension upgrading is split by channel to obtain the initial Key matrix, the initial Value matrix and the original visual feature matrix F split , performing 1×1 convolution on the initial Key matrix and the initial Value matrix to obtain a Key matrix and a Value matrix respectively, and modifying the size of the Key matrix; The modified size Query matrix and the modified size Key matrix are matrix multiplied to obtain a first matrix multiplication result, and the weight feature matrix W of the first matrix multiplication result is extracted through softmax activation, and the weight feature matrix W is matrix multiplied with the Value matrix to obtain a second matrix multiplication result, and the second matrix multiplication result is resized and 1×1 convolved before being combined with the original visual feature matrix F split Performing matrix addition to obtain a second matrix addition result, wherein the second matrix addition result is an image feature map; The shallow features of the two retinal OCT images are extracted in step S3, specifically: Extracting features output by the first densely connected block, the second densely connected block, and the third densely connected block respectively, and dividing the three output features into non-overlapping image blocks respectively; The image blocks corresponding to the three features are stacked along the channel dimension, and the three stacked image blocks are scaled into one-dimensional features through 1×1 convolution. Then, the results of global average pooling and global maximum pooling of the three one-dimensional features are matrix-added to obtain the third matrix addition result, the fourth matrix addition result, and the fifth matrix addition result; The third matrix addition result, the fourth matrix addition result and the fifth matrix addition result are respectively fused into three sub-vectors through two fully connected layers and ReLU activation layers, and finally the three sub-vectors are spliced ​​into a vector output according to channels, wherein the vector output is a shallow feature including local target information; In step S4, input data is constructed based on different retinal disease labels, and the input data is input into the graph convolutional neural network to obtain a vector matrix, and then the vector matrix and the splicing feature are fused to obtain an output result, specifically: Different retinal disease labels are input into the word vector model to construct the initial graph structure, and an adjacency matrix is ​​constructed based on the frequency of occurrence of different retinal disease labels in pairs. Inputting the initial graph structure and the adjacency matrix as input data into the graph convolutional neural network at the same time to obtain a vector matrix; The splicing features are input into the fully connected layer to obtain a fully connected result, and then the fully connected result is matrix multiplied with the transposed vector matrix to obtain a third matrix multiplication result, and then the third matrix multiplication result is spliced ​​with the fully connected result by channel to obtain an output result.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-label classification method for retinal OCT image lesions as claimed in claim 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-label classification method for retinal OCT image lesions as claimed in claim 4 are implemented.