Retinal vein occlusion colorful image classification system based on attention mechanism

Abstract

The present invention discloses a retinal vein occlusion colorful image classification system based on an attention mechanism, comprising: an acquisition module configured to acquire retinal colorful images; a training module configured to extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on an attention mechanism, and train a classification model according to the fusion features; a classification module configured to obtain a pathological classification result of retinal vein occlusion for the retinal colorful images to be classified according to the trained classification model. Multi-scale image features are used as the basis for classification, and an attention mechanism is introduced to fully fuse the global and local features at multiple scales, thereby improving the classification accuracy of retinal vein occlusion colorful images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a retinal vein occlusion colorful image classification system based on an attention mechanism. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Retinal vein occlusion can be divided into central retinal vein occlusion and branch retinal vein occlusion according to the location of blood vessel occlusion. Medical experts are still unclear about the exact pathogenesis of retinal vein occlusion, and there is no effective drug treatment in clinical practice. Therefore, timely diagnosis and intervention are very necessary.

[0004] For images of visually impaired patients obtained during clinical examinations, ophthalmologists need to rely on their own knowledge of medical imaging, ophthalmic pathology, etc., to accurately classify each retinal image of the patient in order to conduct the next examination or adopt an effective treatment plan for the patient. However, classifying each retinal image of the patient requires a lot of time and energy for ophthalmologists, and it is also easy to make subjective misjudgments in this process.

[0005] Therefore, artificial intelligence technology for retinal pathology images can help doctors detect diseases such as retinal vein occlusion at an early stage. However, existing deep learning methods only transplant the classification algorithm of natural images to retinal images, without considering the characteristics of retinal images containing multi-layer vascular tissue structures, so these methods have low accuracy in classifying retinal vein occlusion images. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a retinal vein occlusion colorful image classification system based on attention mechanism, which takes multi-scale image features as the classification basis and introduces the attention mechanism to fully integrate the global and local features at multi-scales, thereby improving the classification accuracy of retinal vein occlusion colorful images.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a retinal vein occlusion colorful image classification system based on an attention mechanism, comprising:

[0009] An acquisition module is configured to acquire a retinal color image;

[0010] The training module is configured to extract features from the retinal colorful image, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model according to the fused features;

[0011] The classification module is configured to obtain a pathological classification result of retinal vein occlusion for the retinal colorful image to be classified according to the trained classification model.

[0012] As an optional implementation, the classification system further includes a preprocessing module for preprocessing the retinal color image by cropping, image enhancement and scale adjustment.

[0013] As an optional implementation, the preprocessing module includes:

[0014] A cropping module is used to crop the edge background of the retinal colorful image;

[0015] An image enhancement module is used to perform image enhancement processing on the cropped retinal color image, including operations of rotation, flipping, translation, sharpening, adding noise and increasing contrast;

[0016] The scale adjustment module is used to perform scale adjustment on the enhanced retinal color image to meet the input requirements of the classification model.

[0017] As an optional implementation, the acquisition module acquires four modalities of retinal colorful images, specifically including: a 488nm blue light reflection image, a 515nm green light reflection image, an 820nm infrared light reflection image and a standard colorful image.

[0018] As an optional implementation, the classification model includes a feature extraction network, and the feature extraction network uses a multi-layer convolutional neural network to extract features from retinal color images of multiple modalities.

[0019] As an optional implementation, the classification model includes a feature fusion network, which introduces an attention mechanism to perform global and local feature fusion on image features of different branches by capturing contextual long-distance information.

[0020] As an optional implementation, the classification model includes a classification network, and the classification network adopts an improved fully connected layer network for pathological classification of retinal vein occlusion in retinal static color images.

[0021] As an optional implementation, the classification network uses the same classification network or multiple different classification networks according to the modality type of the retinal color image.

[0022] In a second aspect, the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, a retinal vein occlusion image classification method based on an attention mechanism is completed, and the classification method includes:

[0023] Get retinal color images;

[0024] Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features;

[0025] The pathological classification results of retinal vein occlusion are obtained for the retinal colorful images to be classified according to the trained classification model.

[0026] In a third aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, a retinal vein occlusion image classification method based on an attention mechanism is performed, and the classification method comprises:

[0027] Get retinal color images;

[0028] Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features;

[0029] The pathological classification results of retinal vein occlusion are obtained for the retinal colorful images to be classified according to the trained classification model.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention proposes a retinal vein occlusion colorful image classification system based on attention mechanism. Based on the general deep learning classification model, a multimodal attention mechanism classification model is proposed. Four modal images of retinal pathological images are input at the same time, and the image features of the four modalities are used as the classification basis at the same time. Multiple features are extracted according to the image information, and the attention mechanism is introduced to fully integrate the global and local features between the modalities. Compared with the traditional image classification method, the information considered is more comprehensive, the robustness of the algorithm is improved, and the classification accuracy of retinal vein occlusion colorful images is improved.

[0032] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0034] Figure 1 A schematic diagram of retinal vein occlusion colorful image classification based on attention mechanism provided in Example 1 of the present invention;

[0035] Figure 2(a)-2(d) A schematic diagram of a retinal colorful image provided in Example 1 of the present invention;

[0036] Figure 3 This is a main structure diagram of the AttResNet convolutional neural network model provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment provides a retinal vein occlusion image classification system based on an attention mechanism, including:

[0043] An acquisition module is configured to acquire a retinal color image;

[0044] The training module is configured to extract features from the retinal colorful image, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model according to the fused features;

[0045] The classification module is configured to obtain a pathological classification result of retinal vein occlusion for the retinal colorful image to be classified according to the trained classification model.

[0046] In this embodiment, the acquisition module acquires the collected retinal colorful images including four modalities, specifically: 488nm blue light reflection image, 515nm green light reflection image, 820nm infrared light reflection image and standard colorful image after signal integration and color rendering. Each eye is scanned once to acquire a total of 4 images, as shown in the following figure. Figure 2(a)-2(d) As shown; the retinal color images are further divided into training set and test set.

[0047] It is understandable that the retinal color image can be pre-stored in the memory of the computer device, and when it needs to be processed, the processor reads the image directly from the memory of the computer device. Of course, the processor can also obtain the retinal color image from an external device. For example, the retinal color image of the object to be detected is stored in the cloud, and when processing operations are required, the processor obtains the retinal color image of the object to be detected from the cloud. This embodiment does not limit the specific way in which the processor obtains the retinal color image.

[0048] In this embodiment, the retinal colorful images are classified as normal or diseased according to the test report, and the four retinal images of the same eye of each patient are placed in the same folder; at the same time, due to the imbalance in the number of the two samples, some normal image data need to be randomly reserved, and this part of the data is predicted by the trained classification model to test the classification effect.

[0049] In this embodiment, the classification system further includes a preprocessing module for preprocessing the retinal colorful image by cropping, image enhancement and scale adjustment, specifically including:

[0050] The cropping module is used to crop the retina color image to cut off the black edge background.

[0051] The image enhancement module is used to perform image enhancement processing on the cropped retinal color images; image enhancement processing purposefully emphasizes the overall or local characteristics of the image and expands the differences between the features of different objects in the image; at the same time, by rotating, flipping, translating, sharpening, increasing noise, increasing contrast and other operations on the image, the number of images is increased to prevent the network from overfitting, and the robustness of the classification network can also be enhanced.

[0052] The scale adjustment module is used to perform scale adjustment processing on the image-enhanced retinal color image, and adjust the size of the image-enhanced retinal color image to 224×224 pixels to meet the input requirements of the classification model.

[0053] In this embodiment, the classification model is trained based on the scale-adjusted retinal color image;

[0054] Specifically, the classification model adopts a deep learning classification framework as the backbone network, including a feature extraction network, a feature fusion network and a classification network; wherein the feature extraction network adopts a multi-layer convolutional neural network as the backbone network to extract features of four retinal color images; the feature fusion network introduces an attention mechanism to perform global and local feature fusion of image features on different branches; the classification network adopts an improved fully connected layer network to perform pathological classification of retinal vein occlusion on retinal static color images.

[0055] Specifically, the four retinal color images are simultaneously input into the feature extraction network, and the output of the first fully connected layer is used as the extracted deep features; the image features of the four modal retinal color images are input into the feature fusion network, and the characteristics of contextual long-distance information are captured through the attention mechanism to fuse the features between the modalities; then the same classification network or different classification networks are used to extract features in a targeted manner.

[0056] In this embodiment, the feature extraction network is composed of multiple convolutional layers, and an AttResNet network is constructed based on the most commonly used ResNet network. Figure 3 As shown in the figure, the first four convolution blocks are used as the feature extraction network. The original image of input (w, h, 3) will be downsampled 16 times after passing through four convolutional network blocks, that is, the output feature map is (w / 16, h / 16, 3).

[0057] In the classification model of this embodiment, the four modal images share a feature extraction network, that is, they are simultaneously input into the same network and then output their respective feature maps; the first few layers of the network learn relatively generalized features, such as edges, colors, etc., while the features learned by the later layers of the network are higher-level.

[0058] In this embodiment, the feature fusion network introduces the attention mechanism module and also adopts different feature extraction methods. In the feature fusion network and the feature extraction network, images captured by laser scanning of different wavelengths of the same eye are used.

[0059] In this embodiment, the trained classification model is used to classify the retinal colorful images to be classified into normal and diseased ones, and the fused image features are used to obtain the classification results through the softmax classifier.

[0060] This embodiment classifies retinal vein occlusion images based on the attention mechanism, improves the deep learning classification model, inputs four modal energy spectrum images at the same time, detects whether the retina has vein occlusion lesion characteristics, and extracts appearance features from the corresponding images for classification. In this way, classification can be performed by extracting retinal colorful pathological image features under different modalities.

[0061] In further embodiments, there is also provided:

[0062] An electronic device includes a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein when the computer instructions are run by the processor, a retinal vein occlusion image classification method based on an attention mechanism is completed, including:

[0063] Get retinal color images;

[0064] Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features;

[0065] The pathological classification results of retinal vein occlusion are obtained for the retinal colorful images to be classified according to the trained classification model.

[0066] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0067] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0068] A computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, a retinal vein occlusion image classification method based on an attention mechanism is performed, comprising:

[0069] Get retinal color images;

[0070] Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features;

[0071] The pathological classification results of retinal vein occlusion are obtained for the retinal colorful images to be classified according to the trained classification model.

[0072] The method can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0073] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0074] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A retinal vein occlusion image classification system based on attention mechanism, characterized by: include: An acquisition module is configured to acquire a retinal color image; The acquisition module acquires four modal retinal colorful images, specifically including: a 488nm blue light reflection image, a 515nm green light reflection image, an 820nm infrared light reflection image, and a standard colorful image; The training module is configured to extract features from the retinal colorful image, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model according to the fused features; A classification module is configured to obtain a pathological classification result of retinal vein occlusion based on a trained classification model for the retinal colorful image to be classified; The classification model includes a feature extraction network, which uses a multi-layer convolutional neural network to extract features from retinal colorful images of four modalities; the feature extraction network constructs an AttResNet network based on a ResNet network; Multiple modal images share the feature extraction network, that is, they are simultaneously input into the same network and then output their respective feature maps.

2. The retinal vein occlusion image classification system based on attention mechanism as claimed in claim 1, characterized in that: The classification system also includes a preprocessing module for preprocessing the retinal colorful image by cropping, image enhancement and scale adjustment.

3. The retinal vein occlusion image classification system based on attention mechanism as claimed in claim 2, characterized in that: The preprocessing module comprises: A cropping module is used to crop the edge background of the retinal colorful image; An image enhancement module is used to perform image enhancement processing on the cropped retinal color image, including operations of rotation, flipping, translation, sharpening, adding noise and increasing contrast; The scale adjustment module is used to perform scale adjustment on the enhanced retinal color image to meet the input requirements of the classification model.

4. The retinal vein occlusion image classification system based on attention mechanism as claimed in claim 1, characterized in that: The classification model includes a feature fusion network, which introduces an attention mechanism to perform global and local feature fusion on image features of different branches by capturing contextual long-distance information.

5. The retinal vein occlusion image classification system based on attention mechanism as claimed in claim 1, characterized in that: The classification model includes a classification network, and the classification network adopts an improved fully connected layer network to perform pathological classification of retinal vein occlusion on retinal static color images.

6. The retinal vein occlusion image classification system based on attention mechanism as claimed in claim 5, characterized in that: The classification network adopts the same classification network or multiple different classification networks according to the modality type of the retinal color image.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, a retinal vein occlusion image classification method based on an attention mechanism is completed. The classification method comprises: Get retinal color images; Obtain four modalities of retinal color images, including: 488nm blue light reflection image, 515nm green light reflection image, 820nm infrared light reflection image and standard color image; Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features; To obtain the pathological classification results of retinal vein occlusion for the retinal colorful images to be classified according to the trained classification model; The classification model includes a feature extraction network, which uses a multi-layer convolutional neural network to extract features from retinal colorful images of four modalities; the feature extraction network constructs an AttResNet network based on a ResNet network; Multiple modal images share the feature extraction network, that is, they are simultaneously input into the same network and then output their respective feature maps.

8. A computer-readable storage medium, characterized in that: Used to store computer instructions, when the computer instructions are executed by the processor, a retinal vein occlusion image classification method based on an attention mechanism is completed, the classification method comprising: Get retinal color images; Obtain four modalities of retinal color images, including: 488nm blue light reflection image, 515nm green light reflection image, 820nm infrared light reflection image and standard color image; Extract features from retinal colorful images, perform global and local feature fusion on the extracted image features of different branches based on the attention mechanism, and train the classification model based on the fused features; To obtain the pathological classification results of retinal vein occlusion for the retinal colorful images to be classified according to the trained classification model; The classification model includes a feature extraction network, which uses a multi-layer convolutional neural network to extract features from retinal colorful images of four modalities; the feature extraction network constructs an AttResNet network based on a ResNet network; Multiple modal images share the feature extraction network, that is, they are simultaneously input into the same network and then output their respective feature maps.

Images