Eye fundus image recognition method, system and equipment based on deep learning

Abstract

The invention discloses an eye fundus image recognition method, system and device based on deep learning, the efficiency and precision of eye fundus disease recognition are improved through a portable handheld terminal and a cloud database, the portable handheld terminal comprises an eye fundus image feature acquisition module, an eye fundus image recognition and judgment device and a man-machine interaction module, wherein the eye fundus image recognition and judgment device comprises a feature engineering module, an image recognition module and a visual auxiliary fault module, the feature engineering module is responsible for data expansion, enhancement and normalization, the image recognition module performs image classification prediction by constructing a deep learning model, and model construction, parameter initialization, model training and pruning optimization are included. The visual auxiliary diagnosis module assists doctors in diagnosis by generating a thermodynamic diagram. Remote storage and analysis of data can be realized, the diagnosis efficiency is improved, the visual loss risk is reduced, and the system is particularly suitable for resource-limited regions.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning image classification and recognition, and specifically relates to a fundus image recognition method, system and equipment based on deep learning. Background Art

[0002] Cataracts, myopia, hyperopia and other retinal diseases can be detected and prevented through regular eye examinations. Treatment options include surgery, medication, and the use of glasses or contact lenses.

[0003] However, there are still such diseases that are not discovered and treated in the early stage. In order to solve this problem, artificial intelligence technology has been used to study the classification, recognition and semantic segmentation of fundus images, showing great potential. It is expected that with the help of classified and recognized fundus images, subsequent extensive screening and early diagnosis of retinal-related diseases can be achieved, thereby alleviating the shortage of medical resources to a certain extent. Through fundus image analysis, glaucoma risk prediction, cataract surgery planning, etc. through automated technology, artificial intelligence provides ophthalmologists with more accurate and efficient assistance. Summary of the invention

[0004] In order to solve the shortcomings of the prior art and achieve the purpose of improving the classification and recognition accuracy and efficiency of fundus images, the present invention adopts the following technical solutions:

[0005] The fundus image recognition method based on deep learning includes the following steps:

[0006] Step S1: constructing a neural network model for fundus image recognition, including a feature pre-extraction, a backbone network, and a classifier connected in sequence;

[0007] The backbone network is constructed by stacking core modules based on residual connection, coordinate attention mechanism, and spatial channel reshaping convolution. Among them, residual connection fuses context information, and coordinate attention mechanism is used to capture long-range dependencies and retain precise location information at the same time, which helps the model to more accurately identify different structures and abnormal features in fundus images. Spatial channel reshaping convolution uses spatial reconstruction unit and channel reconstruction unit to reconstruct the spatial and channel aspects of feature maps, thereby reducing redundant information. The reconstructed feature maps are downsampled using 3*3 convolution and pooling.

[0008] Step S2: Initialize the model parameters, and make the average value of the activation output of each layer of the backbone network approach zero, while keeping the average standard deviation close to 1, so as to effectively maintain the flow of information in the forward propagation and backward propagation process of the model, and keep the variance of the input signals of different layers relatively consistent;

[0009] Step S3: training the neural network model for fundus image recognition, using the cross entropy loss function, and preferably using the Adam optimizer to improve training efficiency;

[0010] Step S4: Pruning optimization, identifying and removing components with low contribution in the entire convolution kernel or channel layer of the model, while maintaining the model recognition accuracy, significantly reducing the number of model parameters and computational complexity, thereby accelerating the inference speed and improving the deployment efficiency and operating performance of the model in practical applications.

[0011] Furthermore, in step S1, the classifier flattens the feature map output by the backbone network through global average pooling, and then uses a multi-layer perceptron and a Softmax layer to perform category mapping.

[0012] Furthermore, in step S2, He normal initialization is adopted, starting from a mean of 0 and a standard deviation of The initial weights are randomly drawn from the normal distribution of , and the formula is expressed as:

[0013]

[0014] Among them, n in represents the number of neuron inputs in this layer, W represents the weight matrix, It means that the mean is 0 and the variance is The normal distribution of .

[0015] Furthermore, in step S3, the model training conditions are set, and the cosine learning rate annealing algorithm is introduced as a learning rate scheduling strategy to dynamically adjust the learning rate to promote faster and better convergence of the model. The cross entropy loss function is used to train the model, and the formula is as follows:

[0016]

[0017] Among them, y i is the unique hot encoding of the true label, indicating the actual value of category i, p i represents the probability that the model predicts that it belongs to category i, and C represents the total number of categories.

[0018] The fundus image recognition system based on deep learning includes a feature engineering module, an image recognition module, and a visualization auxiliary module. The image recognition module adopts the fundus image recognition method based on deep learning, and constructs a convolutional neural network model that integrates residual connection, coordinate attention mechanism and spatial channel reshaping convolution. The initialized model is trained on the fundus image data set expanded by the feature engineering module, and finally the converged model is pruned to derive an efficient and high-precision optimization model.

[0019] Furthermore, the feature engineering module expands and enhances the acquired fundus image data set, including a data expansion unit, a data enhancement unit and a data normalization unit;

[0020] Among them, the data expansion unit includes pre-trained network fine-tuning and data generation;

[0021] Fine-tuning of the pre-trained network adopts a generator and discriminator structure, and introduces a network model pre-trained on a large-scale dataset to accelerate the training process and improve the generation quality. During the fine-tuning of the network, the generator and discriminator focus on learning high-level features by fixing the low-level convolutional layers, so that they can better adapt to the specific feature expression of fundus images. At the same time, the category label is combined as a conditional input so that the generator can generate images according to the features of different categories, ensuring a high degree of consistency between the generated images and the category labels.

[0022] Data generation: by combining random noise vectors with category labels for image generation, the generator can generate diversified fundus images that meet conditional constraints. The generation process continuously optimizes the adversarial training of the generator and the discriminator, so that the generated images are continuously improved in terms of realism and category consistency. At the same time, the discriminator verifies the authenticity and category matching of the generated images through conditional input, ensuring the high quality of the generated data and the consistency of the medical image features.

[0023] The data enhancement unit performs data enhancement processing on the original and expanded data to enrich the diversity of the data and ensure that the model has high robustness. The enhancement methods include: random brightness, random contrast, Gaussian blur, rotation, translation, scaling, etc.

[0024] The data normalization unit performs mean-variance normalization on the expanded and enhanced image data before it is input into the image recognition module to fine-tune the distribution characteristics of the training data set, aiming to significantly improve the efficiency of model training, accelerate its convergence process, and thus improve the overall performance and generalization ability of the model.

[0025] Furthermore, the data normalization unit adopts a global mean variance normalization technique to obtain a pixel value a' by normalizing each pixel value a of the image by its mean variance. The calculation formula is as follows:

[0026]

[0027] Among them, μ represents the mean value of image pixels, and the calculation formula is as follows:

[0028]

[0029] Where N represents the number of pixels, σ represents the standard deviation of the image pixels, and the calculation formula is:

[0030]

[0031] Furthermore, the visualization auxiliary module uses the pre-trained model and input data to perform gradient reverse calculation, thereby drawing the image parts that have a positive impact on classification into a heat map, so that the doctor can pay more attention to the possible location of the lesion.

[0032] Furthermore, the process of drawing the heat map is as follows:

[0033] First, for the target category, calculate its convolution feature map A k Its gradient:

[0034]

[0035] Among them, c represents the target category, y c represents the output of the target category, and k represents the number of convolution layers;

[0036] Then, these gradients are globally averaged pooled in the spatial dimension to obtain the weights of the feature map:

[0037]

[0038] Among them, i and j represent the feature map A respectively. k The row and column index of the elements in, Z represents the feature space A k The size of , used to average the gradient and;

[0039] Finally, these weights are linearly combined with the feature map, and then the Grad-CAM heat map is obtained through the ReLU activation function:

[0040]

[0041] A fundus image recognition device based on deep learning is applied to a portable handheld terminal, and includes a fundus image feature acquisition module, a fundus image recognition and judgment module, and a human-computer interaction module. The fundus image recognition and judgment module includes the image recognition module and the visualization auxiliary module of the fundus image recognition system based on deep learning, and classifies and performs visualization auxiliary processing on the fundus image obtained by the fundus image feature acquisition module, so that a cloud database stores and manages the fundus image data collected and recognized by the portable handheld terminal, thereby realizing remote storage, processing, and analysis of the data.

[0042] The advantages and beneficial effects of the present invention are:

[0043] The fundus image recognition method, system and device based on deep learning of the present invention utilize portable devices to collect fundus images for auxiliary medical diagnosis. Specifically, Raspberry Pi is used as the core processing unit to drive a dedicated fundus camera to capture fundus images. Subsequently, a deep neural network is used to analyze the collected images, extract key information, and store it in a MySQL database for further use. The present invention has rapid analysis capabilities, can respond quickly to fundus lesion images of varying degrees, and provide timely medical decision support to doctors. It is particularly suitable for the classification and identification of fundus disease images where the characteristics of fundus lesions are difficult to judge or not obvious. At the same time, a portable handheld device is designed to meet the different eye position requirements of patients with fundus diseases, so as to quickly adjust the observation angle and adapt to the examination conditions of different patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method in an embodiment of the present invention.

[0045] Figure 2 It is a schematic diagram of the execution process of the system in an embodiment of the present invention.

[0046] Figure 3 It is a schematic diagram of the structure of the device in the embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0048] like Figure 1 As shown, the fundus image recognition method based on deep learning includes the following steps:

[0049] Step S1: construct a neural network model for fundus image recognition, including feature pre-extraction, backbone network, and classifier.

[0050] Feature pre-extraction performs preliminary feature extraction by using two layers of large kernel convolution;

[0051] The backbone network is constructed by stacking core modules based on residual connection, coordinate attention mechanism, and spatial channel reshaping convolution. Among them, residual connection fuses context information, and the coordinate attention mechanism can simultaneously capture long-range dependencies and retain precise location information, which helps the model to more accurately identify different structures and abnormal features in fundus images. The spatial channel reshaping convolution uses spatial reconstruction units and channel reconstruction units to reconstruct the spatial and channel aspects of the feature map, thereby reducing redundant information. The reconstructed feature map is downsampled using 3*3 convolution and pooling.

[0052] The classifier flattens the feature map output by the backbone network through global average pooling, and then uses a multi-layer perceptron and a Softmax layer for category mapping.

[0053] Step S2: Initialize the model parameters, using the He initialization technique to make the average value of the activation output of each layer of the backbone network approach zero, while keeping the average standard deviation close to 1, thereby effectively maintaining the flow of information in the forward and backward propagation processes of the model, and keeping the variance of the input signals of different layers relatively consistent.

[0054] Specifically, the He normal initialization is used, starting from a mean of 0 and a standard deviation of The initial weights are randomly drawn from the normal distribution of , and the formula is expressed as:

[0055]

[0056] Among them, n in represents the number of neuron inputs in this layer, W represents the weight matrix, It means that the mean is 0 and the variance is The normal distribution of .

[0057] Step S3: Train the neural network model for fundus image recognition, use the cross entropy loss function, and select the Adam optimizer to improve training efficiency.

[0058] During the training process, if the model converges or the accuracy does not improve after 20 consecutive iterations, the training will be automatically terminated. At the same time, the cosine learning rate annealing algorithm is introduced as the learning rate scheduling strategy to dynamically adjust the learning rate to promote faster and better convergence of the model. The formula of the cross entropy loss function is defined as follows:

[0059]

[0060] Among them, y i is the unique hot encoding of the true label, indicating the actual value of category i, p i represents the probability that the model predicts that it belongs to category i, and C represents the total number of categories.

[0061] Step S4: Pruning optimization, by identifying and removing components with low contribution in the entire convolution kernel or channel layer, while maintaining the model recognition accuracy, significantly reducing the number of model parameters and computational complexity, thereby accelerating the inference speed and improving the deployment efficiency and operating performance of the model in practical applications.

[0062] like Figure 2As shown, the fundus image recognition system based on deep learning includes a feature engineering module, an image recognition module, and a visualization auxiliary module, wherein the image recognition module adopts the fundus image recognition method based on deep learning, and constructs a convolutional neural network model that integrates residual connection, coordinate attention mechanism and spatial channel reshaping convolution. The initialized model is trained on the fundus image data set expanded by the feature engineering module, and finally the converged model is pruned to derive an efficient and high-precision optimization model.

[0063] The feature engineering module expands and enhances the acquired fundus image data set, including a data expansion unit, a data enhancement unit and a data normalization unit.

[0064] The data expansion unit uses the GAN network to expand the data based on the existing data. It includes two parts: cGAN pre-trained network fine-tuning and data generation. The conditionally generated cGAN ensures that the generated fundus image remains consistent and targeted with the original data by controlling the conditional labels.

[0065] The fine-tuning of the pre-trained network adopts a cGAN-based generator and discriminator structure, and introduces a network model pre-trained on a large-scale dataset to accelerate the training process and improve the generation quality. During the network fine-tuning process, the generator and discriminator focus on learning high-level features by fixing the low-level convolutional layers, so that they can better adapt to the specific feature expression of fundus images. At the same time, combined with the category label as conditional input, the generator can generate images according to the characteristics of different categories, ensuring a high degree of consistency between the generated images and the category labels.

[0066] Data generation uses the cGAN architecture, which combines random noise vectors with category labels for image generation, so that the generator can generate diverse fundus images that meet conditional constraints. The generation process continuously optimizes the adversarial training of the generator and the discriminator, so that the generated images are continuously improved in terms of realism and category consistency. At the same time, the discriminator verifies the authenticity and category matching of the generated images through conditional input, ensuring the high quality of the generated data and the consistency of the medical image features.

[0067] The data enhancement unit performs data enhancement processing on the original and expanded data to enrich the diversity of the data and ensure that the model has high robustness. The enhancement methods include: random brightness, random contrast, Gaussian blur, rotation, translation, scaling, etc.

[0068] The data normalization unit performs mean-variance normalization on the expanded and enhanced image data before inputting it into the image recognition module. Specifically, the global mean-variance normalization technology is adopted to fine-tune the distribution characteristics of the training data set, aiming to significantly improve the efficiency of model training, accelerate its convergence process, and thus improve the overall performance and generalization ability of the model.

[0069] For each pixel value a of the image, the pixel value a' is obtained by normalizing its mean variance. The formula is as follows:

[0070]

[0071] Among them, μ represents the mean value of image pixels, and the calculation formula is as follows:

[0072]

[0073] Where N represents the number of pixels, σ represents the standard deviation of the image pixels, and the calculation formula is:

[0074]

[0075] The image recognition module performs forward reasoning on the input fundus image using the deep learning model proposed in this patent to obtain the corresponding classification prediction results.

[0076] The visualization auxiliary module uses the pre-trained model and input data to perform gradient reverse calculation, thereby drawing the image parts that have a positive impact on classification into a heat map, so that doctors can pay more attention to the possible location of the lesion.

[0077] Specifically, the visualization assistance module integrates the Grad-CAM (Gradient-weighted ClassActivation Mapping) algorithm, which aims to further improve the accuracy and efficiency of doctors' recognition of fundus disease images. The Grad-CAM algorithm calculates the gradient of the input image to the model prediction result to identify the image areas that have a positive contribution to the classification decision, and then highlights these areas to form an intuitive heat map. The specific formula is as follows:

[0078] First, for the target category, we calculate its convolution feature map A k Its gradient:

[0079]

[0080] Among them, c represents the target category, y c represents the output of the target category, and k represents the number of convolutional layers.

[0081] Then, these gradients are globally averaged pooled in the spatial dimension to obtain the weights of the feature map:

[0082]

[0083] Among them, i and j represent the feature map A respectively. k The row and column indices of the elements in, Z represents the feature space A k The size of , used to average the gradient and .

[0084] Finally, these weights are linearly combined with the feature map, and then the Grad-CAM heat map is obtained through the ReLU activation function:

[0085]

[0086] like Figure 3 As shown, a fundus image recognition device based on deep learning is applied to a portable handheld terminal, including a fundus image feature acquisition module, a fundus image recognition and judgment module, and a human-computer interaction module; wherein the fundus image recognition and judgment module adopts the image recognition module and the visualization auxiliary module of the fundus image recognition system based on deep learning to classify and perform visualization auxiliary processing on the fundus image obtained by the fundus image feature acquisition module, so that the cloud database stores and manages the fundus image data collected and recognized by the portable handheld terminal, thereby realizing remote storage, processing and analysis of the data.

[0087] Among them, the feature acquisition of fundus images is equipped with a professional fundus camera, which can capture high-definition fundus images. The image data is efficiently transmitted through the FPC cable through the MIPI interface specification and directly connected to the dedicated CSI of the Raspberry Pi. This design ensures the high speed and stability of image data transmission.

[0088] The fundus image recognition system uses a Raspberry Pi host, including a feature engineering module, an image recognition module, and a visualization auxiliary module; the Raspberry Pi host receives the fundus image collected by the fundus image feature acquisition module, and uses the Grad-CAM algorithm to calculate the gradient of the input image to the model prediction result, identify the image area that has a positive contribution to the classification decision, and form an intuitive heat map to assist doctors in diagnosis. The Raspberry Pi processing unit is also connected to a display device for displaying and analyzing data, and has the function of receiving user instructions. The cloud database is specially designed for the fundus image recognition system, and uses a small and medium-sized MySQL database system to store and manage data. MySQL is a widely used open source relational database management system. The present invention configures a MySQL database under the Linux operating system environment to achieve efficient storage and query of data.

[0089] The system collects fundus image data and transmits it to the Raspberry Pi processing unit, which is pre-installed with a specific software algorithm and control program for image analysis. The processing unit receives data from fundus image feature collection, uses a built-in algorithm for analysis and processing, and stores the analysis results in a cloud database. The Raspberry Pi 4B model chip is selected as the processing core of the present invention. After completing the algorithm analysis of the image data, the chip stores the result data in the cloud database, and collects fundus images to interact with the user through the display screen of the handheld terminal.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fundus image recognition method based on deep learning, characterized in that The steps include: Step S1: constructing a neural network model for fundus image recognition, including a feature pre-extraction, a backbone network, and a classifier connected in sequence; The backbone network is constructed by stacking core modules based on residual connection, coordinate attention mechanism, and spatial channel reshaping convolution. Among them, residual connection fuses context information, coordinate attention mechanism is used to capture long-range dependencies and retain precise location information at the same time, and spatial channel reshaping convolution uses spatial reconstruction unit and channel reconstruction unit to reconstruct the spatial and channel aspects of feature maps. The reconstructed feature maps are downsampled using convolution and pooling. Step S2: Initialize the model parameters, and make the average value of the activation output of each layer of the backbone network approach 0, while keeping the average standard deviation close to 1; Step S3: training a neural network model for fundus image recognition; Step S4: Pruning optimization, identifying and removing components with low contribution in the entire convolution kernel or channel layer of the model.

2. The deep learning-based fundus image recognition method according to claim 1, characterized in that: In step S1, the classifier flattens the feature map output by the backbone network through global average pooling, and then uses a multi-layer perceptron and a Softmax layer to perform category mapping.

3. The method for fundus image recognition based on deep learning according to claim 1, characterized in that: In step S2, He normal initialization is adopted, starting from a mean of 0 and a standard deviation of The initial weights are randomly drawn from the normal distribution of , and the formula is expressed as: Among them, n in represents the number of neuron inputs in this layer, W represents the weight matrix, It means that the mean is 0 and the variance is The normal distribution of .

4. The deep learning-based fundus image recognition method according to claim 1, characterized in that: In step S3, the model training conditions are set, and the cosine learning rate annealing algorithm is introduced as the learning rate scheduling strategy to dynamically adjust the learning rate. The cross entropy loss function is used to train the model. The formula is as follows: Among them, y i is the unique hot encoding of the true label, indicating the actual value of category i, p i represents the probability that the model predicts that it belongs to category i, and C represents the total number of categories.

5. The fundus image recognition system based on deep learning includes a feature engineering module, an image recognition module, and a visualization auxiliary module, and is characterized by: The image recognition module adopts the fundus image recognition method based on deep learning as described in claim 1, and constructs a convolutional neural network model that integrates residual connection, coordinate attention mechanism and spatial channel reshaping convolution. The initialized model is trained on the fundus image data set expanded by the feature engineering module, and finally the converged model is pruned to derive the optimized model.

6. The fundus image recognition system based on deep learning according to claim 5, characterized in that: The feature engineering module expands and enhances the acquired fundus image data set, including a data expansion unit, a data enhancement unit and a data normalization unit; Among them, the data expansion unit includes pre-trained network fine-tuning and data generation; Fine-tuning of the pre-trained network, using a generator and discriminator structure, by introducing a network model pre-trained on a large-scale dataset; during the network fine-tuning process, the generator and discriminator focus on learning high-level features by fixing the low-level convolutional layers; at the same time, the category label is combined as a conditional input so that the generator can generate images based on the features of different categories; Data generation: Image generation is performed by combining random noise vectors with category labels, so that the generator can produce diversified fundus images that meet conditional constraints. The generation process continuously optimizes the adversarial training of the generator and the discriminator. At the same time, the discriminator verifies the authenticity and category matching of the generated image through conditional input. A data enhancement unit performs data enhancement processing on the original and expanded data; The data normalization unit normalizes the mean variance of the expanded and enhanced image data before inputting it into the image recognition module.

7. The fundus image recognition system based on deep learning according to claim 6, characterized in that: The data normalization unit adopts the global mean variance normalization technology to obtain the pixel value a' by normalizing each pixel value a of the image by its mean variance. The calculation formula is as follows: Among them, μ represents the mean value of image pixels, and the calculation formula is as follows: Where N represents the number of pixels, σ represents the standard deviation of the image pixels, and the calculation formula is:

8. The fundus image recognition system based on deep learning according to claim 5, characterized in that: The visualization auxiliary module uses the pre-trained model and input data to perform gradient reverse calculation, thereby drawing the image parts that have a positive impact on classification into a heat map.

9. The fundus image recognition system based on deep learning according to claim 8, characterized in that: The drawing process of the heat map is as follows: First, for the target category, calculate its convolution feature map A k Its gradient: Among them, c represents the target category, y c represents the output of the target category, and k represents the number of convolution layers; Then, these gradients are globally averaged pooled in the spatial dimension to obtain the weights of the feature map: Among them, i and j represent the feature map A respectively. k The row and column indices of the elements in, Z represents the feature space A k The size of , used to average the gradient and; Finally, these weights are linearly combined with the feature map, and then the Grad-CAM heat map is obtained through the ReLU activation function:

10. A fundus image recognition device based on deep learning, applied to a portable handheld terminal, includes a fundus image feature acquisition module, a fundus image recognition and judgment module, and a human-computer interaction module, characterized in that: The fundus image recognition and judgment module includes the image recognition module and visualization auxiliary module of the deep learning-based fundus image recognition system as described in claim 5, which classifies and performs visualization auxiliary processing on the fundus image obtained by the fundus image feature acquisition module, so that the cloud database stores and manages the fundus image data collected and recognized by the portable handheld terminal.

Images