Diabetic retinopathy detection device based on neural network model

Abstract

The present application discloses a diabetic retinopathy detection device based on a neural network model, which uses a neural network model to learn implicit key feature information about the lesion from the retinal image of a diabetic patient, and then uses the implicit key feature information to detect and identify diabetic retinopathy. In this way, reliable auxiliary diagnosis information is provided to doctors, while reducing the workload of doctors.

Description

Technical Field

[0001] The present application relates to the field of intelligent detection, and more specifically, to a diabetic retinopathy detection device based on a neural network model. Background Art

[0002] Diabetic retinopathy refers to damage to the retina of the eye caused by long-term diabetes. Diabetic patients often have high blood sugar levels, and long-term high blood sugar levels can damage the microvessels in the eye, leading to retinal vascular disease and retinal tissue damage, which in turn causes diabetic retinopathy.

[0003] The importance of testing for diabetic retinopathy lies in that timely testing can detect and treat the lesions to prevent the disease from worsening and leading to serious visual impairment or even blindness. Early detection of diabetic retinopathy allows for effective treatment measures, including controlling blood sugar levels, anti-VEGF treatment, photocoagulation treatment, etc., to slow down the progression of the disease and protect vision.

[0004] Traditional diabetic retinopathy detection usually relies on professional ophthalmologists to diagnose lesions through fundus examination and retinal imaging technology (such as fluorescent angiography, OCT scanning, etc.). This method has the following problems: 1. It is highly subjective and relies on the doctor's subjective judgment, which is easily affected by the doctor's experience and subjective factors; 2. It is time-consuming and requires professional doctors to spend a long time on examination and analysis. Therefore, a diabetic retinopathy detection device is expected. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a diabetic retinopathy detection device based on a neural network model, which uses the neural network model to learn implicit key feature information about the lesion from the retinal image of the diabetic patient object, and then uses this implicit key feature information to detect and identify diabetic retinopathy. In this way, reliable auxiliary diagnosis information is provided to doctors, while reducing the workload of doctors.

[0006] According to one aspect of the present application, a diabetic retinopathy detection device based on a neural network model is provided, comprising:

[0007] A retinal image acquisition module, used to acquire a retinal image of a diabetic patient subject;

[0008] A regularization module, used for performing data regularization on the retinal image to obtain a plurality of retinal sub-area images;

[0009] A global essential feature extraction module, used for extracting global image semantic essential features from the multiple retinal sub-region images to obtain a global retinal semantic feature map;

[0010] The lesion assessment decision module is used to determine the lesion assessment result based on the global retinal semantic feature map.

[0011] Compared with the prior art, the present application provides a diabetic retinopathy detection device based on a neural network model, which uses a neural network model to learn implicit key feature information about the lesion from the retinal image of a diabetic patient, and then uses this implicit key feature information to detect and identify diabetic retinopathy. In this way, reliable auxiliary diagnosis information is provided to doctors, while reducing the workload of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 is a block diagram of a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application;

[0014] Figure 2 is a system architecture diagram of a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application;

[0015] Figure 3 is a block diagram of the training phase of a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application;

[0016] Figure 4 is a block diagram of a regularization module in a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application;

[0017] Figure 5 It is a block diagram of a global essential feature extraction module in a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application. DETAILED DESCRIPTION

[0018] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0019] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0020] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0021] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] Traditional diabetic retinopathy detection usually relies on professional ophthalmologists to diagnose lesions through fundus examination and retinal imaging technology (such as fluorescent angiography, OCT scanning, etc.). This method has the following problems: 1. It is highly subjective and relies on the doctor's subjective judgment, which is easily affected by the doctor's experience and subjective factors; 2. It is time-consuming and requires professional doctors to spend a long time on examination and analysis. Therefore, a diabetic retinopathy detection device is expected.

[0024] In the technical solution of the present application, a diabetic retinopathy detection device based on a neural network model is proposed. Figure 1 4 is a block diagram of a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application. Figure 2 FIG. 1 is a system architecture diagram of a diabetic retinopathy detection device based on a neural network model according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to an embodiment of the present application, a diabetic retinopathy detection device 300 based on a neural network model includes: a retinal image acquisition module 310, used to acquire a retinal image of a diabetic patient; a regularization module 320, used to perform data regularization on the retinal image to obtain a plurality of retinal sub-region images; a global essential feature extraction module 330, used to perform global image semantic essential feature extraction on the plurality of retinal sub-region images to obtain a global retinal semantic feature map; and a lesion assessment judgment module 340, used to determine a lesion assessment result based on the global retinal semantic feature map.

[0025] In particular, the retinal image acquisition module 310 is used to acquire retinal images of diabetic patients. It should be understood that retinal images can be used for early screening of diabetic retinopathy, including diabetic retinopathy (such as microvascular lesions, hemorrhage, exudation, etc.) and diabetic macular edema. Early detection of lesions can be timely intervened and treated to reduce the risk of vision loss. Therefore, in the technical solution of the present application, first, a retinal image of a diabetic patient is acquired.

[0026] In particular, the regularization module 320 is used to regularize the retinal image to obtain multiple retinal sub-region images. In particular, in a specific example of the present application, Figure 4 As shown, the regularization module 320 includes: an image preprocessing unit 321, used for preprocessing the retinal image to obtain a preprocessed retinal image; a local area division unit 322, used for dividing the preprocessed retinal image into regions to obtain the multiple retinal sub-region images.

[0027] Specifically, the image preprocessing unit 321 is used to preprocess the retinal image to obtain a preprocessed retinal image. It should be understood that the retinal image of the diabetic patient object can reflect the distribution and state of the eye microvessels. Specifically, the retinal image of the diabetic patient object shows important information such as the morphology of retinal blood vessels and the regional state of the macular area. For example, when the condition of the diabetic patient worsens and affects the eye microvessels, lesions such as edema and exudation may appear in the macular area. Such lesion characteristics such as edema and exudation will appear in the retinal image. These important visual information can serve as an important basis for subsequent models to judge the degree of retinal lesions. However, in actual application scenarios, considering that noise may be introduced by factors such as ambient light and equipment during the acquisition of the retinal image, these noise information will affect subsequent image analysis and processing. In the technical solution of the present application, the noise information in the retinal image is reduced by performing image preprocessing on the retinal image to optimize the quality of the retinal image. In an embodiment of the present application, the retinal image preprocessing means may include image denoising, grayscale transformation, edge enhancement and geometric correction using filters and other methods.

[0028] Specifically, the local area division unit 322 is used to divide the preprocessed retinal image into regions to obtain the multiple retinal sub-region images. It should be understood that by dividing the preprocessed retinal image into regions to obtain multiple retinal sub-region images, each local retinal sub-region image can be analyzed separately, which is conducive to discovering microlesion information and changes.

[0029] It is worth mentioning that in other specific examples of the present application, the retinal image can also be data regularized in other ways to obtain multiple retinal sub-area images, for example: input the retinal image; preprocess the retinal image, including noise removal, contrast enhancement, brightness adjustment and other operations; use image segmentation techniques, such as threshold segmentation, edge detection, region growing and other methods to segment the retina from the background; determine multiple sub-areas of interest in the retina, such as the optic disc, retinal blood vessels, etc., based on the anatomical structure and characteristics of the retina; extract the corresponding multiple sub-area images from the original retinal image based on the determined sub-area positions; perform data regularization on each sub-area image; to obtain the multiple retinal sub-area images.

[0030] In particular, the global essential feature extraction module 330 is used to extract global image semantic essential features from the multiple retinal sub-region images to obtain a global retinal semantic feature map. In particular, in a specific example of the present application, Figure 5As shown, the global essential feature extraction module 330 includes: a local retinal feature extraction unit 331, which is used to use a deep learning network model to perform local retinal feature extraction on the multiple retinal sub-region images respectively to obtain multiple retinal sub-region feature maps; an interactive attention enhancement unit 332, which is used to pass the multiple retinal sub-region feature maps respectively through a triple interactive attention module to obtain a plurality of enhanced retinal sub-region feature maps; an essential feature distillation unit 333, which is used to input the multiple enhanced retinal sub-region feature maps into a feature distiller based on the essential feature extraction network to obtain the global retinal semantic feature map.

[0031] Specifically, the local retinal feature extraction unit 331 is used to perform local retinal feature extraction on the multiple retinal sub-region images respectively using a deep learning network model to obtain multiple retinal sub-region feature maps. Specifically, in a specific example of the present application, the multiple retinal sub-region images are respectively passed through a retinal image feature extractor based on a convolutional neural network model to obtain multiple retinal sub-region feature maps. Among them, the convolutional neural network (CNN) model can learn and extract high-level features in the image. Here, a retinal image feature extractor is constructed using a convolutional neural network model, and discriminative feature information in each of the retinal sub-region images can be gradually extracted through multi-layer convolution and pooling processing within the convolutional neural network model. For example, the convolutional neural network model can extract edge information from the retinal image to help identify the features of the edge contours of blood vessels, lesion areas, etc. in the retina. More specifically, the multiple retinal sub-region images are respectively passed through a retinal image feature extractor based on a convolutional neural network model to obtain multiple retinal sub-region feature maps, including: using each layer of the retinal image feature extractor based on the convolutional neural network model to respectively perform the following on the input data in the forward transfer of the layer: convolution processing on the input data to obtain a convolution feature map; pooling the convolution feature map based on a local feature matrix to obtain a pooled feature map; and, nonlinear activation on the pooled feature map to obtain an activation feature map; wherein the output of the last layer of the retinal image feature extractor based on the convolutional neural network model is the multiple retinal sub-region feature maps, and the input of the first layer of the retinal image feature extractor based on the convolutional neural network model is the multiple retinal sub-region images.

[0032] It is worth noting that the Convolutional Neural Network (CNN) is a deep learning model that is particularly suitable for processing data with grid structures, such as images and videos. The following is the general structure and step-by-step expansion of the Convolutional Neural Network model: Input layer: accepts input data, usually images, audio or text, etc.; Convolution layer: The convolution layer is one of the core components of CNN. It extracts local features from the input data by applying a series of filters (also called convolution kernels). The convolution operation multiplies the filter with the input data element by element and sums them to generate a feature map; Activation function: After the convolution layer, a nonlinear activation function such as ReLU is usually applied to introduce nonlinear features; Pooling layer: The pooling layer is used to reduce the spatial size of the feature map and retain the most important features. Common pooling operations are maximum pooling and average pooling; Fully connected layer: The fully connected layer connects the output of the pooling layer to one or more fully connected layers to map the features to the final output category or regression value. Each neuron in the fully connected layer is connected to all neurons in the previous layer; Output layer: The output layer selects appropriate activation functions according to different tasks, such as softmax function for multi-classification tasks and linear activation function for regression tasks; Loss function: Select appropriate loss functions according to different tasks, such as cross entropy loss function for classification tasks and mean square error loss function for regression tasks; Back propagation and optimization: Through the back propagation algorithm and gradient descent optimization algorithm, the gradient of the model parameters is calculated according to the loss function, and the parameters are updated to minimize the loss function.

[0033] Specifically, the interactive attention enhancement unit 332 is used to pass the multiple retinal sub-region feature maps through a triple interactive attention module to obtain a plurality of enhanced retinal sub-region feature maps. Considering that each of the retinal sub-region feature maps is extracted by convolution encoding of each of the retinal sub-region images through a retinal image feature extractor based on a convolutional neural network model, the implicit image association features of the local retina contained therein are limited by the size of the convolution kernel. In other words, the feature distribution of the retinal sub-region feature map can only characterize and represent the spatial local neighborhood association features in each of the retinal sub-region images, but lacks more global and multi-dimensional feature interaction information. Therefore, in the technical solution of the present application, the multiple retinal sub-region feature maps are further passed through a triple interactive attention module to obtain a plurality of enhanced retinal sub-region feature maps. Among them, the triple interactive attention module can strengthen the representation of the retinal sub-region feature map in the spatial dimension, and can also consider the interaction between the information in the spatial dimension and the channel dimension, so that by adjusting the importance weights of different positions in the retinal sub-region feature map, the network can better focus on the key areas and structures in the image, and make the retinal sub-region feature map more effective in information transmission and interaction in the spatial dimension and the channel dimension, thereby improving the representation ability and diversity of features. More specifically, the multiple retinal sub-region feature maps are respectively passed through the triple interactive attention module to obtain multiple enhanced retinal sub-region feature maps, including: processing the retinal sub-region feature map with the following spatial dimension enhancement formula to obtain a retinal sub-region feature map after spatial dimension enhancement; wherein, the spatial dimension enhancement formula is:

[0034] α1=δ(W 2 *σ(W 1 *X))

[0035]

[0036] Among them, α1 is the spatial information weight matrix, X is the retinal sub-region feature map, is the retinal sub-region feature map after the spatial dimension enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product; the retinal sub-area feature map is processed by the following first space and channel information interaction formula to obtain the retinal sub-area feature map after the first interactive information enhancement; wherein the first space and channel information interaction formula is:

[0037] α2=δ(W 2 *σ(W1 *X T1 ))

[0038]

[0039] Among them, α2 is the first space and channel information interaction weight matrix, X is the retinal sub-area feature map, is the retinal sub-area feature map after the first interactive information enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product, (·) T1 and Trans(·) represents the transposition processing of the feature map; the retinal sub-area feature map is processed by the following second space and channel information interaction formula to obtain the retinal sub-area feature map after the second interactive information enhancement; wherein the second space and channel information interaction formula is:

[0040] α3=δ(W 2 *σ(W 1 *X T2 ))

[0041]

[0042] Among them, α3 is the second space and channel information interaction weight matrix, X is the retinal sub-area feature map, is the retinal sub-area feature map after the second interactive information enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product, (·) T2 and Trans(·) represents the transposition processing of the feature map; the spatial dimension enhanced retinal sub-region feature map, the first interactive information enhanced retinal sub-region feature map and the second interactive information enhanced retinal sub-region feature map are processed by the following fusion formula to obtain the enhanced retinal sub-region feature map; wherein the fusion formula is:

[0043]

[0044] in, is the retinal sub-region feature map after the spatial dimension is enhanced, is the retinal sub-area feature map after the first interactive information is enhanced, is the retinal sub-area feature map after the second interactive information is enhanced, is the enhanced retinal sub-region feature map, and f(·,·) represents the cascade processing of the feature map.

[0045] Specifically, the essential feature distillation unit 333 is used to input the multiple enhanced retinal sub-region feature maps into a feature distiller based on an essential feature extraction network to obtain the global retinal semantic feature map. It should be understood that the multiple enhanced retinal sub-region feature maps are obtained by enhancing the feature extraction and feature interaction information of each retinal sub-region image obtained after the retinal image is divided into regions, and essentially represent the image visual implicit feature information of each local retinal sub-region image. Although these local image visual information can more accurately capture the detail information that is easily overlooked in the retinal image, there is a large amount of background information that is irrelevant to lesion detection and recognition, and this background information is not important for the subsequent target task of lesion detection. Therefore, in the technical solution of the present application, in order to reduce the impact of these redundant information or background information that is not important to the target task on the subsequent analysis, the multiple enhanced retinal sub-region feature maps are further input into a feature distiller based on an essential feature extraction network to capture the overall retinal core image semantic information expressed by each of the enhanced retinal sub-region feature maps, thereby obtaining a global retinal semantic feature map. Specifically, the intrinsic feature extraction network can automatically select and emphasize features that have a greater contribution to the task of lesion detection and identification, while weakening or ignoring those redundant features. This can reduce the impact of redundant information and background information in each enhanced retinal sub-region feature map, and retain more important and useful semantic information. More specifically, the multiple enhanced retinal sub-region feature maps are input into a feature distiller based on the intrinsic feature extraction network to obtain the global retinal semantic feature map, including: processing the multiple enhanced retinal sub-region feature maps with the following intrinsic feature distillation formula to obtain the global retinal semantic feature map; wherein the essential feature distillation formula is:

[0046]

[0047]

[0048] Among them, F i is the i-th enhanced retinal sub-region feature map among the multiple enhanced retinal sub-region feature maps, F j is the jth enhanced retinal sub-region feature map among the multiple enhanced retinal sub-region feature maps, ||·||2 represents the 2-norm of the feature map, M is the length of the multiple enhanced retinal sub-region feature maps -1, F k is the representation of the feature maps of the multiple enhanced retinal sub-regions, Di represents the characteristic difference coefficient, exp(·) represents the natural exponential function operation, N s represents the total number of semantic difference coefficients, P k is the global retinal semantic feature map.

[0049] It is worth mentioning that in other specific examples of the present application, the global image semantic essential features of the multiple retinal sub-area images can be extracted by other methods to obtain a global retinal semantic feature map, for example: inputting the multiple retinal sub-area images; inputting the loaded multiple retinal sub-area images into the convolutional neural network model, and extracting the semantic features of the image through forward propagation; fusing the semantic features extracted from the multiple retinal sub-area images, which can be done by simple average pooling, maximum pooling and other methods, or by more complex feature fusion techniques, such as attention mechanism; mapping the fused semantic features back to the image space to obtain the global retinal semantic feature map.

[0050] In particular, the lesion assessment decision module 340 is used to determine the lesion assessment result based on the global retinal semantic feature map. In particular, in a specific example of the present application, the global retinal semantic feature map is passed through a classifier-based lesion grade evaluator to obtain the lesion assessment result, and the lesion assessment result is used to represent the retinal lesion grade label. Specifically, in an embodiment of the present application, the retinal lesion grade label may include: non-lesion (Normal), mild lesion (Mild), moderate lesion (Moderate), severe lesion (Severe) and proliferative lesion (Proliferative) and other lesion degrees. It is worth mentioning that the retinal lesion grade label can be adjusted and modified according to actual conditions. In this way, the lesion grade assessment result generated intelligently can help doctors make accurate diagnoses and reduce the workload of doctors.

[0051] It is worth noting that a classifier refers to a machine learning model or algorithm that is used to classify input data into different categories or labels. Classifiers are part of supervised learning and perform classification tasks by learning the mapping relationship from input data to output categories.

[0052] It should be understood that before using the above-mentioned neural network model for inference, it is necessary to train the retinal image feature extractor based on the convolutional neural network model, the triple interactive attention module, the feature distiller based on the essential feature extraction network, and the lesion level assessor based on the classifier. That is, according to the diabetic retinopathy detection device 300 based on the neural network model of the present application, it also includes a training stage 400 for training the retinal image feature extractor based on the convolutional neural network model, the triple interactive attention module, the feature distiller based on the essential feature extraction network, and the lesion level assessor based on the classifier.

[0053] Figure 3 FIG. 1 is a block diagram of the training phase of the diabetic retinopathy detection device based on the neural network model according to an embodiment of the present application. Figure 3 As shown, according to the embodiment of the present application, the diabetic retinopathy detection device 300 based on the neural network model includes: a training stage 400, including: a training data acquisition module 410, used to acquire training data, wherein the training data includes a training retinal image of a diabetic patient object and a true value of a retinopathy grade label; a training image preprocessing module 420, used to preprocess the training retinal image to obtain a training preprocessed retinal image; a training area division module 430, used to divide the training preprocessed retinal image into regions to obtain a plurality of training retinal sub-region images; a training retinal sub-region feature extraction module 440, used to pass the plurality of training retinal sub-region images through the retinal image feature extractor based on the convolutional neural network model to obtain a plurality of training retinal sub-region feature maps; a training retinal sub-region feature enhancement module 450, used to pass the plurality of training retinal sub-region feature maps through the triple interactive attention A module is used to obtain multiple training enhanced retinal sub-area feature maps; a training global retinal semantic feature extraction module 460 is used to input the multiple training enhanced retinal sub-area feature maps into the feature distiller based on the essential feature extraction network to obtain a training global retinal semantic feature map; a feature map expansion module 470 is used to perform feature map expansion on the training global retinal semantic feature map to obtain a training global retinal semantic feature vector; a classification loss module 480 is used to pass the training global retinal semantic feature vector through the classifier-based lesion level evaluator to obtain a classification loss function value; a training module is used to train the retinal image feature extractor based on the convolutional neural network model, the triple interactive attention module, the feature distiller based on the essential feature extraction network and the classifier-based lesion level evaluator with the classification loss function value, wherein, in each round of iteration of the training, the training global retinal semantic feature vector is iteratively optimized.

[0054] In particular, in the above technical solution, the multiple training retinal sub-region feature maps respectively express the image semantic features of the pre-processed retinal image in the local image semantic space domain determined by region division in the global image semantic space domain. Therefore, after the multiple training retinal sub-region feature maps are respectively passed through the triple interactive attention module, and the multiple training enhanced retinal sub-region feature maps are input into the feature distiller based on the essential feature extraction network, the image semantic features of the local dimension enhancement based on the spatial distribution-channel distribution interaction can be obtained, and the image semantic feature representation associated with the global image semantic space domain can be extracted based on the image semantic feature essential pattern association between the local image semantic space domains. In this way, considering the multi-dimensional image semantic feature representation based on the training interactive dimension enhanced image semantic features in the local image semantic space domain and the essential image semantic feature association pattern between the local image semantic space domains under the training global image semantic space domain contained in the training global retinal semantic feature map, it is expected to improve the convergence effect of the training global retinal semantic feature map through the class regression iteration of the classifier. Therefore, the applicant of the present application iteratively optimizes the training global retinal semantic feature vector each time the training global retinal semantic feature vector after the training global retinal semantic feature map is expanded passes through the classifier for classification regression iteration, which is expressed as:

[0055]

[0056] where v i is the i-th eigenvalue of the training global retinal semantic feature vector, φ(v i ) represents a probabilistic function of the feature value, for example, a probabilistic function based on mapping the maximum feature value to the interval [0,1], and p is the class probability value obtained by the classifier, and exp[·] represents an exponential operation. Here, the feature value v of the global retinal semantic feature vector is trained by taking the Bernoulli probability likelihood distribution as the authoritative meta-entity i Based on the high-level class decision of Bernoulli probability distribution (p, 1-p), and introducing the probability density space mapping allocation relative to the whole feature set, the eigenvalue v of the training global retinal semantic feature vector is determined. i The key insight with respect to the class probability mapping is to implement the eigenvalues ​​v of the training global retinal semantic feature vector i Based on the correctness of the matching function of class probability distribution (p, 1-p) and class convergence, the superiority of convergence performance can be used to promote sample iterative convergence and improve the classification training speed.

[0057] As described above, the diabetic retinopathy detection device 300 based on a neural network model according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a diabetic retinopathy detection algorithm based on a neural network model. In a possible implementation, the diabetic retinopathy detection device 300 based on a neural network model according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the diabetic retinopathy detection device 300 based on a neural network model can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the diabetic retinopathy detection device 300 based on a neural network model can also be one of the many hardware modules of the wireless terminal.

[0058] Alternatively, in another example, the diabetic retinopathy detection device 300 based on the neural network model and the wireless terminal may also be separate devices, and the diabetic retinopathy detection device 300 based on the neural network model may be connected to the wireless terminal via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0059] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A diabetic retinopathy detection device based on a neural network model, characterized in that: include: A retinal image acquisition module, used to acquire a retinal image of a diabetic patient subject; A regularization module, used for performing data regularization on the retinal image to obtain a plurality of retinal sub-area images; A global essential feature extraction module, used for extracting global image semantic essential features from the multiple retinal sub-region images to obtain a global retinal semantic feature map; A lesion assessment decision module, used to determine a lesion assessment result based on the global retinal semantic feature map; The regularization module comprises: An image preprocessing unit, used for preprocessing the retinal image to obtain a preprocessed retinal image; A local area division unit, used for performing area division on the preprocessed retinal image to obtain the multiple retinal sub-area images; The global essential feature extraction module comprises: A local retinal feature extraction unit, configured to extract local retinal features from the multiple retinal sub-region images respectively using a deep learning network model to obtain multiple retinal sub-region feature maps; An interactive attention enhancement unit, configured to pass the plurality of retinal sub-region feature maps through a triple interactive attention module to obtain a plurality of enhanced retinal sub-region feature maps; An essential feature distillation unit, used for inputting the plurality of enhanced retinal sub-region feature maps into a feature distiller based on an essential feature extraction network to obtain the global retinal semantic feature map; The local retinal feature extraction unit is used to: Passing the multiple retinal sub-region images through a retinal image feature extractor based on a convolutional neural network model to obtain the multiple retinal sub-region feature maps; The interactive attention reinforcement unit is used to: The retinal sub-region feature map is processed using the following spatial dimension enhancement formula to obtain a retinal sub-region feature map after spatial dimension enhancement; wherein the spatial dimension enhancement formula is: α1=δ(W 2 *σ(W 1 *X) Among them, α1 is the spatial information weight matrix, X is the retinal sub-region feature map, is the retinal sub-region feature map after the spatial dimension enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product; The retinal sub-region feature map is processed using the following first spatial and channel information interaction formula to obtain a retinal sub-region feature map after first interactive information enhancement; wherein the first spatial and channel information interaction formula is: α2=δ(W 2 *σ(W 1 *X T1 ) Among them, α2 is the first space and channel information interaction weight matrix, X is the retinal sub-area feature map, is the retinal sub-area feature map after the first interactive information enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product, (·) T1 and Trans(·) represents the transposition of the feature map; The retinal sub-region feature map is processed by the following second space and channel information interaction formula to obtain the retinal sub-region feature map after second interactive information enhancement; wherein the second space and channel information interaction formula is: α3=δ(W 2 *σ(W 1 *X T2 ) Among them, α3 is the second space and channel information interaction weight matrix, X is the retinal sub-area feature map, is the retinal sub-area feature map after the second interactive information enhancement, W 1 represents a 1×1 convolution, W 2 represents a 7×7 convolution, * represents a convolution operation, σ(·) represents a ReLU function, δ(·) represents a Sigmoid function, represents the Hadamard product, (·) T2 and Trans(·) represents the transposition of the feature map; The spatial dimension enhanced retinal sub-region feature map, the first interactive information enhanced retinal sub-region feature map, and the second interactive information enhanced retinal sub-region feature map are processed using the following fusion formula to obtain the enhanced retinal sub-region feature map; wherein the fusion formula is: in, is the retinal sub-region feature map after the spatial dimension is enhanced, is the retinal sub-area feature map after the first interactive information is enhanced, is the retinal sub-area feature map after the second interactive information is enhanced, is the enhanced retinal sub-region feature map, and f(·,·) represents the cascade processing of the feature map.

2. The diabetic retinopathy detection device based on the neural network model according to claim 1, characterized in that: The essential characteristic distillation unit is used for: The multiple enhanced retinal sub-region feature maps are processed using the following essential feature distillation formula to obtain the global retinal semantic feature map; wherein the essential feature distillation formula is: Among them, F i is the i-th enhanced retinal sub-region feature map among the multiple enhanced retinal sub-region feature maps, F j is the jth enhanced retinal sub-region feature map among the multiple enhanced retinal sub-region feature maps, ‖·‖2 represents the 2-norm of the feature map, M is the length of the multiple enhanced retinal sub-region feature maps -1, F k is the representation of the feature maps of the multiple enhanced retinal sub-regions, D i represents the characteristic difference coefficient, exp(·) represents the natural exponential function operation, N s represents the total number of semantic difference coefficients, P k is the global retinal semantic feature map.

3. The diabetic retinopathy detection device based on the neural network model according to claim 2, characterized in that: The lesion assessment and judgment module is used to: The global retinal semantic feature map is passed through a classifier-based lesion grade evaluator to obtain the lesion assessment result, and the lesion assessment result is used to represent a retinal lesion grade label.

4. The diabetic retinopathy detection device based on the neural network model according to claim 3 is characterized in that: It also includes a training module for training the retinal image feature extractor based on the convolutional neural network model, the triple interaction attention module, the feature distiller based on the essential feature extraction network, and the classifier-based lesion level assessor.

5. The diabetic retinopathy detection device based on the neural network model according to claim 4, characterized in that: The training module comprises: A training data acquisition module, used to acquire training data, wherein the training data includes training retinal images of diabetic patients and true values ​​of retinopathy grade labels; A training image preprocessing module, used for preprocessing the training retinal image to obtain a training preprocessed retinal image; A training region division module, used for dividing the training preprocessed retinal image into regions to obtain a plurality of training retinal sub-region images; A training retinal sub-region feature extraction module, used for respectively passing the plurality of training retinal sub-region images through the retinal image feature extractor based on the convolutional neural network model to obtain a plurality of training retinal sub-region feature maps; A training retinal sub-region feature enhancement module, used for respectively passing the plurality of training retinal sub-region feature maps through the triple interactive attention module to obtain a plurality of training enhanced retinal sub-region feature maps; A training global retinal semantic feature extraction module, used for inputting the plurality of training enhanced retinal sub-region feature maps into the feature distiller based on the essential feature extraction network to obtain a training global retinal semantic feature map; A feature map expansion module, used for performing feature map expansion on the training global retinal semantic feature map to obtain a training global retinal semantic feature vector; A classification loss module, used for passing the training global retinal semantic feature vector through the classifier-based lesion grade evaluator to obtain a classification loss function value; A training module is used to train the retinal image feature extractor based on the convolutional neural network model, the triple interactive attention module, the feature distiller based on the essential feature extraction network and the classifier-based lesion grade evaluator with the classification loss function value, wherein in each round of iteration of the training, the training global retinal semantic feature vector is iteratively optimized.

Images