Diabetic retinopathy detection model training method, training device, detection method, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to a diabetic retinopathy detection system and device and a storage medium, and a diabetic retinopathy detection model training method comprises the steps that a training data set is acquired, and the training data set comprises a plurality of fundus sample images and descriptive texts corresponding to the fundus sample images; constructing a network structure of the diabetic retinopathy detection model; performing iterative training on the network of the diabetic retinopathy detection model by using the training data set until a training stop condition is reached, and obtaining a final diabetic retinopathy detection model; in each training, the diabetic retinopathy detection model extracts the image features of the fundus sample image and the text features of the descriptive text, calculates the contrast loss of the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model according to the contrast loss. The accuracy of a diabetic retinopathy detection result can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method and device for training a diabetic retinopathy detection model, a detection method, a device, and a storage medium. Background Art

[0002] Diabetic retinopathy is a microvascular complication caused by diabetes and is a common disease among diabetic patients. Currently, about one-third of diabetic patients suffer from eye diseases. Diabetic retinopathy is divided into two categories: non-proliferative and proliferative. Non-proliferative diabetic retinopathy includes mild, moderate, and severe, while proliferative diabetic retinopathy can be divided into high-risk and non-high-risk. Other characteristics of non-proliferative diabetic retinopathy include soft or hard exudates and retinal microvascular abnormalities. The condition of non-proliferative diabetic retinopathy is relatively stable and is easier to treat, but if not treated and intervened in a timely manner, it may develop into proliferative diabetic retinopathy, and the condition may severely affect the patient's vision and even lead to complete blindness.

[0003] Therefore, in order to prevent the deterioration of diabetic retinopathy and cause irreversible damage to the patient's vision, it is crucial to diagnose and treat diabetic retinopathy early. Currently, the diagnosis of diabetic retinopathy relies on the comprehensive analysis of fundus images and fundus fluorescein angiography by experienced doctors to determine the condition, or uses computers to assist doctors in diagnosing the patient's retinal lesion condition.

[0004] In the study of computer-aided diagnosis of diabetic retinopathy, we face a series of challenges, including insufficient feature extraction and representation, data imbalance and quality problems, and insufficient model robustness. Existing diabetic retinopathy detection models may have limitations in extracting and representing key features from different modalities, directly affecting the diagnostic performance. In addition, multi-modal datasets are often affected by data imbalance and inconsistent quality, which may lead to model bias and overfitting. Summary of the Invention

[0005] In order to improve the accuracy of diabetic retinopathy detection results, a method, system, device, and storage medium for training a diabetic retinopathy detection model are provided.

[0006] In order to achieve the above object of the present invention, the present invention provides a method for training a diabetic retinopathy detection model.

[0007] A method for training a diabetic retinopathy detection model provided by the present invention includes:

[0008] Obtain a training data set, where the training data set includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images;

[0009] Construct the network structure of the diabetic retinopathy detection model;

[0010] Use the training data set to iteratively train the network of the diabetic retinopathy detection model until the training stop condition is reached, and obtain the final diabetic retinopathy detection model;

[0011] In each training, the diabetic retinopathy detection model extracts the image features of the fundus sample image and the text features of the descriptive text, calculates the contrast loss between the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model according to the contrast loss.

[0012] Optionally, the steps for the diabetic retinopathy detection model to extract the image features of the fundus sample image and the text features of the descriptive text are as follows:

[0013] Extract the initial image features of the fundus sample image;

[0014] Project the initial image features into a preset feature space through a first projection layer to obtain image features;

[0015] Extract the initial text features of the descriptive text;

[0016] Project the initial text features into a preset feature space through a second projection layer to obtain text feature vectors.

[0017] Optionally, after projecting the initial image features into a preset feature space through the first projection layer to obtain image features, the method further includes:

[0018] Process the image features through a multi-spectral channel attention mechanism to obtain optimized image features.

[0019] Optionally, the steps of processing the image features through the multi-spectral channel attention mechanism include:

[0020] Decompose the image features into several image sub-features;

[0021] Perform a two-dimensional discrete cosine transform on the spatial dimension of the image sub-features to obtain the frequency-domain sub-features corresponding to the image sub-features;

[0022] Learn the weight relationship between different frequency-domain sub-features through a fully connected layer to obtain weighted frequency-domain sub-features;

[0023] Generate channel attention weights based on the weighted frequency-domain sub-features, and multiply each channel of the image features by the channel attention weights one by one to obtain optimized image features.

[0024] Optionally, the transformation formula for converting the image sub-features from the time domain to the frequency domain through two-dimensional discrete cosine transform is as follows:

[0025]

[0026] where f h，w represents the feature value of the point at coordinates (h, w) in the two-dimensional frequency domain, h represents the vertical frequency index in the two-dimensional frequency domain, w represents the horizontal frequency index in the two-dimensional frequency domain, x (i，j) represents the pixel value of the point corresponding to the i-th row and j-th column of the input image sub-feature in the spatial domain, i represents the vertical index of the input image sub-feature in the spatial domain, j represents the horizontal index of the input image sub-feature in the spatial domain; H represents the height of the input image sub-feature in the spatial domain, and W represents the width of the input image sub-feature in the spatial domain;

[0027] The frequency domain sub-feature Freq corresponding to the image sub-feature of the n-th channel n is:

[0028]

[0029] represents the time domain feature value of the input image sub-feature at the position of the h-th row and w-th column in the n-th channel, and ":, h, w" represents selecting the feature values of the h-th row and w-th column in all channels of the image feature;

[0030] u represents the frequency index in the vertical direction in the two-dimensional frequency domain, v represents the frequency index in the horizontal direction in the two-dimensional frequency domain; N represents the total number of channels of the image feature, n represents the channel index, n is an integer, and N is a positive integer.

[0031] Optionally, the loss function of the diabetic retinopathy detection model is

[0032] where m represents the total number of fundus sample images in the training dataset, a represents the index of the fundus sample image in the training dataset, α is a scaling factor, and λ is a margin parameter; S ab represents the similarity score between the fundus sample image a and the fundus sample image b in the training dataset; P a represents the probability that the diabetic retinopathy classification result of the fundus sample image a belongs to the correct class.

[0033] To solve the above problems, the present invention also provides a diabetic retinopathy detection model training device, including:

[0034] An acquisition module for acquiring a training dataset, where the training dataset includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images;

[0035] A building module for building the network structure of a diabetic retinopathy detection model;

[0036] A training module for training the network of the diabetic retinopathy detection model using a training dataset. In each training, the diabetic retinopathy detection model extracts the image features of fundus sample images and the text features of descriptive texts, calculates the contrast loss between the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model according to the contrast loss to obtain the final diabetic retinopathy detection model.

[0037] To solve the above problems, the present invention also provides a method for detecting diabetic retinopathy, the method comprising:

[0038] Obtain a fundus image to be detected;

[0039] Input the fundus image into the diabetic retinopathy detection model trained by the above-mentioned diabetic retinopathy detection model training method. The diabetic retinopathy detection model analyzes and processes the fundus image and outputs the classification result of diabetic retinopathy of the fundus image.

[0040] To solve the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0041] At least one processor; and,

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned diabetic retinopathy detection model training method or the diabetic retinopathy detection method.

[0044] To solve the above problems, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores at least one computer program. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned diabetic retinopathy detection model training method or the diabetic retinopathy detection method.

[0045] In summary, the present application includes the following beneficial technical effects:

[0046] Through the contrast training of diabetic fundus sample images and descriptive texts, the pre-training of the diabetic retinopathy detection model is completed; by using the alignment between images and texts, the diabetic retinopathy detection model is taught to distinguish relevant and irrelevant features. This process enables the diabetic retinopathy detection model to capture the subtle correlation between the visual patterns in the fundus sample images and the semantic information in the descriptive texts, thereby understanding the data more deeply, enabling the diabetic retinopathy detection model to generalize more effectively to diverse and unlabeled data sets, and improving the accuracy of diabetic retinopathy detection results;

[0047] Optimize the image features using the multi-spectral channel attention mechanism, and enhance the ability of the diabetic retinopathy detection model to capture complex image details by processing the frequency-domain sub-features corresponding to the image sub-features, thereby further improving the accuracy of diabetic retinopathy detection results. Brief Description of the Drawings

[0048] Figure 1 It is a schematic flowchart of the training method of the diabetic retinopathy detection model provided by an embodiment of the present invention;

[0049] Figure 2 It is an explanatory diagram of an embodiment of the training method of the diabetic retinopathy detection model provided by an embodiment of the present invention;

[0050] Figure 3 It is a system architecture diagram of the image editor provided by an embodiment of the present invention;

[0051] Figure 4 It is a specific architecture diagram of each module in the image editor provided by an embodiment of the present invention;

[0052] Figure 5 It is a schematic diagram of the steps for processing image features through the multi-spectral channel attention mechanism provided by an embodiment of the present invention;

[0053] Figure 6 It is a flowchart of processing image features using the multi-spectral channel attention mechanism provided by an embodiment of the present invention;

[0054] Figure 7 It is a module schematic diagram of the image classification module in the inference stage provided by an embodiment of the present invention;

[0055] Figure 8 It is a schematic flowchart of the diabetic retinopathy detection method provided by an embodiment of the present invention;

[0056] Figure 9 It is a schematic structural diagram of the electronic device for implementing the above provided by an embodiment of the present invention.

[0057] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0058] The realization, functional features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with embodiments. Detailed implementation manners

[0059] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0060] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention.

[0061] In the description of the present invention, unless otherwise specified and defined, it should be noted that the terms "installation", "connection" and "connection" should be understood in a broad sense. For example, it may be a mechanical connection or an electrical connection, or it may be the communication inside two elements. It may be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] Figure 1 It is a schematic flow chart of a method for training a diabetic retinopathy detection model provided in an embodiment of the present invention. In this embodiment, the method for training a diabetic retinopathy detection model includes:

[0063] S1. Obtain a training data set.

[0064] The training data set includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images. Specifically, the content of the descriptive text includes literal descriptions of details such as lesion areas, blood vessel textures or color distributions in the fundus sample images.

[0065] Diabetic retinopathy includes two major categories: non - proliferative diabetic retinopathy and proliferative diabetic retinopathy. Among them, non - proliferative diabetic retinopathy includes three sub - categories: mild, moderate, and severe, and proliferative diabetic retinopathy is divided into two sub - categories: high - risk and non - high - risk. During training, fundus sample images and descriptive texts are input into the diabetic retinopathy detection model. The diabetic retinopathy detection model determines which sub - category in which major category of diabetic retinopathy the lesion corresponding to the fundus sample image belongs to by parsing the fundus sample image and descriptive text, and completes the classification of the diabetic retinopathy corresponding to the fundus sample image.

[0066] After the diabetic retinopathy detection model outputs the classification result of the fundus sample image, the classification result is compared with the descriptive text, and the network parameters of the diabetic retinopathy detection model are adjusted according to the comparison result, which can improve the training efficiency of the diabetic retinopathy detection model and the accuracy of the prediction result of the diabetic retinopathy type of the fundus image by the diabetic retinopathy detection model.

[0067] In this embodiment, the fundus images of the training dataset can use the ODIR - 5K dataset. ODIR - 5K (Ocular Disease Intelligent Recognition) is a large - scale eye disease recognition dataset, aiming to promote the research on the automatic recognition and classification of eye diseases. This dataset includes 5,000 fundus photos from different patients, covering a variety of ages, genders, and ethnicities. Each photo is labeled by professional ophthalmologists, including eight major categories of eye diseases such as normal, diabetic retinopathy, glaucoma, cataract, etc. The training dataset can also select other eye disease recognition datasets containing diabetic retinopathy images, and this embodiment does not make restrictions.

[0068] S2. Construct the network structure of the diabetic retinopathy detection model.

[0069] The network structure of the diabetic retinopathy detection model includes an image feature extraction module, a text feature extraction module, and a classification module. The image feature extraction module is used to extract features from the input image to obtain image features, the text feature extraction module is used to extract features from the input text to obtain text features, and the classification module is used to determine the classification prediction result of the diabetic retinopathy corresponding to the input image according to the image features and text features.

[0070] S3. Use the training dataset to iteratively train the network of the diabetic retinopathy detection model until the training stop condition is reached, and obtain the final diabetic retinopathy detection model;

[0071] In each training, the diabetic retinopathy detection model extracts the image features of fundus sample images and the text features of the descriptive text, calculates the contrast loss between the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model according to the contrast loss.

[0072] The training stop condition is that when the number of model training times reaches the maximum number or the loss function value of the diabetic retinopathy detection model fluctuates within a certain range and no longer decreases significantly, it can be considered that the model has converged and the training can end.

[0073] In this embodiment, the difference between the predicted probability and the actual binary label is evaluated by the binomial loss. The diabetic retinopathy detection model adopts the binomial loss function, which effectively improves the accuracy of feature matching, and significantly improves the efficiency of processing image and text data and the accuracy of feature extraction, thus solving the limitations of traditional methods in this field.

[0074] The loss function of the diabetic retinopathy detection model is

[0075]

[0076] where m represents the total number of fundus sample images in the training dataset, a represents the index of the fundus sample image in the training dataset, α is a scaling factor, which is set to 2.0 in this application, λ is a margin parameter, which is set to 0.5 in this application; S ab represents the similarity score between the fundus sample image a and the fundus sample image b in the training dataset; P a represents the probability that the diabetic retinopathy classification result of the fundus sample image a belongs to the correct class.

[0077] Specifically, the calculation formula for the similarity score between the fundus sample image a and the fundus sample image b in the training dataset is

[0078]

[0079] where v a represents the feature vector corresponding to the fundus sample image a, w b represents the feature vector corresponding to the fundus sample image b, ||v a || represents the norm of the feature vector v a || represents the norm of the feature vector w b || represents the norm of the feature vector w b ;

[0080] The probability P that the diabetic retinopathy classification result of the fundus sample image a belongs to the correct class a is obtained by applying the softmax function to the vector representation, P aIt is calculated by summing the indicator function 1[y a = y b for all N samples, excluding the current fundus sample image a, and then raising the result to the K-th power. K is a parameter that adjusts the weight of each term. P a The calculation formula of P

[0081]

[0082] where y a represents the true class of the fundus sample image a, and y b represents the true class of the fundus sample image b. [y a = y b is an indicator function, which takes the value of 1 when y a = y b and 0 otherwise.

[0083] In the preferred embodiment of this implementation, before inputting the fundus sample image and descriptive text into the diabetic retinopathy detection model, it is necessary to preprocess the data in the training dataset. The data preprocessing includes image preprocessing and text preprocessing. The preprocessing of image data includes image resizing and format conversion. Image resizing is to resize all the fundus sample images in the training dataset to a size of 224×224 pixels to meet the size requirements of the model input. Format conversion is used to convert the image data into a tensor format that the diabetic retinopathy detection model can process. Simply put, it is to convert the image data into the tensor format (C×H×W) used by PyTorch, and then map the pixel values from the integer interval [0, 255] to the floating-point interval [0, 1] to unify the image format and prepare for subsequent model calculations.

[0084] The goal of the text preprocessing stage is to convert natural language text (i.e., descriptive text) into a numerical representation form that is more suitable for deep learning models. Specifically, we first tokenize the descriptive text, divide it into several word sequences, and then use a pre-built vocabulary index dictionary to map each word to a specific index value. These index values are then passed to an embedding layer, which converts each index value into a vector representation in a high-dimensional space. By calculating the average value of these vectors, a fixed-dimensional vector representing the entire descriptive text is obtained.

[0085] Through the preprocessing steps, we can provide consistent and standardized image and text data for the diabetic retinopathy detection model, enhance the model's ability to process multi-modal data, and lay a solid foundation for the subsequent feature extraction and model training stages.

[0086] After completing data preprocessing, we use a contrastive architecture to accurately and efficiently process and understand fundus sample images and descriptive texts associated with the fundus sample images.

[0087] Reference Figure 2 ,The steps of extracting the image features of fundus sample images and the text features of descriptive text by the diabetic retinopathy detection model are:

[0088] S31, extracting initial image features of fundus sample images.

[0089] Specifically, the image editor receives the preprocessed fundus sample image f θ Afterwards, the fundus sample image f θ Perform feature extraction to obtain the initial image features using h i .

[0090] Reference Figure 3 , the image editor includes a base block, four stages, a patch embedding block, a MobileOne block and an attention block; among them, the four stages include the first stage ( Figure 3 stage1 in the second stage ( Figure 3 stage2 in the third stage ( Figure 3 stage3) and the fourth stage ( Figure 3 stage4 in the .

[0091] Figure 4 This is a detailed architecture diagram of each module in the image editor, where: Figure 4 (a) represents the first stage / second stage / third stage architecture. The architectures of the first stage, second stage, and third stage are the same. Figure 4 (b) in the figure represents the architecture of the batch normalization layer. Figure 4 (c) in the figure represents the architecture of the base block. Figure 4 (d) in the figure represents the fourth stage architecture. Figure 4 (e) in the figure represents the architecture of the convolutional feedforward network. Figure 4 (f) in the figure represents the architecture of the MobileOne block. The MobileOne block reduces the amount of computation by merging multiple convolutional blocks. The MobileOne block can reduce the computational complexity of the image editor while maintaining efficient feature extraction capabilities.

[0092] Reference Figure 4 ,In the first, second and third stages, the input image first passes through a batch normalization layer, such as Figure 4As shown in (b), the batch normalization layer stabilizes the learning process by re-centering and rescaling the input layer. Next, the normalized input is sent to a deep convolutional layer with a kernel size of 3×3 and a stride of 2. This layer applies a single filter to each input channel, achieving a lightweight filtering process and capturing spatial features. The output of the deep convolutional layer then enters a convolutional feed-forward network (ConvFFN block), which starts with another deep convolutional layer with a larger kernel of 7×7 and a stride of 2 to further process higher-level spatial features while reducing the feature dimension. After this, another round of batch normalization is applied to further normalize the newly transformed features; after that, the features are processed by a 1×1 convolutional layer to merge features between channels; then, an activation function (such as ReLU, the full name of which is Rectified Linear Unit) is applied to introduce nonlinear characteristics of the model, allowing it to learn more complex patterns. After that, another 1×1 convolutional layer is applied to expand the dimension or adjust the number of channels. Finally, the output of the second 1×1 convolutional layer is combined with the initial input of the ConvFFN block through a residual connection, represented by the addition operation in the figure.

[0093] In the fourth stage, we added conditional position encoding (CPE), which can be adapted to understand the location of pixels or regions within the image. The MobileOne block further reduces the computational complexity while maintaining efficient feature extraction capabilities. In the preferred implementation of this embodiment, a multi-head self-attention mechanism is used when fusing the extracted multiple features to enhance the model's perception of spatial relationships within the image. The multi-head self-attention mechanism improves feature representation by learning the dependencies between input features. The self-attention module we designed allows different attention heads to be applied to different feature subsets, thereby improving the accuracy and efficiency of feature extraction.

[0094] S32. Projecting the initial image features to a preset feature space through a first projection layer to obtain image features.

[0095] The preset feature space is a shared space used to align image and text features. Specifically, the initial image feature h i Through the projection layer g θ and the projection layer q θ The function of the projection layer is to apply a certain mapping to the high-dimensional feature vector to obtain a vector represented in a new coordinate system or new dimension, and to normalize the vector, that is, to linearly map the data to the interval [0, 1]. This process can rearrange the high-dimensional features into a feature space that is more suitable for subsequent tasks, that is, to convert the initial image features h i Projected to a new feature space v i , the new feature space v i Used to match image features with text features.

[0096] S33, extracting initial text features of the descriptive text.

[0097] Reference Figure 2 , the text encoder is connected to the preprocessed descriptive text f' i Perform feature extraction to obtain the initial text feature h i ′.

[0098] S34. Projecting the initial text features to a feature space corresponding to the image features through a second projection layer to obtain a text feature vector.

[0099] Reference Figure 2 , initial text feature h i ′Through the projection layer g θ ′, the initial text feature h i ′ is projected into the new feature space z that is the same as the image i ′. These vectors are used to calculate a loss function to measure the similarity between image and text representations. In addition, we use a stop gradient method to prevent gradients from backpropagating to the text branch, reducing the possibility of representation learning falling into a collapsed solution.

[0100] In a preferred implementation of this embodiment, after the initial image features are projected into a preset feature space through a first projection layer to obtain image features, the image features are processed through a multi-spectral channel attention mechanism to obtain optimized image features;

[0101] The network parameters of the diabetic retinopathy detection model are determined based on text features and optimized image features.

[0102] Reference Figure 5 , the multispectral channel attention mechanism is a module that can describe multispectral information. The steps of processing image features through the multispectral channel attention mechanism include:

[0103] S41. Decompose the image feature into several image sub-features.

[0104] Reference Figure 6 , the size of the input image feature map is H×W×C, where H and W are the spatial dimensions (i.e., height and width), and C is the number of channels; each channel of the input feature map is divided into several sub-features C′, and each channel is divided into smaller spectra or regions to form a new representation structure; the output shape is H×W×C′×n, where n is the number of sub-features after segmentation.

[0105] Specifically, the input image feature is recorded as x, and the input x is manually divided into N parts before entering the multi-spectral channel attention module. The divided input x n Use [x 1 , x2 , x 3 , ..., x (N-1) is represented as shown in formula (1).

[0106] The expression of formula (1) is as follows:

[0107]

[0108] Among them, n represents the channel index of the image feature, N represents the total number of channels of the image feature, n is an integer, and N is a positive integer; represents all spatial positions (usually corresponding to height and width) of the n-th channel (feature dimension) in the output feature tensor. Specifically, all spatial positions of the n-th channel (feature dimension) refer to each point in the n-th channel of the output feature tensor; X ：，n，：，： represents the spatial position information of the input image sub-feature. The symbol ":" represents the dimensional data of the input image sub-feature; ":,:" represents the spatial dimension (such as the height H and width W of the feature map).

[0109] S42. Perform a two-dimensional discrete cosine transform on the spatial dimension of the image sub-feature to obtain the frequency-domain sub-feature corresponding to the image sub-feature.

[0110] Specifically, perform a two-dimensional discrete cosine transform (2D-DCT, 2D-Discrete Cosine Transform) on the spatial dimension of each sub-feature C'; DCT maps the input spatial feature to the frequency domain, captures the low-frequency and high-frequency information in the spatial structure, and generates spectral features.

[0111] The output result is a set of frequency-domain sub-features Freq 0 , Freq 1 , ..., Freq n , and each spectral sub-feature corresponds to a specific frequency component.

[0112] The formula for the two-dimensional discrete cosine transform (discrete cosine transform, DCT) refers to formula (2), and the expression of formula (2) is as follows:

[0113]

[0114] h ∈ [0, H - 1], w ∈ [0, W - 1] Among them, f h，w represents the eigenvalue of the point at coordinates (h, w) in the two-dimensional frequency domain. h represents the vertical frequency index in the two-dimensional frequency domain (i.e., the index representing the height direction), and w represents the horizontal frequency index in the two-dimensional frequency domain (i.e., the index representing the width direction); x (i，j)Denote the pixel value of the point corresponding to the \(i\)-th row and \(j\)-th column of the input image sub-feature in the spatial domain. \(i\) represents the vertical index (i.e., the index in the height direction) of the input image sub-feature in the spatial domain, and \(j\) represents the horizontal index (i.e., the index in the width direction) of the input image sub-feature in the input spatial domain; \(H\) represents the height of the input image sub-feature in the spatial domain, and \(W\) represents the width of the input image sub-feature in the spatial domain.

[0115] The input after uniform partitioning can be expressed by formula (3), and the expression of formula (3) is:

[0116]

[0117] where Freq n denotes the frequency-domain sub-feature corresponding to the image sub-feature of the \(n\)-th channel;

[0118] denotes the time-domain feature value at the position of the \(h\)-th row and \(w\)-th column in the \(n\)-th channel of the input image sub-feature. ":, h, w" means selecting the feature values of the \(h\)-th row and \(w\)-th column in all channels of the image feature; \(u\) represents the frequency index in the vertical direction in the two-dimensional frequency domain, \(v\) represents the frequency index in the horizontal direction in the two-dimensional frequency domain; \(N\) represents the total number of channels of the image feature, and \(n\) represents the channel index.

[0119] S43. Learn the weight relationship between different frequency-domain sub-features through a fully connected layer to obtain the weighted frequency-domain sub-features.

[0120] Input the frequency-domain sub-features Freq 0 , Freq 1 ,..., Freq n into the fully connected layer (FC). Learn the weight relationship between different spectral components through the fully connected layer to extract higher-level frequency-domain information; the output after being processed by the fully connected layer FC is a group of weighted frequency-domain sub-features, and the weighted frequency-domain sub-features are denoted as Freq 0’ , Freq 1’ ,..., Freq n’ .

[0121] S44. Generate channel attention weights based on the weighted frequency-domain sub-features, and multiply the channel attention weights with each channel of the image feature one by one to obtain the optimized image feature.

[0122] Generate channel attention weights using the weighted frequency-domain sub-features, multiply the generated weights with each channel of the original input feature map (channel-wise weighting) to strengthen important features and suppress unimportant features, and finally output a set of feature maps optimized by the attention mechanism. The multi-spectral channel attention mechanism enhances the model's ability to capture complex image details, thereby improving the accuracy of diabetic retinopathy detection results.

[0123] Based on the above description, use [u, v] to represent the component indices of the two-dimensional discrete cosine transform. After completing the two-dimensional discrete cosine transform, the obtained frequency blocks need to be concatenated together, and the concatenation formula is shown in Formula (4). The expression of Formula (4) is:

[0124] Frequency=Concat(Freq 0 ,Freq 1 ,...,Freq n )

[0125] Among them, the Concat(.) function is used to concatenate two or more arrays. The Concat(.) method does not change the existing arrays and only returns a copy of the concatenated arrays.

[0126] In the preferred implementation of this embodiment, after obtaining the optimized image features, the optimized image features are processed again through a fully connected layer and a Sigmoid activation function to obtain the mask information output (Output) for the attention mechanism. Subsequently, the mask information output is added to the original input image features to obtain the attention output. We further introduce an Attention-based Token Mixer (AttnTokenMixer). Token Mixer is a neural network architecture used in deep learning, mainly used to process and mix information units - tokens in the model. The core function of TokenMixer is to enhance the model's sensitivity to important details and improve learning efficiency by dynamically selecting key information for mixing. The AttnTokenMixer module effectively processes the feature map by combining Conditional Position Encoding (CPE) with the attention mechanism, captures long-range dependencies, and enhances the model's spatial perception ability by dynamically adjusting the position information in the feature map.

[0127] After completing the pre-training of the model and constructing the feature extraction layer, we enter the inference stage of the model, aiming to effectively classify new image data and retrieve using the pre-trained model. To achieve this goal, we add two fully connected (FC) layers at the end of the model, which are specifically responsible for mapping the extracted high-dimensional features to a predetermined class space.

[0128] Based on the pre-trained model, we added two fully connected layers to form a simple classifier. The image classification module in the model inference stage is as Figure 7 shown. The first fully connected layer is used to further integrate and transform the feature representation, and the second fully connected layer is used to output the final classification prediction result. An activation function is added between the first fully connected layer and the second fully connected layer to enhance the non-linear representation ability of the model. In the preferred embodiment of this embodiment, a dropout layer can also be inserted between the two fully connected layers to prevent overfitting.

[0129] We used a dataset containing 100 fundus photos to fine-tune the model to adapt to a specific image classification task. By learning from these new images, the weights of the diabetic retinopathy detection model are adjusted to make it more suitable for the target task. During the fine-tuning process, we kept most of the pre-trained layer parameters unchanged and only updated the parameters of the fully connected layers and the first few layers in front of them to avoid overly damaging the learned feature representation.

[0130] After fine-tuning, the diabetic retinopathy detection model is ready to classify new images. In the inference stage, the newly input fundus image first undergoes the same preprocessing as during training, including resizing and format conversion, and then is input into the diabetic retinopathy detection model.

[0131] The diabetic retinopathy detection model of this application can not only complete the image classification task, but also perform the image retrieval task. When performing the image retrieval task, first obtain the image database, and identify the image in the image database that best matches the input text description by analyzing and comparing the text descriptions and image features in the image database. In the image feature extraction stage, the diabetic retinopathy detection model uses a pre-trained network with an image encoder to iteratively extract features from each image in the training dataset, and uses global average pooling to reduce the dimension of the features, and stores the features after dimensionality reduction; for the input text description, the system first uses a text conversion function and a text encoder to convert it into a feature vector. Subsequently, the image features are transformed into the dimension space matching the text features through a feature transformation layer (a neural network module containing a linear layer); finally, the cosine similarity is used to calculate the similarity between the text features and each image feature, so as to identify the image that best matches the input text, return the single image index that best matches the input text description, and provide a series of the most relevant image indexes. In addition, the trained diabetic retinopathy detection model also includes a keyword checking function, which is used to verify whether the text description of the most matching image contains specific keywords.

[0132] The diabetic retinopathy detection model trained by using the diabetic retinopathy detection model training method of the present application was tested using the Kaggle APTOS dataset. Kaggle APTOS is the dataset of the "APTOS 2019 Blindness Detection" competition. The Kaggle APTOS dataset is mainly used for the automatic detection of diabetic retinopathy (DR). The Kaggle APTOS dataset contains high-quality fundus photos, aiming to help researchers develop and test algorithms for automatically diagnosing the severity of diabetic retinopathy.

[0133] During verification, we performed a binary classification task on diabetic retinopathy in the Kaggle APTOS dataset and used the five-fold cross-validation method to evaluate the performance of the model. The five-fold cross-validation method is a model evaluation technique widely used in the fields of machine learning and data science. It divides the original dataset into five equal-sized subsets in a specific way, and then sequentially uses one of the subsets as the test set, and the remaining four subsets are combined as the training set to train and evaluate the model. The process of training and evaluating the model needs to be repeated five times to ensure that each subset has been used as the test set once, so as to comprehensively evaluate the performance of the model. In this embodiment, the results of the verification of the data in the Kaggle APTOS dataset are referred to Table 1.

[0134] Table 1 - Binary classification results:

[0135]

[0136] As can be seen from Table 1, the average accuracy of the diabetic retinopathy detection model is between 90.20% and 92.38%, and the highest accuracy is 92.38%, which appears in the second fold. The area under the receiver operating characteristic curve (AUC) is between 0.933 and 1.00, and the highest value of 0.9583 is reached in the third fold. The sensitivity remains unchanged at 0.933 in the results of the third to fifth folds. The F1 score of the model reaches the highest (0.914) in the third fold, and the F1 scores in the fourth and fifth folds are the lowest (0.909), and the recall rate varies between 0.833 and 0.849. The specificity shows stability, ranging from 0.897 to 0.920, and the highest is 0.920 in the third fold.

[0137] These results indicate that our diabetic retinopathy detection model performs stably and reliably in the binary classification task of identifying diabetic retinopathy, accurately distinguishing pathological and non-pathological samples.

[0138] Based on the same inventive concept, an apparatus for training a diabetic retinopathy detection model according to an embodiment of the present invention includes an acquisition module, a construction module, and a training module.

[0139] The acquisition module is capable of acquiring a training data set, which includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images.

[0140] The construction module is capable of constructing a network structure of the diabetic retinopathy detection model.

[0141] The training module is capable of training the network of the diabetic retinopathy detection model using the training data set. In each training, the diabetic retinopathy detection model extracts image features of the fundus sample images and text features of the descriptive texts, calculates the contrast loss between the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model according to the contrast loss, so as to obtain the final diabetic retinopathy detection model.

[0142] The apparatus for training a diabetic retinopathy detection model according to the present invention can be loaded into an electronic device. The modules described in the present invention can also be referred to as units, which refer to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and are stored in the memory of the electronic device.

[0143] The various change modes and specific examples in the diabetic retinopathy detection model training method provided in the above embodiment are equally applicable to the apparatus for training a diabetic retinopathy detection model in this embodiment. Through the foregoing detailed description of the diabetic retinopathy detection model training method, those skilled in the art can clearly know the implementation method of the apparatus for training a diabetic retinopathy detection model in this embodiment. For the sake of brevity of the specification, it will not be elaborated herein.

[0144] Refer to Figure 8 , based on the same inventive concept, an embodiment of the present invention further provides a diabetic retinopathy detection method, and the method includes:

[0145] S51. Acquire a fundus image to be detected.

[0146] S52. Input the fundus image into the diabetic retinopathy detection model trained by using the diabetic retinopathy detection model training method provided in the above embodiment. The diabetic retinopathy detection model analyzes and processes the fundus image, and outputs the diabetic retinopathy classification result of the fundus image.

[0147] Refer to Figure 7 and Figure 8, in the diabetic retinopathy detection model, image features are extracted by a pre-trained backbone network and then passed to newly added fully connected layers, which process these features and output the prediction probabilities for each class. Based on these probabilities, the classification of the input fundus image is determined to complete the classification task and improve the accuracy of the diabetic retinopathy detection results.

[0148] This application also discloses an electronic device, as Figure 9 shown, which is a schematic structural diagram of an electronic device for the method of training a diabetic retinopathy detection model provided by an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively connected to the at least one processor, a communication bus 12, and a communication interface 13, and may further include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the method of training a diabetic retinopathy detection model.

[0149] Among them, the processor 10 may be composed of integrated circuits in some embodiments. For example, it may be composed of a single packaged integrated circuit, or may be composed of multiple integrated circuits with the same or different functions packaged, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting various components of the entire electronic device through various interfaces and lines, and by running or executing programs or modules stored in the memory 11 (such as executing the method of training a diabetic retinopathy detection model or the method of diabetic retinopathy detection, etc.), and calling data stored in the memory 11, to execute various functions of the electronic device and process data.

[0150] The memory 11 includes at least one type of readable storage medium, which includes flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device, such as the mobile hard disk of the electronic device. In some other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device. Further, the memory 11 can also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software installed in the electronic device and various types of data, such as the code of the method program for diabetic retinopathy detection, etc., but also to temporarily store data that has been output or will be output.

[0151] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection communication between the memory 11 and at least one processor 10, etc.

[0152] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), and is generally used to establish a communication connection between this electronic device and other electronic devices. The user interface can be a display, an input unit (such as a keyboard), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. Among them, the display can also be appropriately referred to as a display screen or a display unit, and is used to display the information processed in the electronic device and to display a visual user interface.

[0153] Figure 9 Only the electronic device with components is shown, and those skilled in the art can understand that Figure 9The structures shown do not constitute a limitation on the electronic device, and it may include fewer or more components than those shown, or combine certain components, or have different component arrangements.

[0154] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering each component. Preferably, the power source can be logically connected to at least one processor 10 through a power management device, so as to implement functions such as charging management, discharging management, and power consumption management through the power management device. The power source may also include any components such as one or more DC or AC power sources, a recharge device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The electronic device may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated here.

[0155] It should be understood that the embodiments are only for illustration purposes and are not limited by this structure in the scope of the patent application.

[0156] Furthermore, if the modules / units integrated in the electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0157] The embodiments of the present application provide a computer-readable storage medium, for example, including: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory). The computer-readable storage medium stores a computer program that can be loaded and executed by a processor to perform the diabetes retinopathy detection model training method in the above embodiments.

[0158] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", "one implementation manner", "one preferred implementation manner" or "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0159] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purposes of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A diabetic retinopathy detection model training method, characterized in that: Acquire a training data set, where the training data set includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images; Constructing the network structure of the diabetic retinopathy detection model; The network of the diabetic retinopathy detection model is iteratively trained using the training data set until the training stop condition is reached to obtain the final diabetic retinopathy detection model; In each training, the diabetic retinopathy detection model extracts image features of fundus sample images and text features of descriptive text, calculates the contrast loss of image features and text features, and determines the network parameters of the diabetic retinopathy detection model based on the contrast loss.

2. The diabetic retinopathy detection model training method according to claim 1, characterized in that: The steps of extracting image features of fundus sample images and text features of descriptive text by the diabetic retinopathy detection model are as follows: Extracting initial image features of fundus sample images; The initial image features are projected into a preset feature space through a first projection layer to obtain image features; Extracting initial text features of descriptive text; The initial text features are projected into the preset feature space through the second projection layer to obtain the text feature vector.

3. The diabetic retinopathy detection model training method according to claim 2, characterized in that: After projecting the initial image features to the preset feature space through the first projection layer to obtain the image features, the method further includes: The image features are processed through the multi-spectral channel attention mechanism to obtain optimized image features.

4. The diabetic retinopathy detection model training method according to claim 3, characterized in that: The step of processing image features through a multi-spectral channel attention mechanism comprises: Decompose the image features into several image sub-features; Perform a two-dimensional discrete cosine transform on the spatial dimension of the image sub-feature to obtain the frequency domain sub-feature corresponding to the image sub-feature; The weight relationship between different frequency domain sub-features is learned through the fully connected layer to obtain the weighted frequency domain sub-features; Channel attention weights are generated based on the weighted frequency domain sub-features, and the channel attention weights are multiplied with each channel of the image feature one by one to obtain optimized image features.

5. The diabetic retinopathy detection model training method according to claim 4, characterized in that: The transformation formula of image sub-features from time domain to frequency domain through two-dimensional discrete cosine transform is as follows: Among them, f h,w represents the eigenvalue of the point at coordinates (h, w) in the two-dimensional frequency domain, h represents the vertical frequency index in the two-dimensional frequency domain, w represents the horizontal frequency index in the two-dimensional frequency domain, x (i,j) Represents the pixel value of the point corresponding to the i-th row and j-th column of the input image sub-feature in the spatial domain, i represents the vertical index of the input image sub-feature in the spatial domain, j represents the horizontal index of the input image sub-feature in the spatial domain; H represents the height of the input image sub-feature in the spatial domain, and W represents the width of the input image sub-feature in the spatial domain; The frequency domain sub-feature Freq corresponding to the image sub-feature of the nth channel n for: Among them, Freq n Represents the frequency domain sub-feature corresponding to the image sub-feature of the nth channel; It represents the time domain feature value of the input image sub-feature at the position of the h-th row and the w-th column in the n-th channel. ":,h,w" means selecting the feature value of the h-th row and the w-th column in all channels of the image feature. u represents the frequency index in the vertical direction in the two-dimensional frequency domain, v represents the frequency index in the horizontal direction in the two-dimensional frequency domain; N represents the total number of channels of the image feature, n represents the index of the channel, n is an integer, and N is a positive integer.

6. The diabetic retinopathy detection model training method according to any one of claims 1 to 5, characterized in that: The loss function of the diabetic retinopathy detection model is Where m represents the total number of fundus sample images in the training data set, a represents the index of the fundus sample image in the training data set, α is the scaling factor, and λ is the marginal parameter; S ab represents the similarity score between fundus sample image a and fundus sample image b in the training data set; P a Represents the probability that the diabetic retinopathy classification result of fundus sample image a belongs to the correct category.

7. A diabetic retinopathy detection model training device, characterized in that: An acquisition module is used to acquire a training data set, where the training data set includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images; A building module for constructing the network structure of a diabetic retinopathy detection model; The training module is used to iteratively train the network of the diabetic retinopathy detection model using the training data set until the training stop condition is reached to obtain the final diabetic retinopathy detection model; in each training, the diabetic retinopathy detection model extracts image features of fundus sample images and text features of descriptive texts, calculates the contrast loss of image features and text features, and determines the network parameters of the diabetic retinopathy detection model based on the contrast loss.

8. A method for detecting diabetic retinopathy, characterized in that: include: Acquiring a fundus image to be detected; The fundus image is input into a diabetic retinopathy detection model trained by the diabetic retinopathy detection model training method according to any one of claims 1 to 6. The diabetic retinopathy detection model analyzes and processes the fundus image and outputs a diabetic retinopathy classification result of the fundus image.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the method as claimed in claim 1 or 2 or 3 or 4 or 5 or 6 or 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the method according to claim 1 or 2 or 3 or 4 or 5 or 6 or 8 is implemented.

Images