Diabetic retinopathy detection model training method, training device, detection method, equipment and storage medium

Abstract

The application relates to the technical field of image processing, and relates to a diabetic retinopathy detection system, equipment and a storage medium, wherein a diabetic retinopathy detection model training method comprises the following steps: acquiring a training data set, wherein the training data set comprises a plurality of fundus sample images and corresponding descriptive texts of the fundus sample images; constructing a network structure of a diabetic retinopathy detection model; iteratively training the network of the diabetic retinopathy detection model by using the training data set until a training stop condition is reached, so that a final diabetic retinopathy detection model is obtained; in each training, the diabetic retinopathy detection model extracts image features of the fundus sample images and extracts text features of the descriptive texts, calculates a contrast loss of the image features and the text features, and determines network parameters of the diabetic retinopathy detection model according to the contrast loss. The application can improve the accuracy of a diabetic retinopathy detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a diabetic retinopathy detection model training method, a training device, a detection method, equipment and a storage medium. BACKGROUND

[0002] Diabetic retinopathy is a microvascular complication caused by diabetes and is a common disease among diabetic patients. At present, about one-third of diabetic patients suffer from eye diseases. Diabetic retinopathy is divided into two categories: non-proliferative and proliferative. Non-proliferative diabetic retinopathy includes mild, moderate and severe, while proliferative diabetic retinopathy can be divided into high risk and non-high risk. Other characteristics of non-proliferative diabetic retinopathy include soft or hard exudates and retinal microvascular abnormalities. Non-proliferative diabetic retinopathy is relatively stable and is easier to treat, but if not treated and intervened in time, it can develop into proliferative diabetic retinopathy, which can seriously affect the patient's vision and even lead to complete blindness.

[0003] Therefore, in order to prevent diabetic retinopathy from worsening and causing irreversible damage to the patient's vision, it is crucial to diagnose and treat diabetic retinopathy as early as possible. At present, the diagnosis of diabetic retinopathy relies on the comprehensive analysis of experienced doctors on fundus images and fundus fluorescence angiography to determine the condition, or the use of computers to assist doctors in diagnosing the patient's retinopathy.

[0004] In the research of computer-aided diabetic retinopathy diagnosis, we face a series of challenges, including insufficient feature extraction and representation, data imbalance and quality problems, and insufficient model robustness. Existing diabetic retinopathy detection models may have limitations in extracting and representing key features from different modalities, which directly affects the diagnostic performance. In addition, multi-modal data sets are often affected by data imbalance and quality inconsistency, which can lead to model bias and overfitting. SUMMARY

[0005] In order to improve the accuracy of diabetic retinopathy detection results, a diabetic retinopathy detection model training method, system, equipment and storage medium are provided.

[0006] In order to achieve the above-mentioned purpose of the present application, the present application provides a diabetic retinopathy detection model training method.

[0007] The diabetic retinopathy detection model training method provided by the present application comprises:

[0008] Obtaining a training data set, the training data set comprising a plurality of fundus sample images and descriptive text corresponding to the fundus sample images;

[0009] constructing a network structure of a diabetic retinopathy detection model;

[0010] iteratively training the network of the diabetic retinopathy detection model using the training data set until a training stop condition is reached, to obtain a final diabetic retinopathy detection model;

[0011] In each training, the diabetic retinopathy detection model extracts image features of the fundus sample image and text features of the descriptive text, calculates a contrast loss of the image features and the text features, and determines network parameters of the diabetic retinopathy detection model according to the contrast loss.

[0012] Optionally, the steps of extracting the image features of the fundus sample image and the text features of the descriptive text by the diabetic retinopathy detection model are:

[0013] extracting initial image features of the fundus sample image;

[0014] projecting the initial image features to a preset feature space through a first projection layer to obtain the image features;

[0015] extracting initial text features of the descriptive text;

[0016] projecting the initial text features to the preset feature space through a second projection layer to obtain a text feature vector.

[0017] Optionally, after the initial image features are projected to the preset feature space through the first projection layer to obtain the image features, the method further includes:

[0018] processing the image features through a multi-spectral channel attention mechanism to obtain optimized image features.

[0019] Optionally, the step of processing the image features through the multi-spectral channel attention mechanism includes:

[0020] decomposing the image features into a plurality of image sub-features;

[0021] performing two-dimensional discrete cosine transformation on a spatial dimension of the image sub-features to obtain frequency domain sub-features corresponding to the image sub-features;

[0022] learning a weight relationship between different frequency domain sub-features through a fully connected layer to obtain weighted frequency domain sub-features;

[0023] generating channel attention weights based on the weighted frequency domain sub-features, and multiplying the channel attention weights with each channel of the image features one by one to obtain the optimized image features.

[0024] Optionally, the image sub-feature is converted from the time domain to the frequency domain by a two-dimensional discrete cosine transform, and the conversion formula is as follows:

[0025]

[0026] Wherein, f h，w represents the feature value of the point with coordinates (h, w) in the two-dimensional frequency domain, h represents the vertical frequency index in the two-dimensional frequency domain, w represents the horizontal frequency index in the two-dimensional frequency domain, x (i，j) represents the pixel value of the point corresponding to the input image sub-feature in the i-th row and the j-th column in the spatial domain, i represents the vertical index of the input image sub-feature in the spatial domain, j represents the horizontal index of the input image sub-feature in the spatial domain; H represents the height of the input image sub-feature in the spatial domain, and W represents the width of the input image sub-feature in the spatial domain.

[0027] The frequency domain sub-feature Freq n is:

[0028]

[0029] represents the time domain feature value of the input image sub-feature in the n-th channel at the position of the h-th row and the w-th column, and "h, w" represents the feature value of the h-th row and the w-th column in all channels of the image feature.

[0030] u represents the vertical frequency index in the two-dimensional frequency domain, v represents the horizontal frequency index in the two-dimensional frequency domain; N represents the total number of channels of the image feature, n represents the index of the channel, n is an integer, and N is a positive integer.

[0031] Optionally, the loss function of the diabetic retinopathy detection model is

[0032] Wherein, m represents the total number of sample images in the training data set, a represents the index of the fundus sample image in the training data set, alpha is a scaling factor, and lambda is a marginal parameter; S ab represents the similarity score between the fundus sample image a and the fundus sample image b in the training data set; P a represents the probability that the diabetic retinopathy classification result of the fundus sample image a belongs to the correct class.

[0033] In order to solve the above problems, the application also provides a diabetic retinopathy detection model training device, comprising:

[0034] The acquisition module is used for acquiring a training data set, and the training data set comprises a plurality of fundus sample images and descriptive texts corresponding to the fundus sample images;

[0035] a construction module configured to construct a network structure of the diabetic retinopathy detection model;

[0036] a training module configured to train the network of the diabetic retinopathy detection model by using a training data set, in each training, the diabetic retinopathy detection model extracts image features of the fundus sample image and text features of the descriptive text, calculates a contrast loss of the image features and the text features, and determines network parameters of the diabetic retinopathy detection model according to the contrast loss to obtain a final diabetic retinopathy detection model.

[0037] To solve the above problems, the present application further provides a diabetic retinopathy detection method, which comprises:

[0038] obtaining a fundus image to be detected;

[0039] inputting the fundus image into the diabetic retinopathy detection model trained by the diabetic retinopathy detection model training method, and outputting a diabetic retinopathy classification result of the fundus image by analyzing and processing the fundus image by the diabetic retinopathy detection model.

[0040] To solve the above problems, the present application further provides an electronic device, which comprises:

[0041] at least one processor; and

[0042] a memory connected in communication with the at least one processor; wherein

[0043] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the diabetic retinopathy detection model training method or the diabetic retinopathy detection method.

[0044] To solve the above problems, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the diabetic retinopathy detection model training method or the diabetic retinopathy detection method.

[0045] In summary, the present application has the following beneficial technical effects:

[0046] Through the contrast training of the diabetic fundus sample image and the descriptive text, the pre-training of the diabetic retinopathy detection model is completed; by using the alignment between the image and the text, the diabetic retinopathy detection model is taught to distinguish the relevant and irrelevant features. This process enables the diabetic retinopathy detection model to capture the subtle correlation between the visual patterns in the fundus sample image and the semantic information in the descriptive text, thereby more deeply understanding the data, so that the diabetic retinopathy detection model can be more effectively generalized to the diversified and unlabeled data set, and the accuracy of the diabetic retinopathy detection result is improved;

[0047] The multi-spectral channel attention mechanism is used to optimize the image features, the ability of the diabetic retinopathy detection model to capture complex image details is enhanced by processing the frequency domain sub-features corresponding to the image sub-features, and therefore the accuracy of the diabetic retinopathy detection result is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the diabetic retinopathy detection model training method provided by an embodiment of the present application is shown;

[0049] Figure 2 The explanatory diagram of an embodiment of the diabetic retinopathy detection model training method provided by an embodiment of the present application is shown;

[0050] Figure 3 The system architecture diagram of the image editor provided by an embodiment of the present application is shown;

[0051] Figure 4 The specific architecture diagram of each module in the image editor provided by an embodiment of the present application is shown;

[0052] Figure 5 The step diagram of processing the image features by the multi-spectral channel attention mechanism provided by an embodiment of the present application is shown;

[0053] Figure 6 The flowchart of processing the image features by the multi-spectral channel attention mechanism provided by an embodiment of the present application is shown;

[0054] Figure 7 The module diagram of the image classification module in the inference stage provided by an embodiment of the present application is shown;

[0055] Figure 8 The flowchart of the diabetic retinopathy detection method provided by an embodiment of the present application is shown;

[0056] Figure 9 The structural diagram of the electronic device provided by an embodiment of the present application is shown.

[0057] Reference signs: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0058] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0059] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0060] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0061] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connection", "connection" should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between the two elements inside, it can be directly connected, or indirectly connected through an intermediate medium, and those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0062] Figure 1 The flowchart of the diabetes retinopathy detection model training method provided by an embodiment of the present application is shown. In this embodiment, the diabetes retinopathy detection model training method comprises:

[0063] S1, obtaining a training data set.

[0064] The training data set includes a plurality of fundus sample images and descriptive texts corresponding to the fundus sample images. Specifically, the content of the descriptive text includes a textual description of the details such as the lesion area, blood vessel texture or color distribution in the fundus sample image.

[0065] The diabetic retinopathy includes two categories of non-proliferative diabetic retinopathy and proliferative diabetic retinopathy, wherein the non-proliferative diabetic retinopathy includes three sub-categories of mild, moderate and severe, and the proliferative diabetic retinopathy is divided into two sub-categories of high risk and non-high risk; during the training, the fundus sample image and the descriptive text are input into the diabetic retinopathy detection model, and the diabetic retinopathy detection model determines, through analysis of the fundus sample image and the descriptive text, to which sub-category of which category of diabetic retinopathy the fundus sample image corresponds, thereby completing classification of the diabetic retinopathy corresponding to the fundus sample image.

[0066] After the diabetic retinopathy detection model outputs the classification result of the fundus sample image, the classification result is compared with the descriptive text, and the network parameters of the diabetic retinopathy detection model are adjusted according to the comparison result, thereby improving the training efficiency of the diabetic retinopathy detection model and improving the accuracy of the diabetic retinopathy type prediction result of the diabetic retinopathy detection model on the fundus image.

[0067] In the embodiment, the fundus image of the training data set can adopt the ODIR-5K data set, and the ODIR-5K (Ocular Disease Intelligent Recognition) is a large-scale eye disease recognition data set, which aims to promote the automatic recognition and classification research of eye diseases. The data set includes 5,000 fundus photos from different patients, covering multiple ages, genders and races. Each photo is annotated by professional ophthalmologists, including eight categories of eye diseases such as normal, diabetic retinopathy, glaucoma and cataract; the training data set can also select other eye disease recognition data sets containing diabetic retinopathy images, and the embodiment does not make any limitation.

[0068] S2, constructing a network structure of a diabetic retinopathy detection model.

[0069] The network structure of the diabetic retinopathy detection model includes an image feature extraction module, a text feature extraction module and a classification module, the image feature extraction module is used for feature extraction of the input image to obtain image features, the text feature extraction module is used for feature extraction of the input text to obtain text features, and the classification module is used for determining the classification prediction result of the diabetic retinopathy corresponding to the input image according to the image features and the text features.

[0070] S3, iteratively training the network of the diabetic retinopathy detection model by using the training data set until a training stop condition is reached, thereby obtaining a final diabetic retinopathy detection model;

[0071] In each training session, the diabetic retinopathy detection model extracts image features of fundus sample images and text features of descriptive text, calculates the contrast loss of image features and text features, and determines the network parameters of the diabetic retinopathy detection model based on the contrast loss.

[0072] The training stopping condition is that when the model training times reaches the maximum times or the loss function value of the diabetic retinopathy detection model fluctuates within a certain range and no longer decreases significantly, the model can be considered to have converged and the training can be terminated.

[0073] In this embodiment, the binomial loss function is used to evaluate the difference between the predicted probability and the actual binary label. The diabetic retinopathy detection model adopts the binomial loss function, which effectively improves the accuracy of feature matching and significantly improves the efficiency of processing image and text data and the accuracy of feature extraction, thereby solving the limitations of traditional methods in this field.

[0074] The loss function of the diabetic retinopathy detection model is

[0075]

[0076] Wherein, m represents the total number of fundus sample images in the training dataset, a represents the index of the fundus sample image in the training dataset, α is the scaling factor, which is set to 2.0 in this application, and λ is the margin parameter, which is set to 0.5 in this application; S ab represents the similarity score between fundus sample image a and fundus sample image b in the training dataset; P a It represents the probability that the diabetic retinopathy classification result of fundus sample image a belongs to the correct category.

[0077] Specifically, the calculation formula for the similarity score between fundus sample image a and fundus sample image b in the training data set is:

[0078]

[0079] Among them, v a represents the feature vector corresponding to the fundus sample image a, w b represents the feature vector corresponding to the fundus sample image b, ||v a || represents the feature vector v a The modulus, ||w b || represents the eigenvector w b The model;

[0080] The probability P that the diabetic retinopathy classification result of fundus sample image a belongs to the correct category a is obtained by applying the softmax function to the vector representation, P aIt is obtained by the indicator function 1[y a =y b ] is summed, excluding the current fundus sample image a, and then the result is raised to the Kth power to calculate, where K is a parameter that adjusts the weight of each item. a The calculation formula is

[0081]

[0082] Among them, y a represents the true category of fundus sample image a, y b represents the true category of fundus sample image b. [y a =y b ] is an indicator function, y a =y b The value is 1 when , otherwise it is 0.

[0083] In a preferred embodiment of this embodiment, before the fundus sample images and descriptive texts are input into the diabetic retinopathy detection model, the data in the training data set needs to be preprocessed. Data preprocessing includes image preprocessing and text preprocessing. The preprocessing of image data includes image resizing and format conversion. Image resizing is to adjust the fundus sample images in the training data set to a size of 224×224 pixels to meet the size requirements of the model input. Format conversion is used to convert image data into a tensor format that can be processed by the diabetic retinopathy detection model. Simply put, it converts the image data into the tensor format (C×H×W) used by PyTorch, and then maps the pixel values ​​from the integer [0, 255] interval to the floating point number [0, 1] interval to achieve the unification of the image format and prepare for subsequent model calculations.

[0084] The goal of the text preprocessing stage is to convert natural language text (i.e., descriptive text) into a numerical representation that is more suitable for deep learning models. Specifically, we first perform word segmentation on the descriptive text, dividing it into several word sequences, and then use a pre-built vocabulary index dictionary to map each word to a specific index value. These index values ​​are then passed to an embedding layer, which converts each index value into a vector representation in a high-dimensional space. By calculating the average of these vectors, a fixed-dimensional vector representing the entire descriptive text is obtained.

[0085] Through the preprocessing step, we can provide consistent and standardized image and text data for the diabetic retinopathy detection model, enhance the model's ability to handle multimodal data, and lay a solid foundation for the subsequent feature extraction and model training stages.

[0086] After completing data preprocessing, we use a contrastive architecture to accurately and efficiently process and understand fundus sample images and descriptive text associated with the fundus sample images.

[0087] Reference Figure 2 The steps for the diabetic retinopathy detection model to extract image features of fundus sample images and text features of descriptive text are as follows:

[0088] S31: extracting initial image features of the fundus sample image.

[0089] Specifically, the image editor receives the pre-processed fundus sample image f θ Afterwards, the fundus sample image f θ Perform feature extraction to obtain the initial image features using h i .

[0090] Reference Figure 3 , the image editor includes a base block, four stages, patch embedding block, MobileOne block and attention block; among them, the four stages include the first stage ( Figure 3 stage1 in the second stage ( Figure 3 stage2 in the third stage ( Figure 3 stage3) and the fourth stage ( Figure 3 stage4 in the .

[0091] Figure 4 This is a detailed architecture diagram of each module in the image editor, where: Figure 4 (a) represents the first stage / second stage / third stage architecture. The architecture of the first stage, second stage, and third stage is the same. Figure 4 (b) in the figure represents the architecture of the batch normalization layer. Figure 4 (c) in the figure represents the architecture of the base block. Figure 4 (d) in the figure represents the fourth stage of the architecture. Figure 4 (e) in represents the architecture of the convolutional feedforward network. Figure 4 (f) in the figure represents the architecture of the MobileOne block. The MobileOne block reduces the amount of computation by merging multiple convolutional blocks. The MobileOne block can reduce the computational complexity of the image editor while maintaining efficient feature extraction capabilities.

[0092] Reference Figure 4 ,In the first, second and third stages, the input image first passes through a batch normalization layer, such as Figure 4(b) As shown, the batch normalization layer stabilizes the learning process by centering and rescaling the input layer. Next, the normalized input is fed into a deep convolutional layer with a kernel size of 3x3 and a stride of 2. This layer applies a single filter to each input channel, implementing a lightweight filtering process that captures spatial features. The output of the deep convolutional layer is then passed into a convolutional feed-forward network (ConvFFN block), which starts with another deep convolutional layer with a larger kernel size of 7x7 and a stride of 2, further processing higher-level spatial features while reducing the feature dimension. After this, another round of batch normalization is applied to further standardize the newly transformed features; then, the features are processed by a 1x1 convolutional layer that merges features across channels; next, an activation function (e.g., ReLU) is applied to introduce nonlinearity into the model, enabling it to learn more complex patterns. Subsequently, another 1x1 convolutional layer is applied to expand the dimension or adjust the number of channels. Finally, the output of the second 1x1 convolutional layer is combined with the initial input of the ConvFFN block through a residual connection, represented by the addition operation in the figure.

[0093] In the fourth stage, we added conditional positional encoding (CPE), which can adapt to understand the positions of pixels or regions within an image. The MobileOne block further reduces computational complexity while maintaining efficient feature extraction capabilities. In the preferred implementation of the present embodiment, a multi-head self-attention mechanism is used when fusing multiple extracted features to enhance the model's perception of spatial relationships within the image. The multi-head self-attention mechanism improves feature representation by learning dependencies between input features. The self-attention module we designed allows different attention heads to be applied to different subsets of features, thereby improving the accuracy and efficiency of feature extraction.

[0094] S32, projecting the initial image features to a preset feature space through a first projection layer to obtain image features.

[0095] wherein the preset feature space is a shared space for aligning image features and text features; specifically, the initial image features h i through the projection layer g θ and the projection layer q θ The role of the projection layer is to apply a certain mapping to a high-dimensional feature vector to obtain a vector represented in a new coordinate system or new dimension, and to normalize the vector, i.e., linearly map the data to the [0, 1] interval. This process can rearrange high-dimensional features into a feature space more suitable for subsequent tasks, i.e., project the initial image features h i to a new feature space v i The new feature space v i is used to enable the image features to match the text features.

[0096] S33. Extract initial text features of the descriptive text.

[0097] Reference Figure 2 , the text encoder is connected to the preprocessed descriptive text f' i Perform feature extraction to obtain the initial text feature h i ′.

[0098] S34. Project the initial text features to a feature space corresponding to the image features through a second projection layer to obtain a text feature vector.

[0099] Reference Figure 2 , initial text feature h i 'Through the projection layer g θ ′, the initial text feature h i ′ is projected into the new feature space z that is the same as the image i ′. These vectors are used to compute a loss function that measures the similarity between the image and text representations. In addition, we use a stopped gradient method to prevent gradients from backpropagating to the text branch, reducing the possibility of representation learning falling into a collapsed solution.

[0100] In a preferred implementation of this embodiment, after the initial image features are projected into a preset feature space by the first projection layer to obtain image features, the image features are processed by a multi-spectral channel attention mechanism to obtain optimized image features;

[0101] The network parameters of the diabetic retinopathy detection model are determined based on text features and optimized image features.

[0102] Reference Figure 5 , the multispectral channel attention mechanism is a module that can describe multispectral information. The steps of processing image features through the multispectral channel attention mechanism include:

[0103] S41. Decompose the image feature into several image sub-features.

[0104] Reference Figure 6 , the input image feature map size is H×W×C, where H and W are the spatial dimensions (i.e., height and width), and C is the number of channels; each channel of the input feature map is divided into several sub-features C′, and each channel is divided into smaller spectra or regions to form a new representation structure; the output shape is H×W×C′×n, where n is the number of sub-features after division.

[0105] Specifically, the input image feature is recorded as x, and the input x is manually divided into N parts before entering the multi-spectral channel attention module. The divided input x n Use [x 1 , x2 , x 3 ,...,x (N-1) ] is expressed as shown in formula (1).

[0106] The expression of formula (1) is:

[0107]

[0108] Where n represents the channel index of the image feature, N represents the total number of channels of the image feature, n is an integer, and N is a positive integer; Represents all spatial positions (usually corresponding to height and width) of the nth channel (feature dimension) in the output feature tensor. Specifically, all spatial positions of the nth channel (feature dimension) refer to the points of the nth channel in the output feature tensor; X ：，n，：，： Represents the spatial position information of the input image sub-feature, the symbol “:” represents the dimensional data of the input image sub-feature; “:,:” represents the spatial dimension (such as the height H and width W of the feature map).

[0109] S42. Perform a two-dimensional discrete cosine transform on the spatial dimension of the image sub-feature to obtain a frequency domain sub-feature corresponding to the image sub-feature.

[0110] Specifically, a two-dimensional discrete cosine transform (2D-DCT) is performed on the spatial dimension of each sub-feature C′; DCT maps the input spatial features to the frequency domain, captures the low-frequency and high-frequency information in the spatial structure, and generates spectral features.

[0111] The output result is a set of frequency domain sub-features Freq 0 , Freq 1 ,...,Freq n , each spectral sub-feature corresponds to a specific frequency component.

[0112] The two-dimensional discrete cosine transform (DCT) formula refers to formula (2), and the expression of formula (2) is:

[0113]

[0114] h∈[0,H-1],w∈[0,W-1]where f h，w represents the eigenvalue of the point at coordinates (h, w) in the two-dimensional frequency domain, h represents the vertical frequency index in the two-dimensional frequency domain (i.e., the index representing the height direction), and w represents the horizontal frequency index in the two-dimensional frequency domain (i.e., the index representing the width direction); x (i，j)represents the pixel value of the point corresponding to the i-th row and j-th column in the spatial domain of the input image sub-feature, i represents the vertical index (i.e., the index in the height direction) of the input image sub-feature in the spatial domain, j represents the horizontal index (i.e., the index in the width direction) of the input image sub-feature in the input spatial domain; H represents the height of the input image sub-feature in the spatial domain, and W represents the width of the input image sub-feature in the spatial domain.

[0115] The input after uniform division can be represented by formula (3), and the expression of formula (3) is:

[0116]

[0117] wherein, Freq n represents the frequency domain sub-feature corresponding to the image sub-feature of the n-th channel.

[0118] represents the time domain feature value of the input image sub-feature at the position of the h-th row and w-th column in the n-th channel, and “: h, w” represents the feature value of the h-th row and w-th column in all channels of the image feature; u represents the frequency index in the vertical direction in the two-dimensional frequency domain, and v represents the frequency index in the horizontal direction in the two-dimensional frequency domain; N represents the total number of channels of the image feature, and n represents the index of the channel.

[0119] S43, learning the weight relationship between different frequency domain sub-features through a fully connected layer to obtain weighted frequency domain sub-features.

[0120] The frequency domain sub-features Freq 0 , Freq 1 , …, Freq n are input into a fully connected layer (Fully Connected, FC), the weight relationship between different frequency components is learned through the fully connected layer, and higher-level frequency domain information is extracted; the output after the processing of the fully connected layer FC is a set of weighted frequency domain sub-features, and the weighted frequency domain sub-features are represented as Freq 0’ , Freq 1’ , …, Freq n’ .

[0121] S44, generating channel attention weights based on the weighted frequency domain sub-features, and multiplying the channel attention weights with each channel of the image feature one by one to obtain an optimized image feature.

[0122] The channel attention weight is generated using the weighted frequency domain sub-feature, and the generated weight is multiplied with each channel of the original input feature map (channel-by-channel weighting), which strengthens important features and suppresses unimportant features. Finally, a set of feature maps optimized by the attention mechanism is output. The multi-spectral channel attention mechanism enhancement model can capture complex image details, thereby improving the accuracy of diabetic retinopathy detection results.

[0123] Based on the above description, [u, v] is used to represent the component index of two-dimensional discrete cosine transform. After completing the two-dimensional discrete cosine transform, the obtained frequency block needs to be connected together. The connection formula is shown in formula (4), and the expression of formula (4) is:

[0124] Frequency=Concat(Freq 0 ,Freq 1 ,...,Freq n )

[0125] Wherein, the Concat(.) function is used to connect two or more arrays. The Concat(.) method does not change the existing array, but only returns a copy of the connected array.

[0126] In the preferred embodiment of the present embodiment, after obtaining the optimized image features, the optimized image features are again processed through a fully connected layer and a Sigmoid activation function to obtain the mask information output (output) for the attention mechanism. Then, the mask information output is added to the original input image features to obtain the attention output. We further introduce an attention-based Token Mixer (AttnTokenMixer). Token Mixer is a neural network architecture used in deep learning, mainly used to process and mix tokens in the model. The core function of TokenMixer is to mix key information by dynamic selection, enhance the sensitivity of the model to important details, and improve the learning efficiency. The AttnTokenMixer module effectively processes the feature map by combining the conditional position encoding (CPE) with the attention mechanism, dynamically adjusts the position information in the feature map, captures long-distance dependencies and enhances the spatial perception ability of the model.

[0127] After completing the pre-training of the model and constructing the feature extraction layer, we enter the inference stage of the model, which aims to effectively classify new image data and use the pre-trained model for retrieval. To achieve this goal, we add two fully connected (FC) layers at the end of the model, which are specifically responsible for mapping the extracted high-dimensional features to the predetermined category space.

[0128] Based on the pre-trained model, we added two fully connected layers to form a simple classifier. The image classification module in the model inference phase is as follows: Figure 7 As shown, the first fully connected layer is used to further integrate and transform feature representations, and the second fully connected layer is used to output the final classification prediction results. An activation function is added between the first and second fully connected layers to enhance the nonlinear representation capabilities of the model. In the preferred implementation of this embodiment, a dropout layer can also be inserted between the two fully connected layers to prevent overfitting.

[0129] We fine-tuned the model using a dataset of 100 fundus images for a specific image classification task. Learning from these new images adjusted the weights of the diabetic retinopathy detection model, making it more suitable for the target task. During fine-tuning, we kept most of the pre-trained layer parameters unchanged, updating only the parameters of the fully connected layers and their preceding layers to avoid excessively disrupting the learned feature representations.

[0130] After fine-tuning, the diabetic retinopathy detection model is ready to classify new images. During the inference phase, new input fundus images first undergo the same preprocessing as during training, including resizing and format conversion, before being fed into the diabetic retinopathy detection model.

[0131] The diabetic retinopathy detection model of this application can not only complete image classification tasks, but also perform image retrieval tasks. When performing image retrieval tasks, it first obtains an image database, analyzes and compares the text descriptions and image features in the image database, and identifies the image from the image database that best matches the input text description. In the image feature extraction stage, the diabetic retinopathy detection model uses a pre-trained network with an image encoder to iteratively extract features from each image in the training dataset, and uses global average pooling to reduce the feature dimensionality and store the reduced features. For the input text description, the system first uses a text conversion function and a text encoder to convert it into a feature vector. Subsequently, a feature conversion layer (a neural network module containing a linear layer) converts the image features into a dimensional space that matches the text features. Finally, cosine similarity is used to calculate the similarity between the text features and each image feature, thereby identifying the image that best matches the input text, returning the single image index that best matches the input text description, and providing a list of the most relevant image indexes. In addition, the trained diabetic retinopathy detection model also includes a keyword check function, which is used to verify whether the text description of the best matching image contains specific keywords.

[0132] The diabetic retinopathy detection model trained using the diabetic retinopathy detection model training method described in this application was tested using the Kaggle APTOS dataset. Kaggle APTOS is the dataset for the "APTOS 2019 Blindness Detection" competition. The Kaggle APTOS dataset is primarily used for automatically detecting diabetic retinopathy (DR). The Kaggle APTOS dataset contains high-quality fundus images and is designed to help researchers develop and test algorithms for automatically diagnosing the severity of diabetic retinopathy.

[0133] During validation, we performed a binary classification task on diabetic retinopathy in the Kaggle APTOS dataset and used a five-fold cross-validation method to evaluate the performance of the model. The five-fold cross-validation method is a model evaluation technique widely used in the fields of machine learning and data science. It divides the original dataset into five equal-sized subsets in a specific way, and then uses one of the subsets as a test set in turn, and the remaining four subsets are combined as training sets to train and evaluate the model. The process of training and evaluating the model needs to be repeated five times to ensure that each subset is used as a test set once, so as to comprehensively evaluate the performance of the model. In this embodiment, the results of the verification of the data in the Kaggle APTOS dataset refer to Table 1.

[0134] Table 1-Two-classification results:

[0135]

[0136] As shown in Table 1, the average accuracy of the diabetic retinopathy detection model ranged from 90.20% to 92.38%, with the highest accuracy of 92.38% occurring in the second fold. The area under the receiver operating characteristic curve (AUC) ranged from 0.933 to 1.00, reaching a peak of 0.9583 in the third fold. Sensitivity remained constant at 0.933 from the third to fifth folds. The model's F1 score reached its highest point (0.914) in the third fold and was lowest (0.909) in the fourth and fifth folds. Recall varied between 0.833 and 0.849. Specificity remained stable, ranging from 0.897 to 0.920, reaching a peak of 0.920 in the third fold.

[0137] These results demonstrate that our diabetic retinopathy detection model performs stably and reliably in the binary classification task of identifying diabetic retinopathy, accurately distinguishing between pathological and non-pathological samples.

[0138] Based on the same inventive concept, an embodiment of the present invention provides a diabetic retinopathy detection model training device, comprising an acquisition module, a construction module and a training module.

[0139] The acquisition module can acquire a training data set, which includes a number of fundus sample images and descriptive texts corresponding to the fundus sample images;

[0140] The building block is capable of constructing the network structure of the diabetic retinopathy detection model;

[0141] The training module can use the training data set to train the network of the diabetic retinopathy detection model. In each training, the diabetic retinopathy detection model extracts the image features of the fundus sample image and the text features of the descriptive text, calculates the contrast loss of the image features and the text features, and determines the network parameters of the diabetic retinopathy detection model based on the contrast loss to obtain the final diabetic retinopathy detection model.

[0142] The diabetic retinopathy detection model training device described in the present invention can be installed in an electronic device. The module described in the present invention, also known as a unit, refers to a series of computer program segments that can be executed by an electronic device processor and can perform fixed functions, and is stored in the memory of the electronic device.

[0143] The various variations and specific examples of the diabetic retinopathy detection model training method provided in the above embodiments are also applicable to the diabetic retinopathy detection model training device of this embodiment. Through the above detailed description of the diabetic retinopathy detection model training method, those skilled in the art can clearly know the implementation method of the diabetic retinopathy detection model training device of this embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0144] Reference Figure 8 Based on the same inventive concept, an embodiment of the present invention further provides a method for detecting diabetic retinopathy, the method comprising:

[0145] S51 : Acquire a fundus image to be detected.

[0146] S52: Input the fundus image into the diabetic retinopathy detection model trained by the diabetic retinopathy detection model training method provided by the above embodiment. The diabetic retinopathy detection model analyzes and processes the fundus image and outputs a diabetic retinopathy classification result of the fundus image.

[0147] Reference Figure 7 and Figure 8In the diabetic retinopathy detection model, image features are extracted through a pretrained backbone network and then passed to newly added fully connected layers. These layers process these features and output predicted probabilities for each class. Based on these probabilities, the input fundus image is classified, completing the classification task and improving the accuracy of diabetic retinopathy detection results.

[0148] This application also discloses an electronic device, such as Figure 9 FIG2 is a schematic diagram of the structure of an electronic device for implementing a diabetic retinopathy detection model training method according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 communicatively coupled to the at least one processor, a communication bus 12, and a communication interface 13. The electronic device may also include a computer program stored in the memory 11 and executable on the processor 10, such as a diabetic retinopathy detection model training method program.

[0149] In some embodiments, the processor 10 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and utilizes various interfaces and circuits to connect the various components of the entire electronic device. It executes or runs programs or modules stored in the memory 11 (for example, executing a diabetic retinopathy detection model training method or a diabetic retinopathy detection method, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0150] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device, such as the code of a method program for detecting diabetic retinopathy, etc., but can also be used to temporarily store data that has been output or is to be output.

[0151] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0152] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0153] Figure 9 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 9The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0154] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0155] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0156] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0157] The present application provides a computer-readable storage medium, for example, including any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). The computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the diabetic retinopathy detection model training method of the above-described embodiment.

[0158] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0159] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A diabetic retinopathy detection model training method, characterized in that: Obtain a training dataset, which includes several fundus sample images and descriptive text corresponding to the fundus sample images; the descriptive text includes a text description of detailed information such as the lesion area, vascular texture or color distribution in the fundus sample images; Constructing the network structure of the diabetic retinopathy detection model; The network of the diabetic retinopathy detection model is iteratively trained using the training dataset until the training stop condition is reached to obtain the final diabetic retinopathy detection model; In each training session, the diabetic retinopathy detection model extracts image features of fundus sample images and text features of descriptive text, calculates the contrast loss between image features and text features, and determines the network parameters of the diabetic retinopathy detection model based on the contrast loss. The steps for obtaining image features and text features are: Extracting initial image features of fundus sample images; The initial image features are projected into a preset feature space through the first projection layer to obtain image features; Process image features through multi-spectral channel attention mechanism to obtain optimized image features; Extracting initial text features of descriptive text; The initial text features are projected into the preset feature space through the second projection layer to obtain the text feature vector; After obtaining the optimized image features, the optimized image features are processed again through a fully connected layer and a sigmoid activation function to obtain the mask information output for the attention mechanism; then, the mask information output is added to the original input image features to obtain the attention output; The loss function of the diabetic retinopathy detection model is in, Represents the total number of fundus sample images in the training dataset, represents the index of the fundus sample image in the training dataset, is the scaling factor, is the marginal parameter; Represents the fundus sample image in the training dataset and fundus sample images The similarity score between Represents a sample fundus image The probability that the diabetic retinopathy classification result belongs to the correct category.

2. The diabetic retinopathy detection model training method according to claim 1, wherein: The step of processing image features through the multi-spectral channel attention mechanism includes: Decompose image features into several image sub-features; Perform a two-dimensional discrete cosine transform on the spatial dimension of the image sub-feature to obtain the frequency domain sub-feature corresponding to the image sub-feature; The weight relationship between different frequency domain sub-features is learned through the fully connected layer to obtain the weighted frequency domain sub-features; Channel attention weights are generated based on the weighted frequency domain sub-features, and the channel attention weights are multiplied with each channel of the image feature one by one to obtain the optimized image features.

3. The diabetic retinopathy detection model training method according to claim 1, wherein: The transformation formula of image sub-features from time domain to frequency domain through two-dimensional discrete cosine transform is as follows: in ， Indicates that the coordinates in the two-dimensional frequency domain are ( ) at the point, represents the vertical frequency index in the two-dimensional frequency domain, represents the horizontal frequency index in the two-dimensional frequency domain, Indicates that the input image sub-feature is in the spatial domain Row, No. The pixel value of the point corresponding to the column, Represents the vertical index of the input image sub-feature in the spatial domain, Represents the horizontal index of the input image sub-feature in the spatial domain; Represents the height of the input image sub-feature in the spatial domain ， Represents the width of the input image sub-feature in the spatial domain; No. Frequency domain sub-features corresponding to the image sub-features of the channel for: in, Indicates the The frequency domain sub-features corresponding to the image sub-features of the channel; Represents the input image sub-feature in The ℎth and ℎth rows in the channel The time domain eigenvalues ​​of this position are listed. Indicates selecting the ℎth row and the ℎth row in all channels of the image feature Eigenvalues ​​of columns; Represents the frequency index in the vertical direction in the two-dimensional frequency domain, Represents the frequency index in the horizontal direction in the two-dimensional frequency domain; The total number of channels representing image features, Indicates the index of the channel, is an integer, Is a positive integer.

4. A diabetic retinopathy detection model training device, characterized in that: An acquisition module is used to acquire a training data set, which includes a number of fundus sample images and descriptive text corresponding to the fundus sample images; the descriptive text includes a text description of detailed information such as the lesion area, vascular texture or color distribution in the fundus sample images; A building block for constructing a network structure for a diabetic retinopathy detection model; a training module for iteratively training the network of the diabetic retinopathy detection model using the training data set until a training stop condition is reached, thereby obtaining a final diabetic retinopathy detection model; during each training session, the diabetic retinopathy detection model extracts image features of fundus sample images and text features of descriptive text, calculates contrast loss between the image features and the text features, and determines network parameters of the diabetic retinopathy detection model based on the contrast loss; The steps for obtaining image features and text features are: Extracting initial image features of fundus sample images; The initial image features are projected into a preset feature space through the first projection layer to obtain image features; Process image features through multi-spectral channel attention mechanism to obtain optimized image features; Extracting initial text features of descriptive text; The initial text features are projected into the preset feature space through the second projection layer to obtain the text feature vector; After obtaining the optimized image features, the optimized image features are processed again through a fully connected layer and a sigmoid activation function to obtain the mask information output for the attention mechanism; then, the mask information output is added to the original input image features to obtain the attention output; The loss function of the diabetic retinopathy detection model is in, Represents the total number of fundus sample images in the training dataset, represents the index of the fundus sample image in the training dataset, is the scaling factor, is the marginal parameter; Represents the fundus sample image in the training dataset and fundus sample images The similarity score between Represents a sample fundus image The probability that the diabetic retinopathy classification result belongs to the correct category.

5. A method for detecting diabetic retinopathy, characterized in that: include: Acquiring a fundus image to be detected; The fundus image is input into a diabetic retinopathy detection model trained using the diabetic retinopathy detection model training method according to any one of claims 1 to 3. The diabetic retinopathy detection model analyzes and processes the fundus image and outputs a diabetic retinopathy classification result of the fundus image.

6. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively coupled to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so as to enable the at least one processor (10) to perform the method according to claim 1, 2, 3 or 5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the method according to claim 1, 2, 3 or 5 is implemented.

Images