Image classification methods, devices and electronic equipment
By extracting features from fundus images and using a trained model to determine the type and extent of retinal artery occlusion, the problem of slow classification speed in existing technologies is solved, and fast and accurate automatic classification is achieved.
Patent Information
- Application Number
- CN202211671964.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-12-23
AI Technical Summary
In existing technologies, it is difficult to quickly and accurately classify fundus images according to different types of retinal artery occlusion and different degrees of impact on visual function, and the process is slow and relies on professional medical personnel.
By extracting image features from fundus images to be classified, a pre-trained retinal artery occlusion classification model is used to determine the type of retinal artery occlusion, and the degree of impact on visual function is determined by the location and area of the lesion, thus achieving automatic classification.
It enables rapid and accurate automatic classification of fundus images according to different types of retinal artery occlusion and varying degrees of impact on visual function, reducing manual intervention and improving classification speed and accuracy.
Smart Images

Figure CN115984206B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image classification method, apparatus and electronic device. Background Technology
[0002] When the perfusion of the retinal arteries cannot meet the metabolic needs of the inner retina, acute ischemic lesions, namely retinal artery occlusion, occur, affecting visual function. Current techniques involve analyzing fundus images to explore the pathogenesis of retinal artery occlusion and develop more scientific and effective treatments.
[0003] To obtain more accurate results, analysis of fundus images from retinal artery occlusions of the same type and with similar degrees of impact on visual function is necessary. Therefore, before analyzing fundus images used as analytical samples, it is crucial to classify the images according to different types of retinal artery occlusions and their varying degrees of impact on visual function. How to quickly and accurately classify fundus images according to different types of retinal artery occlusions and their varying degrees of impact on visual function is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] Based on the above needs, this application proposes an image classification method, apparatus, and electronic device. The method can quickly and accurately classify fundus images according to different types of retinal artery occlusion and different degrees of impact on visual function.
[0005] The technical solution proposed in this application is as follows:
[0006] On the one hand, this application provides an image classification method, including:
[0007] Extract image features from the fundus image to be classified;
[0008] Based on the image features of the fundus image to be classified, the type of retinal artery occlusion to which the fundus image to be classified belongs is determined; the type of retinal artery occlusion includes at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion.
[0009] If the retinal artery occlusion type of the fundus image to be classified is central retinal artery occlusion or branch retinal artery occlusion, then the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function is determined based on the image features of the fundus image to be classified.
[0010] Furthermore, in the image classification method described above, the step of extracting image features from the fundus image to be classified, and determining the retinal artery occlusion type of the fundus image to be classified based on the image features, includes:
[0011] The fundus image to be classified is input into a pre-trained retinal artery occlusion classification model, so that the retinal artery occlusion classification model extracts the image features of the fundus image to be classified, and determines the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features of the fundus image to be classified.
[0012] Furthermore, in the image classification method described above, determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the image features of the fundus image to be classified includes:
[0013] Based on the image features of the fundus image to be classified, the lesion region of the fundus image to be classified is determined;
[0014] Based on the lesion area, determine the degree of impact on visual function caused by the central retinal artery occlusion or the branch retinal artery occlusion.
[0015] Furthermore, in the image classification method described above, determining the lesion region of the fundus image to be classified based on its image features includes:
[0016] Based on the image features of the fundus image to be classified, a heat map of the abnormal retinal artery occlusion region of the fundus image to be classified is generated;
[0017] The lesion area of the fundus image to be classified is determined from the heat map of the abnormal retinal artery occlusion area in the fundus image to be classified.
[0018] Furthermore, in the image classification method described above, the step of generating a heatmap of the retinal artery occlusion abnormality region of the fundus image to be classified based on the image features of the fundus image to be classified includes:
[0019] Backpropagation is performed based on the classification results output by the pre-trained retinal artery occlusion classification model to obtain the gradient value of each channel feature map of the last convolutional layer in the retinal artery occlusion classification model relative to the classification result.
[0020] Calculate the product of the gradient value and the corresponding channel feature map to obtain a heatmap-like image;
[0021] The heatmap is activated by an activation function, and then the activated heatmap is superimposed on the fundus image to be classified to obtain a heatmap of the abnormal retinal artery occlusion region in the fundus image to be classified.
[0022] Furthermore, in the image classification method described above, the step of overlaying the activated heatmap onto the fundus image to be classified includes:
[0023] The activated heatmap is upsampled to the same size as the fundus image to be classified, and then superimposed on the fundus image to be classified.
[0024] Furthermore, in the image classification method described above, determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the lesion area includes:
[0025] The degree of impact on visual function caused by the occlusion of the central retinal artery or the occlusion of the branch retinal arteries is determined based on the location and area of the lesion.
[0026] Furthermore, in the image classification method described above, determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the location and area of the lesion region includes:
[0027] The distance between the location of the lesion area and the location of the macula in the fundus image to be classified is inversely proportional to the degree of influence of the central retinal artery occlusion or the branch retinal artery occlusion on visual function; the area of the lesion area is directly proportional to the degree of influence of the central retinal artery occlusion or the branch retinal artery occlusion on visual function.
[0028] On the other hand, this application also provides an image classification apparatus, comprising:
[0029] The extraction module is used to extract image features from the fundus image to be classified;
[0030] The first determining module is used to determine the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features of the fundus image to be classified; the type of retinal artery occlusion includes at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion.
[0031] The second determining module is used to determine the degree of impact of the central retinal artery occlusion or the branch retinal artery occlusion on visual function based on the image features of the fundus image to be classified if the retinal artery occlusion type to which the fundus image to be classified belongs is the central retinal artery occlusion or the branch retinal artery occlusion.
[0032] On the other hand, this application also provides an electronic device, including:
[0033] Memory and processor;
[0034] The memory is used to store programs;
[0035] The processor is configured to implement any of the above methods by running a program in the memory.
[0036] The image classification method proposed in this application can extract image features from fundus images to be classified, and determine the type of retinal artery occlusion to which the fundus images belong based on the image features. If the type of retinal artery occlusion to which the fundus images belong is central retinal artery occlusion or branch retinal artery occlusion, then the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function is determined based on the image features of the fundus images to be classified. This achieves the goal of quickly and accurately classifying fundus images automatically according to different types of retinal artery occlusion and different degrees of impact on visual function. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0038] Figure 1 This is a flowchart illustrating an image classification method provided in an embodiment of this application.
[0039] Figure 2 This is a fundus image of central retinal artery occlusion provided in an embodiment of this application.
[0040] Figure 3 This is a fundus image of retinal branch artery occlusion provided in an embodiment of this application.
[0041] Figure 4 This is a flowchart illustrating the process of determining the degree of impact on visual function provided in an embodiment of this application.
[0042] Figure 5 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application.
[0043] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0044] Application Overview
[0045] The technical solution of this application is applicable to the application scenario of classifying fundus images. By adopting the technical solution of this application, fundus images can be automatically classified according to different types of retinal artery occlusion and different degrees of impact on visual function.
[0046] Retinal artery occlusion is more common in middle-aged and elderly people, especially those with risk factors for arteriosclerosis such as hypertension, diabetes, hyperlipidemia, and high homocysteine levels. Young people with irregular lifestyles and those who smoke are also prone to retinal artery occlusion.
[0047] Retinal artery occlusion is a relatively dangerous condition. Because the retinal arteries supply nutrients to the retina, occlusion leads to ischemic and hypoxic changes in the retina. Over time, this can cause retinal necrosis or optic nerve atrophy, resulting in irreversible vision loss. Therefore, timely diagnosis and treatment of retinal artery occlusion, including the specific type of occlusion, can prevent further deterioration and delays in treatment, thus increasing the chances of vision recovery.
[0048] In existing technologies, in order to explore the pathogenesis of retinal artery occlusion and develop more scientific and effective treatment methods, the current approach involves recruiting a large number of volunteers, collecting their fundus images as analysis samples, and then analyzing these samples to achieve the aforementioned goal of exploring the pathogenesis of retinal artery occlusion and developing more scientific and effective treatment methods.
[0049] More accurate results can only be obtained by analyzing fundus images of retinal artery occlusions of the same type and with similar degrees of impact on visual function. Therefore, before analyzing retinal artery occlusions, the samples need to be classified according to different types of retinal artery occlusions and their varying degrees of impact on visual function. Currently, the classification of samples is performed by professional medical personnel, which not only requires a high level of expertise but is also slow.
[0050] Based on this, this application proposes an image classification method, apparatus, and electronic device. This technical solution can automatically determine the type of retinal artery occlusion in the fundus image and the degree of impact of the type of retinal artery occlusion on visual function based on the image features of the fundus image, thereby achieving the purpose of automatically classifying fundus images.
[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0052] Exemplary methods
[0053] This application proposes an image classification method, which can be executed by an electronic device. This electronic device can be any device with data and instruction processing capabilities, such as a computer, a smart terminal, or a server. See also... Figure 1 As shown, the method includes:
[0054] S101. Extract image features from the fundus images to be classified.
[0055] The fundus is a collective term for the tissues at the back of the eyeball, including structures such as the optic disc, macula, and blood vessels. Images obtained by taking pictures of the fundus using a fundus camera, optical coherence tomography (OCT) scanner, or fluorescein angiography machine are called fundus images. Through these images, doctors can clearly see the tissue structures of the fundus, analyze whether there are any abnormalities, and finally, based on a detailed screening and diagnosis, provide a specific treatment plan.
[0056] In the embodiments of this application, after obtaining the fundus image, a preprocessing operation can be performed on the fundus image to obtain a preprocessed fundus image to be classified. The preprocessing operation includes at least one of the following: region of interest extraction, denoising, normalization, and enhancement.
[0057] Region of Interest (ROI) extraction involves extracting the relevant regions from a fundus image, removing invalid regions such as the background, and reducing interference from non-fundus areas. This can be achieved by first performing channel separation on the color fundus image, where the background region appears dark in the red channel. Thresholding segmentation is used to segment the red channel image, and candidate ROIs are obtained using the average gray value and area proportion of the dark regions. Then, morphological and positional features of the image are used to further refine the candidate ROIs, resulting in the final ROI region.
[0058] Denoising is performed to reduce noise interference during shooting and camera imaging. It is achieved by using low-pass filtering, which converts the image from the spatial domain to the frequency domain and removes low-frequency components, thereby achieving the purpose of noise reduction.
[0059] Normalization processes use mean calibration to adjust the color, brightness, and size of each image to a uniform range, thereby reducing differences between images and minimizing deviations in brightness and color. Brightness normalization involves converting the color space to LAB space, performing mean calibration on the L space, and then converting it back to RGB space to achieve brightness normalization. Size normalization normalizes the image size to 384×384.
[0060] The enhancement process involves using the contrast-limited adaptive histogram equalization (CLAHE) algorithm within the ROI region to enhance the image. This algorithm divides the image into different small blocks, performs gray-level-limited enhancement on each block, and interpolates gray levels between adjacent blocks to eliminate gray-level differences between block boundaries.
[0061] For example, a denoising process can be performed first to obtain a denoised fundus image. Then, a normalization process can be performed on the denoised fundus image to obtain a normalized fundus image. Finally, an enhancement process can be performed on the normalized fundus image to obtain an enhanced fundus image. Alternatively, only denoising, normalization, or enhancement processes can be performed on the fundus image. That is, the above preprocessing operations can be selected according to actual needs, and this application does not impose specific limitations.
[0062] In this embodiment, after acquiring the fundus image to be classified, feature extraction is performed on the fundus image. For example, a feature extraction model can be pre-trained to extract image features from the fundus image to be classified.
[0063] The training process of the feature extraction model is as follows: A neural network model can be selected as the base model for the feature extraction model. For example, ResNet, VGG, or other neural network models can be used; this embodiment is not limited to any particular model. A large number of preprocessed fundus images are selected as training samples, and the image features corresponding to the training samples are used as training labels to train the feature extraction model. During the training process, the training samples are input into the feature extraction model to obtain the output results. Based on the output results and training labels, the loss value of the feature extraction model is determined, and the parameters of the feature extraction model are adjusted in the direction of reducing the loss value. The above training process is repeated until the loss value of the feature extraction model is less than a set value, at which point the feature extraction model training is complete. It should be noted that the above set value can be set according to the actual situation; this embodiment is not limited to any particular set value.
[0064] The fundus image to be classified is input into the above feature extraction model to obtain the image features of the fundus image to be classified output by the feature extraction model.
[0065] S102. Based on the image features of the fundus image to be classified, determine the type of retinal artery occlusion to which the fundus image to be classified belongs.
[0066] Types of retinal artery occlusion include at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion.
[0067] Central retinal artery occlusion refers to a blockage at the beginning of the artery's path into the retina, preventing nutrients from reaching the retina. This can cause sudden blindness, a rapid decline in vision to the point of complete blindness, or even loss of light perception. Figure 2 The image shown is a fundus image of a central retinal artery occlusion. Retinal branch artery occlusion refers to blockage of the branch segments of the artery after it enters the retina, which may lead to varying degrees of visual impairment and visual field defects, manifesting as varying degrees of visual loss and the appearance of shadows obstructing the view in front of the eyes. Figure 3 The image shown is a fundus image of a central retinal artery occlusion. "No retinal artery occlusion" means that no obvious blockage of the retinal arteries is found in the fundus image; such fundus images generally indicate that the retinal arteries are not blocked.
[0068] Fundus images of different types of retinal artery occlusion possess different image features. For example, in an embodiment of this application, a large number of fundus images of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion can be pre-extracted, providing image features. Then, the image features of the fundus images of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion are compared with the image features of the fundus image to be classified based on the above steps. The retinal artery occlusion type with the highest similarity is determined as the retinal artery occlusion type of the fundus image to be classified.
[0069] Another example is that a feature classification model can be pre-trained, and the image features of the fundus image to be classified can be input into the feature classification model so that the feature classification model can determine the type of retinal artery occlusion of the fundus image to be classified based on the image features of the fundus image to be classified.
[0070] During training, the feature classification model uses image features from fundus images as training samples and the retinal artery occlusion type corresponding to the training samples as labels. The specific training process of the feature classification model is the same as that of the feature extraction model in the above embodiments. Those skilled in the art can refer to the description of the training process of the feature extraction model in the above embodiments, and it will not be repeated here.
[0071] Based on the above steps, the analyzed samples can be categorized into three types: central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion. If a study is needed focusing on central retinal artery occlusion, fundus images of occlusion types categorized as central retinal artery occlusion can be quickly extracted from the analyzed samples. Similarly, if a study is needed focusing on branch retinal artery occlusion, fundus images of occlusion types categorized as branch retinal artery occlusion can be quickly extracted from the analyzed samples. Furthermore, if a study is needed focusing on both central and branch retinal artery occlusion, fundus images of occlusion types categorized as central retinal artery occlusion and branch retinal artery occlusion, respectively, can be quickly extracted from the analyzed samples.
[0072] S103. If the retinal artery occlusion type of the fundus image to be classified is central retinal artery occlusion or retinal branch artery occlusion, then the degree of impact of central retinal artery occlusion or retinal branch artery occlusion on visual function is determined based on the image characteristics of the fundus image to be classified.
[0073] Furthermore, if the retinal artery occlusion type of the fundus image to be classified is determined to be "no retinal artery occlusion," it means that there is no obvious retinal artery occlusion in the fundus image to be classified, and it is not necessary to further confirm the degree of impact of such images on visual function. If the retinal artery occlusion type of the fundus image to be classified is determined to be central retinal artery occlusion or branch retinal artery occlusion, it is necessary to further determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
[0074] In this embodiment, the lesion region in the fundus image to be classified can be determined based on the image features extracted from the fundus image to be classified in the steps of the above embodiments. Then, the degree of impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion can be determined based on the location of the lesion region. Specifically, the degree of impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion can be determined based on the size and location of the lesion region.
[0075] For example, Figure 2 The image shown is a fundus image of a central artery occlusion. Figure 2 Region S1 in the diagram represents the lesion area of central artery occlusion; Figure 3 The image shown is a fundus image of a branch artery blockage. Figure 3 Region S2 in the diagram represents the lesion area of branch artery blockage. Based on... Figure 2 and Figure 3As can be seen in fundus images, the lesion area of central retinal artery occlusion appears as grayish-white opacity and edema in the posterior pole of the retina, with the macula appearing relatively red. The lesion area of branch artery occlusion appears as regional superficial opacity or whitening (grayish-white edema) in the affected retina (most commonly in the areas supplied by branches of the retinal arteries in the inferior and superior temporal regions), and may be accompanied by cotton wool spots. Based on these findings, computer vision technology can be used to extract the lesion area, and then, according to the lesion area, the degree of impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion can be determined.
[0076] Furthermore, based on the above steps, after classifying the analyzed samples according to the required types, the analyzed samples of central retinal artery occlusion and / or branch retinal artery occlusion types can be further classified according to the degree of impact on visual function.
[0077] In the above embodiments, image features of the fundus images to be classified are extracted. Based on the image features of the fundus images to be classified, the type of retinal artery occlusion to which the fundus images to be classified belong is determined. If the type of retinal artery occlusion to which the fundus images to be classified belong is central retinal artery occlusion or branch retinal artery occlusion, then based on the image features of the fundus images to be classified, the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function is determined. This achieves the goal of quickly and accurately classifying fundus images according to different types of retinal artery occlusion and different degrees of impact on visual function.
[0078] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, which involve extracting image features from the fundus image to be classified and determining the retinal artery occlusion type of the fundus image to be classified based on the image features, may specifically include the following steps:
[0079] The fundus image to be classified is input into a pre-trained retinal artery occlusion classification model, so that the retinal artery occlusion classification model can extract the image features of the fundus image to be classified, and determine the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features.
[0080] The aforementioned retinal artery occlusion classification model is a feature extraction classification model. That is, based on the aforementioned retinal artery occlusion classification model, image features of the fundus image to be classified can be extracted from the fundus image to be classified, and then the type of retinal artery occlusion to which the fundus image to be classified is determined based on the image features of the fundus image to be classified.
[0081] The training process of the retinal artery occlusion classification model is as follows: A neural network model can be selected as the base model for the feature extraction model. For example, ResNet50 can be used, but this embodiment is not limited to it. A large number of fundus images are selected as training samples, and the retinal artery occlusion type corresponding to the training samples is used as the training label to train the retinal artery occlusion classification model. During the training process, the training samples are input into the retinal artery occlusion classification model to obtain the output result of the retinal artery occlusion classification model. Based on the output result of the retinal artery occlusion classification model and the training label, the loss value of the retinal artery occlusion classification model is determined. The parameters of the retinal artery occlusion classification model are adjusted in the direction of reducing the loss value of the retinal artery occlusion classification model. The above training process is repeated until the loss value of the retinal artery occlusion classification model is less than the set value, and the feature extraction model training is completed. It should be noted that the above set value can be set according to the actual situation, and this embodiment is not limited to it.
[0082] In the above embodiments, the retinal artery occlusion classification model can quickly and accurately classify the fundus images to be classified and determine the type of retinal artery occlusion to which the fundus images to be classified belong.
[0083] As an optional implementation method, such as Figure 4 As shown in another embodiment of this application, the steps of the above embodiments are based on image features of the fundus image to be classified to determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function, which may specifically include the following steps:
[0084] S401. Based on the image features of the fundus image to be classified, determine the lesion area of the fundus image to be classified.
[0085] In the embodiments of this application, the lesion area of the fundus image to be classified can be determined based on the image features of the fundus image to be classified.
[0086] For example, a lesion region extraction model can be pre-trained. Image features of the fundus image to be classified are input into the lesion region extraction model to obtain the lesion region image output by the model. During training, the lesion region extraction model uses image features of the fundus image as training samples and the corresponding lesion regions as labels. The specific training process of the lesion region extraction model is the same as the training process of the feature extraction model in the above embodiments. Those skilled in the art can refer to the description of the training process of the feature extraction model in the above embodiments, and it will not be repeated here.
[0087] In addition, the lesion area of the fundus image to be classified can be determined based on computer vision, as described in the above embodiments. This embodiment does not limit this method.
[0088] S402. Determine the degree of impact on visual function of central retinal artery occlusion or branch retinal artery occlusion based on the lesion area.
[0089] Once the lesion area is identified, its size and location can be measured. Based on the size and location of the lesion area, the extent to which central retinal artery occlusion or branch retinal artery occlusion affects visual function can be determined.
[0090] In the above embodiments, if the retinal artery occlusion type of the fundus image to be classified is central retinal artery occlusion or branch retinal artery occlusion, the lesion area of the fundus image to be classified can be effectively determined based on the image features of the fundus image to be classified, thereby determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
[0091] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, based on the image features of the fundus image to be classified, determine the lesion region of the fundus image to be classified, which may specifically include the following steps:
[0092] Based on the image features of the fundus image to be classified, a heat map of the abnormal retinal artery occlusion area of the fundus image to be classified is generated; the lesion area of the fundus image to be classified is determined from the heat map of the abnormal retinal artery occlusion area of the fundus image to be classified.
[0093] In this embodiment, a heatmap of the lesion region in the fundus image to be classified can be constructed based on the image features of the fundus image to be classified. The heatmap can clearly show the response degree of the retinal artery occlusion classification model to the lesion region. Therefore, the lesion region of the fundus image to be classified can be determined according to the color distribution and position distribution of the heatmap.
[0094] In this embodiment, areas with higher response levels in the heatmap can be identified as lesion areas in the fundus image to be classified. For example, the areas covered by red regions in the heatmap can be identified as lesion areas in the fundus image to be classified, or the areas covered by red, yellow, green, and blue regions in the heatmap can be identified as lesion areas in the fundus image to be classified, etc., which will not be elaborated in this embodiment.
[0095] Specifically, a heatmap of abnormal retinal artery occlusion areas in the fundus image to be classified can be generated based on the following steps:
[0096] Backpropagation is performed based on the classification results output by the pre-trained retinal artery occlusion classification model. The gradient value of each channel feature map in the last convolutional layer of the retinal artery occlusion classification model relative to the classification result is obtained. The product of the gradient value and the corresponding channel feature map is calculated to obtain a class heatmap. The class heatmap is activated using an activation function, and then superimposed on the fundus image to be classified to obtain a heatmap of the abnormal retinal artery occlusion region in the fundus image to be classified.
[0097] In this embodiment, backpropagation begins from the node containing the classification result output by the retinal artery occlusion classification model. Specifically, it iterates through all channels of the last convolutional layer of the retinal artery occlusion classification model and performs global average pooling. After softmax, the classification result is obtained. The gradient value of each channel feature map in the last convolutional layer of the retinal artery occlusion classification model relative to the classification result is calculated. The product of the gradient value and the corresponding channel feature map is calculated to obtain a class heatmap, which represents the importance of each channel feature map to the model's classification decision. The gradient value corresponds to the channel feature map that participated in the gradient calculation when generating the gradient value. After activation function and normalization processing, an ordered class mapping map, i.e., the activated class heatmap, is generated. The class mapping map is superimposed on the fundus image to be classified to obtain a heatmap of the abnormal region of retinal artery occlusion in the fundus image to be classified.
[0098] In the above embodiments, the generated heatmap can visualize the location of lesions, facilitating a more intuitive observation of the characteristics of abnormal arterial blockage areas. When the edge regions are not clearly defined, the heatmap can more accurately determine the lesion areas in the fundus images to be classified.
[0099] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments overlay the activated heatmap onto the fundus image to be classified, which may specifically include the following steps:
[0100] The activated heatmap is upsampled to the same size as the fundus image to be classified, and then superimposed on the fundus image to be classified.
[0101] Specifically, the class heatmap after activation can be upsampled according to the size of the fundus image to be classified. The generated class mapping map is upsampled to the size of the fundus image to be classified and superimposed on the fundus image to be classified to obtain a visual heatmap of the fundus image to be classified.
[0102] In the above embodiments, by upsampling the activated heatmap, the area of the activated heatmap can be increased so that the activated heatmap can be superimposed on the fundus image to be classified.
[0103] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, based on the lesion area, determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function, specifically including the following steps:
[0104] The extent of the impact on visual function by central retinal artery occlusion or branch retinal artery occlusion is determined based on the location and size of the lesion area.
[0105] The distance between the lesion area and the macula in the fundus image to be classified is inversely proportional to the degree of impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion. In other words, the closer the lesion area is to the macula in the fundus image, the greater the impact of central retinal artery occlusion or branch retinal artery occlusion on visual function; conversely, the farther the lesion area is from the macula in the fundus image, the lower the impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
[0106] The size of the lesion area is directly proportional to the degree of impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion. In other words, the larger the lesion area, the greater the impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion; the smaller the lesion area, the less the impact on visual function caused by central retinal artery occlusion or branch retinal artery occlusion.
[0107] Visual function includes visual acuity and visual field. Based on the color and location distribution of retinal artery occlusion lesions in a thermal map, the degree of impact on visual acuity can be determined. The red areas in the thermal map have the greatest impact; that is, the larger the red area, the greater the impact on visual acuity from central or branch retinal artery occlusion. The closer the red area is to the macula in the fundus image being classified, the greater the impact on visual acuity from central or branch retinal artery occlusion. If the lesion is located around the macula, the patient may lose part of their visual field, resulting in partial blindness. The thermal map's effect on visual acuity is determined by the fact that red areas have the greatest impact, while surrounding colors have the least impact.
[0108] In the above embodiments, based on the location and area of the lesion region, the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function can be determined, thereby achieving the purpose of further classifying according to the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
[0109] Furthermore, the technical solutions described above in this application can also assist in the diagnosis of retinal artery occlusion. Specifically, if central retinal artery occlusion occurs, the patient will experience painless, sudden blindness. This blindness is painless, and the patient will suddenly experience a decrease in vision or even a complete loss of light perception without realizing it. If it is a branch artery occlusion, the area supplied by the artery will be blocked, and the corresponding area will be invisible. Often, the patient will feel a sudden loss of vision or a decrease in visual acuity. In some cases, if the area of arterial occlusion is small, the patient may not even feel it.
[0110] When a patient experiences a sudden decrease in vision or blindness, after obtaining the patient's fundus image, the technical solution described in the above embodiments can be used to determine whether the patient has retinal artery occlusion. If the patient is determined to have central retinal artery occlusion or retinal branch artery occlusion, the extent of the impact of central retinal artery occlusion or retinal branch artery occlusion on the patient's visual function can be further determined. This helps doctors to diagnose the cause of the patient's blindness more accurately and comprehensively, reduces the workload of doctors, improves the efficiency of medical services, assists doctors in formulating treatment plans, and effectively addresses the pain points of primary care physicians. Moreover, by accurately, efficiently, and quantitatively differentiating arterial occlusion lesions, it can also save doctors' diagnostic time, enabling patients to receive timely treatment and avoiding further deterioration of the patient's condition due to delayed diagnosis, thus preventing the loss of the optimal treatment time.
[0111] Exemplary device
[0112] Corresponding to the image classification method described above, this application also discloses an image classification device, see [link to relevant documentation]. Figure 5 As shown, the device includes:
[0113] Extraction module 100 is used to extract image features from fundus images to be classified;
[0114] The first determining module 110 is used to determine the type of retinal artery occlusion of the fundus image to be classified based on the image features of the fundus image to be classified; the type of retinal artery occlusion includes at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion.
[0115] The second determining module 120 is used to determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the image features of the fundus image to be classified if the retinal artery occlusion type of the fundus image to be classified is central retinal artery occlusion or branch retinal artery occlusion.
[0116] As an optional implementation, another embodiment of this application discloses that, in the above embodiments, the extraction module 100 extracts image features from the fundus image to be classified, and the first determining module 110, based on the image features of the fundus image to be classified, determines the retinal artery occlusion type of the fundus image to be classified, specifically for:
[0117] The fundus image to be classified is input into a pre-trained retinal artery occlusion classification model, so that the retinal artery occlusion classification model can extract the image features of the fundus image to be classified, and determine the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features.
[0118] As an optional implementation, another embodiment of this application discloses the second determining module 120 of the above embodiments, which includes:
[0119] The first determining unit is used to determine the lesion region of the fundus image to be classified based on the image features of the fundus image to be classified.
[0120] The second determining unit is used to determine the degree of impact on visual function of central retinal artery occlusion or branch retinal artery occlusion based on the lesion area.
[0121] As an optional implementation, another embodiment of this application discloses that when the first determining unit of the above embodiments determines the lesion region of the fundus image to be classified based on the image features of the fundus image to be classified, it is specifically used for:
[0122] Based on the image features of the fundus image to be classified, a heat map of the abnormal retinal artery occlusion area of the fundus image to be classified is generated; the lesion area of the fundus image to be classified is determined from the heat map of the abnormal retinal artery occlusion area of the fundus image to be classified.
[0123] As an optional implementation, another embodiment of this application discloses that when the first determining unit of the above embodiments generates a heatmap of the abnormal retinal artery occlusion region of the fundus image to be classified based on the image features of the fundus image to be classified, it is specifically used for:
[0124] Backpropagation is performed based on the classification results output by the pre-trained retinal artery occlusion classification model to obtain the gradient value of each channel feature map in the last convolutional layer of the retinal artery occlusion classification model relative to the classification result. The product of the gradient value and the corresponding channel feature map is calculated to obtain the class heatmap. The class heatmap is activated by an activation function, and the activated class heatmap is superimposed on the fundus image to be classified to obtain the heatmap of the abnormal region of retinal artery occlusion in the fundus image to be classified.
[0125] As an optional implementation, another embodiment of this application discloses that when the first determining unit of the above embodiments overlays the activated heatmap onto the fundus image to be classified to obtain a heatmap of the abnormal retinal artery occlusion region in the fundus image to be classified, it is specifically used for:
[0126] The activated heatmap is upsampled to the same size as the fundus image to be classified, and then superimposed on the fundus image to be classified.
[0127] As an optional implementation, another embodiment of this application discloses that when the second determining unit of the above embodiments determines the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the lesion area, it is specifically used for:
[0128] The extent of the impact on visual function by central retinal artery occlusion or branch retinal artery occlusion is determined based on the location and size of the lesion area.
[0129] As an optional implementation, another embodiment of this application discloses that when the second determining unit of the above embodiments determines the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the location and area of the lesion region, it is specifically used for:
[0130] The distance between the location of the lesion area and the location of the macula in the fundus image to be classified is inversely proportional to the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function; the area of the lesion area is directly proportional to the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
[0131] Exemplary electronic devices, computer program products, and storage media
[0132] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 6 As shown, the electronic device includes:
[0133] Memory 200 and processor 210;
[0134] The memory 200 is connected to the processor 210 and is used to store programs;
[0135] The processor 210 is configured to implement the image classification method disclosed in any of the above embodiments by running a program stored in the memory 200.
[0136] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.
[0137] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:
[0138] A bus can include a pathway for transmitting information between various components of a computer system.
[0139] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0140] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.
[0141] The memory 200 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.
[0142] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.
[0143] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.
[0144] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.
[0145] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the image classification method provided in the above embodiments of this application.
[0146] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by processor 210, cause processor 210 to perform the various steps of the image classification method provided in the above embodiments.
[0147] Computer program products can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0148] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions, which, when executed by a processor, cause the processor 210 to perform the various steps of the image classification method provided in the above embodiments.
[0149] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0150] Specifically, the specific working content of each part of the aforementioned electronic device, computer program product, and storage medium, as well as the specific processing content of the computer program product or the computer program on the aforementioned storage medium when run by the processor, can all be found in the various embodiments of the aforementioned image classification method, and will not be repeated here.
[0151] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0152] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0153] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.
[0154] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.
[0155] It should be understood, in the several embodiments provided in this application, that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative; for instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules. It may be electrical, mechanical, or other forms.
[0156] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.
[0157] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.
[0158] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0159] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0160] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0161] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image classification method, characterized in that, include: Extract image features from the fundus image to be classified; Based on the image features of the fundus image to be classified, the type of retinal artery occlusion to which the fundus image to be classified belongs is determined; the type of retinal artery occlusion includes at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion. The process of determining the retinal artery occlusion type of the fundus image to be classified based on its image features includes: calculating the similarity between the image features of fundus images with central retinal artery occlusion, fundus images with branch retinal artery occlusion, and fundus images without retinal artery occlusion and the image features of the fundus image to be classified; and determining the retinal artery occlusion type with the highest similarity as the retinal artery occlusion type of the fundus image to be classified. If the retinal artery occlusion type of the fundus image to be classified is the central retinal artery occlusion or the branch retinal artery occlusion, then the degree of impact of the central retinal artery occlusion or the branch retinal artery occlusion on visual function is determined based on the image features of the fundus image to be classified. The step of determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the image features of the fundus image to be classified includes: determining the lesion area of the fundus image to be classified based on the image features of the fundus image to be classified; and determining the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the lesion area. The step of determining the degree of impact of the central retinal artery occlusion or the branch retinal artery occlusion on visual function based on the lesion area includes: determining the degree of impact of the central retinal artery occlusion or the branch retinal artery occlusion on visual function based on the location and area of the lesion area; The distance between the location of the lesion area and the location of the macula in the fundus image to be classified is inversely proportional to the degree of influence of the central retinal artery occlusion or the branch retinal artery occlusion on visual function; the area of the lesion area is directly proportional to the degree of influence of the central retinal artery occlusion or the branch retinal artery occlusion on visual function.
2. The image classification method according to claim 1, characterized in that, The step of extracting image features from the fundus image to be classified, and determining the retinal artery occlusion type of the fundus image to be classified based on the image features, includes: The fundus image to be classified is input into a pre-trained retinal artery occlusion classification model, so that the retinal artery occlusion classification model extracts the image features of the fundus image to be classified, and determines the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features of the fundus image to be classified.
3. The image classification method according to claim 1, characterized in that, The step of determining the lesion region of the fundus image to be classified based on its image features includes: Based on the image features of the fundus image to be classified, a heat map of the abnormal retinal artery occlusion region of the fundus image to be classified is generated; The lesion area of the fundus image to be classified is determined from the heat map of the abnormal retinal artery occlusion area in the fundus image to be classified.
4. The image classification method according to claim 3, characterized in that, The step of generating a heatmap of the abnormal retinal artery occlusion region in the fundus image to be classified based on the image features of the fundus image to be classified includes: Backpropagation is performed based on the classification results output by the pre-trained retinal artery occlusion classification model to obtain the gradient value of each channel feature map of the last convolutional layer in the retinal artery occlusion classification model relative to the classification result. Calculate the product of the gradient value and the corresponding channel feature map to obtain a heatmap-like image; The heatmap is activated by an activation function, and then the activated heatmap is superimposed on the fundus image to be classified to obtain a heatmap of the abnormal retinal artery occlusion region in the fundus image to be classified.
5. The image classification method according to claim 4, characterized in that, The step of overlaying the activated heatmap onto the fundus image to be classified includes: The activated heatmap is upsampled to the same size as the fundus image to be classified, and then superimposed on the fundus image to be classified.
6. An image classification device, characterized in that, include: The extraction module is used to extract image features from the fundus image to be classified; The first determining module is used to determine the type of retinal artery occlusion to which the fundus image to be classified belongs based on the image features of the fundus image to be classified; the type of retinal artery occlusion includes at least one of central retinal artery occlusion, branch retinal artery occlusion, and no retinal artery occlusion. The process of determining the retinal artery occlusion type of the fundus image to be classified based on its image features includes: calculating the similarity between the image features of fundus images with central retinal artery occlusion, fundus images with branch retinal artery occlusion, and fundus images without retinal artery occlusion and the image features of the fundus image to be classified; and determining the retinal artery occlusion type with the highest similarity as the retinal artery occlusion type of the fundus image to be classified. The second determining module is used to determine the degree of impact of the central retinal artery occlusion or the branch retinal artery occlusion on visual function based on the image features of the fundus image to be classified if the retinal artery occlusion type of the fundus image to be classified is the central retinal artery occlusion or the branch retinal artery occlusion. The second determining module includes: a first determining unit, used to determine the lesion area of the fundus image to be classified based on the image features of the fundus image to be classified; and a second determining unit, used to determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the lesion area. Specifically, when the second determining unit determines the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the lesion area, it is used to: determine the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function based on the location and area of the lesion area. Among them, the distance between the location of the lesion area and the location of the macula in the fundus image to be classified is inversely proportional to the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function; the area of the lesion area is directly proportional to the degree of impact of central retinal artery occlusion or branch retinal artery occlusion on visual function.
7. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to implement the method as described in any one of claims 1 to 5 by running a program in the memory.
Citation Information
Patent Citations
Method of fully automatically classifying and partitioning branch retinal artery obstruction based on three-dimensional OCT image
CN105551038A
Fundus lesion analysis method and device
CN109993731A
Multi-mode multi-disease long-tail distribution ophthalmic disease classification model training method and device
CN113011485A
Eye fundus image retinal artery occlusion detection method and system based on deep learning
CN114098630A
New coronal pneumonia focus segmentation method based on Grad-CAM
CN114972272A
