A method, apparatus and electronic device for classifying fundus images

Abstract

This application relates to the field of image processing technology, specifically disclosing a method, apparatus, and electronic device for classifying fundus images. The method includes: acquiring a fundus color photograph and an image to be classified; extracting features from the fundus color photograph and the image to be classified; segmenting the image to be classified according to the field of view of the fundus color photograph to obtain multiple feature vectors; fusing the multiple feature vectors into a single feature vector through multiple instance learning; and classifying the fused feature vector using a fully connected layer to obtain a disease classification result. By segmenting a range similar in size to the field of view of the fundus color photograph from the feature map of the image to be classified, the field of view corresponding to each feature vector is similar to that of the fundus color photograph, thereby reducing inter-domain differences. This allows for the improvement of classification performance for wide-angle and ultra-wide-angle images using a large number of labeled fundus color photograph images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for classifying fundus images. Background Technology

[0002] Artificial intelligence technology can efficiently analyze lesions in fundus images and is suitable for large-scale screening. Traditional color fundus photography has a relatively small field of view, while wide-angle and ultra-wide-angle fundus images have a much wider field of view. Therefore, wide-angle and ultra-wide-angle fundus images are more likely to detect lesions outside the field of view of traditional color fundus photography. However, the amount of data in wide-angle and ultra-wide-angle fundus images is limited and insufficient for training deep learning models. Therefore, models are usually trained using a large number of traditional color fundus photographs and a small number of wide-angle and ultra-wide-angle fundus images.

[0003] Existing technologies typically employ domain adaptation methods, utilizing a large amount of labeled fundus color image data to improve the classification performance of wide-angle and ultra-wide-angle fundus images. However, existing technologies are based on eliminating inter-domain differences, while the differences between traditional fundus color images and wide-angle and ultra-wide-angle fundus images mainly lie in the size of the field of view and color, which are inherent differences. This results in poor image classification performance from existing domain adaptation methods. Summary of the Invention

[0004] This application provides a fundus image classification method, apparatus, and electronic device that can solve the problem of large inherent differences between fundus color photographs and wide-angle and ultra-wide-angle images.

[0005] In a first aspect, embodiments of this application provide a fundus image classification method, the method comprising:

[0006] Acquire a fundus color photograph and an image to be classified, wherein the field of view of the image to be classified is larger than the field of view of the fundus color photograph;

[0007] Extract features from the fundus photograph and the image to be classified;

[0008] The image to be classified is segmented according to the field of view of the fundus color photograph to obtain multiple feature vectors;

[0009] The multiple feature vectors are fused into a single feature vector through multi-instance learning;

[0010] The fused feature vectors are classified using a fully connected layer to obtain disease classification results.

[0011] In one alternative design, it also includes:

[0012] Input a first training image and a second training image, and use the feature extraction module to be trained to extract features. The first training image is a labeled fundus photograph, and the second training image is an image with a field of view larger than that of the fundus photograph.

[0013] The features of the first training image and the second training image are compared, and the inter-domain difference loss is calculated.

[0014] Supervised training is performed based on the annotations on the first training image to obtain the feature extraction model.

[0015] In one optional design, extracting features from the fundus photograph and the image to be classified includes:

[0016] The fundus photograph and the image to be classified are input into the feature extraction model to perform feature extraction and obtain a feature map.

[0017] In one alternative design, after acquiring fundus color photographs and images to be classified, the process also includes:

[0018] The image to be classified is cut according to a preset size, wherein the preset size is smaller than the size of the image to be classified and larger than the size of the fundus photograph;

[0019] The segmented image to be classified is input into the feature extraction model for feature extraction, resulting in a partial feature map.

[0020] The feature maps corresponding to the image to be classified are merged to obtain the feature map of the image to be classified.

[0021] In one optional design, segmenting the image to be classified according to the visual field size of the fundus photograph includes:

[0022] Extract a feature map from the image to be classified, the feature map having a size of (H, W);

[0023] Calculate the ratio K between the visual field size of the image to be classified and the visual field size of the fundus photograph;

[0024] Based on the aforementioned ratio K, and with kernel sizes of (H / K, W / K), the feature maps of the image to be classified are pooled to obtain K*K feature vectors.

[0025] In an optional design, the input first training image and the second training image are used to extract features using a feature extraction module, including:

[0026] The first training image and the second training image are input sequentially according to a preset ratio.

[0027] Secondly, embodiments of this application provide an apparatus, the apparatus comprising:

[0028] The receiving module is used to acquire fundus color photographs and images to be classified, wherein the field of view of the images to be classified is larger than the field of view of the fundus color photographs;

[0029] The feature extraction module is used to extract features from the fundus photograph and the image to be classified;

[0030] The segmentation module is used to segment the image to be classified according to the field of view of the fundus color photograph, and obtain multiple feature vectors;

[0031] The processing module is used to fuse the multiple feature vectors into a single feature vector through multi-instance learning;

[0032] The classification module is used to classify the fused feature vector using a fully connected layer to obtain the disease classification result.

[0033] Thirdly, embodiments of this application provide an electronic device, the electronic device including a memory and one or more processors; wherein, the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs some or all of the steps of the method in the first aspect or various possible implementations of the first aspect.

[0034] Fourthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform some or all of the steps of the method in the first aspect or various possible implementations of the first aspect.

[0035] This application provides a fundus image classification method, comprising: acquiring a fundus color photograph and an image to be classified, wherein the field of view of the image to be classified is larger than that of the color photograph; segmenting the image to be classified according to the field of view of the fundus color photograph to obtain a segmented image; inputting the fundus color photograph and the segmented image into a convolutional network to extract features; fusing the features of the segmented image; comparing the features of the segmented image and the fundus color photograph to ensure consistent feature distribution; and classifying the segmented image and the fundus color photograph respectively to obtain a disease classification result. By segmenting the image to be classified according to the field of view of the fundus color photograph, a unified field of view standard is achieved, thereby reducing the inter-domain differences between fundus color photographs and wide-angle and ultra-wide-angle images, thus achieving better classification results for wide-angle and ultra-wide-angle images using a large number of labeled fundus color photographs. Attached Figure Description

[0036] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart of a fundus image classification method provided in this application embodiment;

[0038] Figure 2 A schematic diagram of a fundus photograph provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of a wide-angle image provided for an embodiment of this application;

[0040] Figure 4 A schematic diagram of an ultra-wide-angle image provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the structure of a fundus image classification device provided in an embodiment of this application;

[0042] Figure 6 This is an exemplary structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0044] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that although the terms first, second, etc., may be used in the following embodiments to describe a class of objects, the objects are not limited to these terms. These terms are only used to distinguish specific objects of that class of objects.

[0045] Improving the classification performance of wide-angle and ultra-wide-angle images using labeled fundus color photographs falls under the domain adaptation field. In this application, fundus color photographs correspond to the source domain in domain adaptation, while wide-angle and ultra-wide-angle images correspond to the target domain. Domain adaptation can improve image classification performance in the target domain with the assistance of source domain data. Existing technologies achieve this by eliminating inter-domain differences, which include differences in representation, background environment, and subclasses within the same major category. However, the differences between fundus color photographs and wide-angle and ultra-wide-angle images are inherent differences, mainly stemming from the shooting equipment. These differences are generally reflected in the field of view and color of the image. Therefore, directly using existing domain adaptation methods is not effective. To address this, this application proposes a fundus image classification method that can reduce inter-domain differences and achieve better classification performance for wide-angle and ultra-wide-angle images.

[0046] The following describes a fundus image classification method according to several implementation methods.

[0047] like Figure 1 As shown, Figure 1 This paper illustrates a fundus image classification method 100 (hereinafter referred to as method 100), which includes the following steps:

[0048] Step S101: Obtain a fundus color photograph and an image to be classified, wherein the field of view of the image to be classified is larger than the field of view of the fundus color photograph.

[0049] In this embodiment, the field of view of a typical camera has fixed parameters; for example, fundus photography uses a 45-degree field of view. Figure 2 The image shown is a fundus photograph with a 45-degree field of view; other fundus photographs have a 30-degree field of view. Ordinary fundus photographs have a smaller field of view, and therefore cover a smaller area. Wide-angle and ultra-wide-angle images have a larger field of view than ordinary fundus photographs, such as... Figure 3 , Figure 4 The illustrations show wide-angle and ultra-wide-angle images; for example, the Zeiss wide-angle image has a 133-degree field of view, and the Optos ultra-wide-angle image has a 200-degree field of view. Using wide-angle and ultra-wide-angle images with a wider field of view allows lesions to be seen outside the field of view of a fundus photograph, which is more beneficial for screening early lesions. The above describes the image field of view sizes used in the embodiments of this application. Both wide-angle and ultra-wide-angle images are images to be classified. Of course, fundus photographs and other wide-angle and ultra-wide-angle images also include images with other field of view sizes, and this application does not limit this.

[0050] Step S102: Extract features from the fundus photograph and the image to be classified;

[0051] In this embodiment, a convolutional neural network is used to extract features from the fundus color photograph to be classified, resulting in an extracted feature map.

[0052] Step S103: Divide the image to be classified according to the field of view of the fundus color photograph to obtain multiple feature vectors.

[0053] In this embodiment, since the field of view sizes of fundus color photographs and images to be classified are different, the feature distributions are inconsistent. Therefore, this application uses the field of view size of fundus color photographs as the standard to segment the feature map of images to be classified, and segments out a range from wide-angle and ultra-wide-angle images that is similar to the field of view size of fundus color photographs. This makes the field of view size corresponding to the features of the extracted images to be classified similar to that of fundus color photographs, thereby obtaining feature vectors with the same field of view size, reducing the inter-domain difference between the source domain and the target domain.

[0054] Step S104: The multiple feature vectors are fused into a single feature vector through multi-instance learning.

[0055] Step S105: Use a fully connected layer to classify the fused feature vector to obtain the disease classification result.

[0056] This application provides a fundus image classification method. By extracting features from the image to be classified to obtain a feature map, a range similar in size to the field of view of ordinary fundus color images is segmented from the feature maps of wide-angle and ultra-wide-angle images. By pooling to unify the field of view standard, the difference in field of view size between fundus color images and wide-angle and ultra-wide-angle images is reduced, and feature vectors of the image to be classified with a field of view size similar to that of fundus color images are obtained. Then, the feature vectors of the same field of view size are made to have a consistent distribution, further reducing the difference between domains. Thus, a large number of labeled fundus color images can be used to improve the classification effect of wide-angle and ultra-wide-angle images.

[0057] In one alternative implementation, it further includes:

[0058] Input a first training image and a second training image, and use the feature extraction module to be trained to extract features. The first training image is a labeled fundus photograph, and the second training image is an image with a field of view larger than that of the fundus photograph.

[0059] The features of the first training image and the second training image are compared, and the inter-domain loss is calculated.

[0060] Supervised training is performed based on the annotations on the first training image to obtain the feature extraction model.

[0061] In this embodiment, since there is a large amount of data in ordinary fundus color photographs, a large number of readily available labeled fundus color photographs can be used as training data. However, the collection and labeling costs of wide-angle and ultra-wide-angle images are high, resulting in a smaller amount of data that is insufficient to directly train the model. The first training image in this application is a large number of pre-prepared labeled ordinary fundus color photographs, and the second training image is a wide-angle image and an ultra-wide-angle image. The same feature extraction module to be trained is used for feature extraction. By comparing the features of fundus color photographs with those of wide-angle and ultra-wide-angle images, the loss of inter-domain differences is calculated, thereby adjusting and reducing inter-domain differences. Through supervised training with labeled information, a feature extraction model capable of extracting features from color photographs is obtained.

[0062] In one optional implementation, extracting features from the fundus photograph and the image to be classified includes:

[0063] The fundus photograph and the image to be classified are input into the feature extraction model to perform feature extraction and obtain a feature map.

[0064] In this embodiment, after the feature classification model is trained, the feature extraction model can be applied during the image classification process to obtain feature maps.

[0065] In one optional implementation, after acquiring the fundus color photograph and the image to be classified, the method further includes:

[0066] The image to be classified is cut according to a preset size, wherein the preset size is smaller than the size of the image to be classified and larger than the size of the fundus photograph;

[0067] The segmented image to be classified is input into the feature extraction model for feature extraction, resulting in a partial feature map.

[0068] The feature maps corresponding to the image to be classified are merged to obtain the feature map of the image to be classified.

[0069] In this embodiment, when the field of view of the image to be classified is large, the size of the obtained image to be classified is also large, resulting in high computational complexity and a large amount of calculation during feature extraction. This application first segments the large wide-angle or ultra-wide-angle image to obtain smaller segmented images. Then, when performing feature extraction on the segmented image to be classified, the smaller image size reduces the computational load, thus decreasing the system's computational burden and improving the efficiency of feature extraction. Finally, the feature maps extracted from the segmented images corresponding to the same image to be classified are merged to obtain a large feature map of the entire image to be classified, shortening the overall feature extraction time.

[0070] In one optional implementation, segmenting the image to be classified according to the visual field size of the fundus photograph includes:

[0071] The size of the feature map extracted from the image to be classified is (H, W);

[0072] Calculate the ratio K between the visual field size of the image to be classified and the visual field size of the fundus photograph;

[0073] Based on the aforementioned ratio K, and with kernel sizes of (H / K, W / K), the feature maps of the image to be classified are pooled to obtain K*K feature vectors.

[0074] In this embodiment, the field of view size of the image to be classified and the field of view size of the fundus color photograph are first obtained respectively. The ratio between the two is the proportional relationship. Based on the proportional relationship, the kernel size of pooling of the image to be classified is determined, that is, the feature map is divided into K*K sub-regions, and then K*K feature vectors are obtained. The field of view size corresponding to each feature vector is similar to that of the fundus color photograph. Each feature vector corresponds to the feature of a certain position in the wide-angle or ultra-wide-angle image under the field of view size of the ordinary fundus color photograph, thus reducing the difference between domains.

[0075] In one optional implementation, the input of the first training image and the second training image, and the extraction of features using the feature extraction module, includes:

[0076] The first training image and the second training image are input sequentially according to a preset ratio.

[0077] In this embodiment, during the training process of this application, the ratio of fundus color photos to wide-angle and ultra-wide-angle images entering the network in one iteration is 1:1. Of course, other ratios can also be used to input the network. During the training process, the input ratio can be adjusted in real time based on the training situation.

[0078] In summary, the fundus image classification method of this application segmentes the feature maps of wide-angle and ultra-wide-angle images according to the field of view of the fundus color photograph and performs pooling at each position, so that the feature vector is similar to the feature range of the fundus color photograph. This achieves domain adaptation from fundus color photograph to wide-angle and ultra-wide-angle images, unifies the field of view standard, and makes the feature distribution consistent after feature vector fusion. It can be trained using a large amount of labeled fundus color photograph data, thereby improving the classification effect of wide-angle and ultra-wide-angle images.

[0079] Corresponding to method 100, this application embodiment also provides an apparatus for performing the above method.

[0080] like Figure 5 As shown, Figure 5 This diagram illustrates a fundus image classification device 500, the device comprising:

[0081] The receiving module 501 is used to acquire a fundus color photograph and an image to be classified, wherein the field of view of the image to be classified is larger than the field of view of the fundus color photograph;

[0082] Feature extraction module 502 is used to extract features from the fundus photograph and the image to be classified;

[0083] The segmentation module 503 is used to segment the image to be classified according to the field of view of the fundus color photograph to obtain multiple feature vectors;

[0084] Processing module 504 is used to fuse the multiple feature vectors into a single feature vector through multi-instance learning;

[0085] The classification module 505 is used to classify the fused feature vector using a fully connected layer to obtain the disease classification result.

[0086] It is understandable that the above division of modules / units is merely a logical functional division. In actual implementation, the functions of these modules can be integrated into hardware entities. For example, the processing module can be integrated into a processor, and the receiving module can be integrated into a transceiver. The programs and instructions that implement the functions of these modules can be maintained in memory. For example, Figure 6 An electronic device 61 is provided, which includes a processor 611, a transceiver 612, and a memory 613. The transceiver 612 is used for transmitting and receiving data and signals in method 100. The memory 613 can be used to store programs / code required by the processor 611 to execute method 100.

[0087] In a specific implementation, corresponding to the aforementioned electronic device 61, this application embodiment also provides a computer storage medium. The computer storage medium disposed in the electronic device 61 can store a program. When the program is executed, it can implement some or all of the steps in the various embodiments of method 100. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0088] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a cloud gaming control device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. A method for classifying fundus images, characterized in that, The method includes: Acquire a fundus color photograph and an image to be classified, wherein the field of view of the image to be classified is larger than that of the fundus color photograph; the image to be classified is a wide-angle or ultra-wide-angle fundus image; Extract features from the fundus photograph and the image to be classified; The image to be classified is segmented according to the field of view of the fundus color photograph to obtain multiple feature vectors; The multiple feature vectors are fused into a single feature vector through multi-instance learning; The fused feature vectors are classified using a fully connected layer to obtain disease classification results; The image to be classified is segmented according to the field of view size of the fundus photograph, including: The size of the feature map extracted from the image to be classified is (H, W); Calculate the ratio K between the visual field size of the image to be classified and the visual field size of the fundus photograph; According to the ratio K, the feature map of the image to be classified is pooled according to the kernel size (H / K, W / K) to obtain K*K feature vectors. The field of view corresponding to each feature vector is the same as that of the fundus color photograph. Each feature vector corresponds to the feature of a certain position in the wide-angle or ultra-wide-angle image under the field of view of the ordinary fundus color photograph, thus reducing the inter-domain difference.

2. The method as described in claim 1, characterized in that, Also includes: Input a first training image and a second training image, and use the feature extraction module to be trained to extract features. The first training image is a labeled fundus photograph, and the second training image is an image with a field of view larger than that of the fundus photograph. The features of the first training image and the second training image are compared, and the inter-domain difference loss is calculated. Supervised training is performed based on the annotations on the first training image to obtain the feature extraction model.

3. The method as described in claim 2, characterized in that, The extraction of features from the fundus photograph and the image to be classified includes: The fundus photograph and the image to be classified are input into the feature extraction model to perform feature extraction and obtain a feature map.

4. The method as described in claim 3, characterized in that, After obtaining fundus color images and images to be classified, the process also includes: The image to be classified is cut according to a preset size, wherein the preset size is smaller than the size of the image to be classified and larger than the size of the fundus photograph; The segmented image to be classified is input into the feature extraction model for feature extraction, resulting in a partial feature map. The feature maps corresponding to the image to be classified are merged to obtain the feature map of the image to be classified.

5. The method as described in claim 2, characterized in that, The input first training image and the second training image are used to extract features using a feature extraction module, including: The first training image and the second training image are input sequentially according to a preset ratio.

6. A fundus image classification device, characterized in that, The apparatus is used to implement the method as described in any one of claims 1-5, the apparatus comprising: The receiving module is used to acquire fundus color photographs and images to be classified, wherein the field of view of the images to be classified is larger than the field of view of the fundus color photographs; The feature extraction module is used to extract features from the fundus photograph and the image to be classified; The segmentation module is used to segment the image to be classified according to the field of view of the fundus color photograph, and obtain multiple feature vectors; The processing module is used to fuse the multiple feature vectors into a single feature vector through multi-instance learning; The classification module is used to classify the fused feature vector using a fully connected layer to obtain the disease classification result.

7. An electronic device, characterized in that, The electronic device includes a memory and one or more processors; wherein the memory is used to store computer program code, the computer program code including computer instructions; when the computer instructions are executed by the processor, the electronic device performs the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 5.

Images