Multi-modal eye fundus image classification method and device and computer readable medium

Abstract

The invention provides a multi-modal eye fundus image classification method and device and a computer readable medium, and the method comprises the steps: carrying out the class prediction based on a to-be-detected eye fundus image, and outputting a first image classification result; if the first image classification result represents that the eye fundus image to be detected is a color eye fundus image, detecting whether the color eye fundus image is a confocal laser eye fundus image to obtain a first detection result; if the first detection result represents that the color eye fundus image is not the confocal laser eye fundus image, detecting whether the color eye fundus image is a wide-angle eye fundus image, and obtaining a second detection result; and if the second detection result represents that the color eye fundus image is not the wide-angle eye fundus image, determining a camera type corresponding to the color eye fundus image based on a signal-to-noise ratio and a definition distribution diagram corresponding to the color eye fundus image, and outputting a second image classification result corresponding to the camera type. According to the embodiment, the color eye fundus images can be classified based on the camera type, so that a scientific basis is provided for subsequent image analysis and image quality evaluation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a method, apparatus and computer-readable medium for classifying multimodal fundus images. Background Technology

[0002] The fundus images collected by existing scientific research data are generally of many types, including multimodal images. These images need to be classified before they can be processed. Currently, there is no suitable classification method, and even if the existing classification models can perform simple classification, they cannot identify the camera type from the fundus images and classify the fundus images based on the camera type.

[0003] Color fundus images captured by different cameras vary, which affects image quality assessment and image parameter analysis to varying degrees. In order to meet the needs of scientific research and subsequent image processing, it is necessary to correct, specialize, and professionally adjust color fundus images captured by different cameras to improve the accuracy and reliability of subsequent image processing and scientific data analysis. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, embodiments of the present invention provide a method, apparatus, and computer-readable medium for classifying multimodal fundus images. This method can classify multimodal eye images based on a trained model, and can further classify color fundus images in the classification results based on camera type, thereby providing a scientific basis for subsequent image analysis and image quality assessment research projects.

[0005] According to a first aspect of the present invention, a classification method for multimodal fundus images is provided. The method includes: performing category prediction based on a fundus image to be tested, and outputting a first image classification result; the first image classification result is used to characterize the image modality of the fundus image to be tested, the image modality including non-fundus images, OCT images, OCTA images, fundus angiography images, and color fundus images.

[0006] Optionally, the classification method further includes: if the first image classification result indicates that the fundus image to be tested is a color fundus image, then detecting whether the color fundus image is a confocal laser fundus image, and obtaining a first detection result; if the first detection result indicates that the color fundus image is not a confocal laser fundus image, then detecting whether the color fundus image is a wide-angle fundus image, and obtaining a second detection result; if the second detection result indicates that the color fundus image is not a wide-angle fundus image, then determining the camera type corresponding to the color fundus image based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, and outputting a second image classification result corresponding to the camera type.

[0007] Optionally, the step of performing category prediction based on the fundus image to be tested and outputting a first image classification result includes: performing size conversion processing on the fundus image to be tested based on a preset size standard to generate a quasi-fundus image; performing color normalization processing on the quasi-fundus image to generate a normalized fundus image; splitting the image channels of the normalized fundus image to generate several channel images; for any channel image among the several channel images: extracting the gray-level histogram corresponding to each channel image to generate a channel gray-level histogram; using the channel gray-level histogram corresponding to each channel image and the normalized fundus image together as the test sample; performing category prediction on the test sample based on a classification model and outputting a first image classification result.

[0008] Optionally, detecting whether the color fundus image is a confocal laser fundus image to obtain a first detection result includes: extracting optic disc region color features from the color fundus image; and detecting whether the color fundus image is a confocal laser fundus image based on the optic disc region color features, using a model or preset rules, to obtain the first detection result.

[0009] Optionally, detecting whether the color fundus image is a wide-angle fundus image to obtain a second detection result includes: extracting interest from the color fundus image to generate a region of interest (ROI) image; segmenting the optic disc in the ROI image to obtain a segmented optic disc region; locating the macula in the ROI image to obtain the center position of the macula; and determining whether the color fundus image is a wide-angle fundus image based on the ROI image, the segmented optic disc region, and the center position of the macula to obtain the second detection result.

[0010] Optionally, determining whether the color fundus image is a wide-angle fundus image based on the region of interest image, the optic disc segmentation region, and the macula center position to obtain a second detection result includes: determining the circumscribed circle radius value corresponding to the region of interest image; determining the distance between the center position of the optic disc segmentation region and the macula center position to obtain the optic disc-macula distance; determining the ratio between the optic disc-macula distance and the circumscribed circle radius value as a first ratio; and determining the ratio between the diameter value of the optic disc segmentation region and the circumscribed circle radius value as a second ratio; if the first ratio is less than a first preset threshold and / or the second ratio is less than a second preset threshold, then the color fundus image is determined to be a wide-angle fundus image; if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, then the color fundus image is determined to be a non-wide-angle fundus image.

[0011] Optionally, the step of determining the camera type corresponding to the color fundus image based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, and outputting a second image classification result corresponding to the camera type, includes: extracting interest from the color fundus image to generate a region of interest image; determining the image signal-to-noise ratio and sharpness distribution map corresponding to the region of interest image; determining the average sharpness value corresponding to the sharpness distribution map; if the image signal-to-noise ratio is less than a third preset threshold and the average sharpness value is less than a fourth preset threshold, then the color fundus image is determined to be a handheld camera fundus image or a horizontal camera fundus image; if the image signal-to-noise ratio is not less than a third preset threshold and / or the average sharpness is not less than a fourth preset threshold, then the color fundus image is determined to be a desktop camera fundus image.

[0012] Optionally, the classification method further includes: labeling the original fundus images to obtain labeled pre-training samples; wherein the labels include: non-fundus labels, OCT labels, OCTA labels, fundus angiography labels, and color fundus labels; performing size conversion processing on the pre-training samples based on a preset size standard to generate quasi-sample images; performing color normalization processing on the quasi-sample images to generate normalized sample images; splitting the image channels of the normalized sample images to generate several channel sample images; for any channel sample image among the several channel sample images: extracting the gray-level histogram corresponding to each sample channel image to generate a sample gray-level histogram; using the sample gray-level histogram corresponding to each channel sample image and the normalized sample image together as training samples; and using a classification network to train a model on the several training samples to generate a classification model.

[0013] According to a second aspect of the present invention, a classification apparatus for multimodal fundus images is also provided. The apparatus includes: a category prediction module, configured to perform category prediction based on a fundus image to be tested, and output a first image classification result; the first image classification result is used to characterize the image modality of the fundus image to be tested, the image modality including non-fundus images, OCT images, OCTA images, fundus angiography images, and color fundus images.

[0014] According to a third aspect of the present invention, an application method based on multimodal fundus image classification is also provided, which applies the image classification results formed by the multimodal fundus image classification method as described in the first aspect to the fields of medical research, image database construction, telemedicine, clinical trials, medical insurance claims, image processing algorithm development, education and training, or intelligent ophthalmic diagnostic systems.

[0015] According to a fourth aspect of the present invention, an electronic device is also provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the classification method as described in the first aspect.

[0016] According to a fifth aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the classification method as described in the first aspect.

[0017] This invention provides a method, apparatus, and computer-readable medium for classifying multimodal fundus images. One embodiment of the method includes: First, based on a fundus image to be tested, performing category prediction and outputting a first image classification result; the first image classification result is used to characterize the image modality of the fundus image to be tested, the image modality including non-fundus images, OCT images, OCTA images, fundus angiography images, and color fundus images; Second, if the first image classification result indicates that the fundus image to be tested is a color fundus image, then detecting whether the color fundus image is a confocal laser fundus image, obtaining a first detection result; Then, if the first detection result indicates that the color fundus image is not a confocal laser fundus image, then detecting whether the color fundus image is a wide-angle fundus image, obtaining a second detection result; Finally, if the second detection result indicates that the color fundus image is not a wide-angle fundus image, then based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, determining the camera type corresponding to the color fundus image, and outputting a second image classification result corresponding to the camera type. This embodiment first classifies multimodal images of the eye based on a trained model, and then performs image processing on the color fundus images in the classification results, thereby realizing further classification of color fundus images based on camera type, thus providing a scientific basis for subsequent image analysis and image quality assessment and other scientific research projects. Attached Figure Description

[0018] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a flowchart illustrating a multimodal fundus image classification method provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the process of determining whether a color fundus image is a wide-angle fundus image in one embodiment of the present invention; Figure 3 This is a schematic diagram of the process for outputting a second image classification result corresponding to the camera type in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a multimodal fundus image classification device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] like Figure 1 The diagram shown is a flowchart illustrating a multimodal fundus image classification method provided in an embodiment of the present invention.

[0021] A classification method for multimodal fundus images, the method comprising at least the following steps: S101, Based on the fundus image to be tested, perform category prediction and output a first image classification result; the first image classification result is used to characterize the image modality of the fundus image to be tested, and the image modality includes non-fundus image, OCT image, OCTA image, fundus angiography image and color fundus image; S102, If the first image classification result characterizes the fundus image to be tested as a color fundus image, then detect whether the color fundus image is a confocal laser fundus image and obtain a first detection result; S103, if the first detection result indicates that the color fundus image is not a confocal laser fundus image, then detect whether the color fundus image is a wide-angle fundus image to obtain the second detection result; S104, if the second detection result indicates that the color fundus image is not a wide-angle fundus image, then based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, determine the camera type corresponding to the color fundus image, and output the second image classification result corresponding to the camera type.

[0022] In step S101, the fundus image to be tested is preprocessed to obtain a preprocessed fundus image. Based on a trained model, the preprocessed fundus image is used to predict its category, and a first image classification result is output. Alternatively, based on the fundus image to be tested, a category prediction is performed using preset rules, and a first image classification result is output. The first image classification result can be a non-fundus image, an OCT fundus image, an OCTA fundus image, a fundus angiography image, or a color fundus image, etc.

[0023] In step S102, color fundus images are selected from several first image classification results to obtain a first image dataset. For any color fundus image in the first image dataset, it is detected whether the color fundus image is a confocal laser fundus image, resulting in a first detection result. From several first detection results, confocal laser fundus images and non-confocal laser fundus images are selected to generate a first subset of the dataset consisting of several confocal laser fundus images and a second subset of the dataset consisting of non-confocal laser fundus images. Here, the detection method for confocal laser fundus images is not limited; it can be based on a model or on preset rules.

[0024] In step S103, for any color fundus image in the second subset: detect whether the color fundus image is a wide-angle fundus image to obtain a second detection result; select wide-angle fundus images and non-wide-angle fundus images from several second detection results to generate a third subset consisting of several wide-angle fundus images and a fourth subset consisting of several non-wide-angle fundus images. Here, the detection method for wide-angle fundus images is not limited; it can be based on a model or on preset rules.

[0025] In S104, for any color fundus image in the fourth subset of the dataset: based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, the camera type used to capture the color fundus image is determined, and a second image classification result corresponding to the camera type is output. Examples of the second image classification results are: fundus image from a desktop camera, fundus image from a handheld camera, and fundus image from a horizontal camera.

[0026] This embodiment first classifies the fundus images to be tested, then traces the camera type of the color fundus images in the first image classification results, and further classifies the color fundus images based on the camera type; thereby improving the accuracy and reliability of subsequent image processing and scientific data analysis.

[0027] In a preferred embodiment of this example, the step of performing category prediction based on the fundus image to be tested and outputting a first image classification result includes: performing a size conversion process on the fundus image to be tested based on a preset size standard to generate a quasi-fundus image; performing color normalization processing on the quasi-fundus image to generate a normalized fundus image; splitting the image channels of the normalized fundus image to generate several channel images; for any channel image among the several channel images: extracting the gray-level histogram corresponding to each channel image to generate a channel gray-level histogram; using the channel gray-level histogram corresponding to each channel image and the normalized fundus image together as the test sample; performing category prediction on the test sample based on a classification model and outputting a first image classification result.

[0028] This embodiment reduces image noise and improves the accuracy of the model's image classification prediction by preprocessing the fundus image to be tested.

[0029] In another preferred embodiment of this example, the step of detecting whether the color fundus image is a confocal laser fundus image and obtaining a first detection result includes: extracting optic disc region color features from the color fundus image; and detecting whether the color fundus image is a confocal laser fundus image based on the optic disc region color features, using a model or preset rules, and obtaining a first detection result.

[0030] Specifically, based on a neural network algorithm, color features of the optic disc region are extracted from a color fundus image. Histogram mean squared processing is then performed on the color features of the optic disc region to obtain a histogram feature image. The average gray value and gray variance corresponding to the histogram feature image are obtained. If the average gray value is less than a preset gray value threshold and the gray variance is greater than a preset gray variance threshold, then the color fundus image is determined to be a confocal laser fundus image. If the average gray value is not less than a preset gray value threshold and / or the variance is not greater than a preset variance threshold, then the color fundus image is determined not to be a confocal laser fundus image.

[0031] Alternatively, the color features of the optic disc region can be input into a trained binary classification model to output a classification result; there are two possibilities for the classification result: one is a confocal laser fundus image, and the other is a non-confocal laser fundus image.

[0032] The optic disc region in confocal laser fundus images is usually black. This embodiment determines whether a color fundus image is a confocal laser fundus image based on the color characteristics of the optic disc region. As a result, confocal laser fundus images can be effectively identified in color fundus images, thereby accurately classifying color fundus images and improving the accuracy of confocal laser fundus image recognition.

[0033] It should be understood that, in the various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0034] like Figure 2 The diagram shown is a flowchart for determining whether a color fundus image is a wide-angle fundus image in one embodiment of the present invention.

[0035] Determining whether a color fundus image is a wide-angle fundus image involves at least the following steps: S201, Extract the region of interest from the color fundus image and generate an image of the region of interest; S202, Perform segmentation processing on the visual disk in the region of interest image to obtain the visual disk segmentation region; S203, perform localization processing on the macula in the region of interest image to obtain the center position of the macula; S204. Based on the region of interest image, the segmented optic disc region, and the center position of the macula, determine whether the color fundus image is a wide-angle fundus image and obtain the second detection result.

[0036] Specifically, no restrictions are placed on the methods of segmentation and localization, as long as the segmented area of ​​the optic disc and the center position of the macula can be obtained.

[0037] For example, based on the region of interest image, the optic disc segmentation region, and the macula center position, determining whether the color fundus image is a wide-angle fundus image to obtain a second detection result includes: determining the circumscribed circle radius value corresponding to the region of interest image to determine the distance between the center position of the optic disc segmentation region and the macula center position, obtaining the optic disc-macula distance; determining the ratio between the optic disc-macula distance and the circumscribed circle radius value as a first ratio; and determining the ratio between the diameter value of the optic disc segmentation region and the circumscribed circle radius value as a second ratio; if the first ratio is less than a first preset threshold and / or the second ratio is less than a second preset threshold, then the color fundus image is determined to be a wide-angle fundus image; if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, then the color fundus image is determined to be a non-wide-angle fundus image.

[0038] Therefore, it is possible to effectively detect wide-angle fundus images from non-confocal laser color fundus images, which not only improves the accuracy of wide-angle fundus image detection, but also facilitates subsequent identification of camera types based on color fundus images and classification of color fundus images according to camera types, thereby improving the accuracy of the second image classification results.

[0039] like Figure 3 The diagram shown is a flowchart illustrating the process of outputting a second image classification result corresponding to the camera type in one embodiment of the present invention.

[0040] Output the second image classification result corresponding to the camera type; including at least the following steps: S301, Extract the region of interest from the color fundus image and generate an image of the region of interest; S302, Determine the image signal-to-noise ratio and sharpness distribution map corresponding to the region of interest; S303, Determine the average sharpness value corresponding to the sharpness distribution map; S304, if the image signal-to-noise ratio is less than the third preset threshold and the average sharpness value is less than the fourth preset threshold, then determine that the color fundus image is a handheld camera fundus image or a horizontal camera fundus image. S305, if the image signal-to-noise ratio is not less than the third preset threshold and / or the average sharpness is not less than the fourth preset threshold, then the color fundus image is determined to be a fundus image from a desktop camera.

[0041] Based on the average sharpness value and image signal-to-noise ratio, this embodiment can identify the camera type from color fundus images of non-broadband and non-confocal lasers, and thus output an image classification result corresponding to the camera type. Therefore, it can effectively identify the type of camera based on the image features of the color image, improving the accuracy of the second image classification result.

[0042] The classification method for multimodal fundus images provided in this embodiment will be described in detail below with reference to specific application scenarios.

[0043] A classification method for multimodal fundus images includes at least the following steps: S1. Label the original fundus images to obtain labeled pre-training samples; wherein, the labels include: non-fundus labels, OCT labels, OCTA labels, fundus angiography labels, and color fundus labels; based on a preset size standard, perform size conversion processing on the pre-training samples to generate quasi-sample images; perform color normalization processing on the quasi-sample images to generate normalized sample images; split the image channels of the normalized sample images to generate several channel sample images; for any channel sample image among the several channel sample images: extract the gray-level histogram corresponding to each sample channel image to generate a sample gray-level histogram; use the sample gray-level histogram corresponding to each channel sample image and the normalized sample image together as training samples; use a classification network to train a model on several training samples to generate a classification model.

[0044] S2, based on a preset size standard, the fundus image to be tested is subjected to size conversion processing to generate a quasi-fundus image; the quasi-fundus image is subjected to color normalization processing to generate a normalized fundus image; the image channels of the normalized fundus image are split to generate several channel images; for any channel image among the several channel images: the gray-level histogram corresponding to each channel image is extracted to generate a channel gray-level histogram; the channel gray-level histogram corresponding to each channel image and the normalized fundus image are used together as the test sample; the category of the test sample is predicted based on a classification model, and a first image classification result is output; wherein, the first image classification result is used to characterize the image modality of the fundus image to be tested, and the image modality includes non-fundus image, OCT image, OCTA image, fundus angiography image and color fundus image. S3, if the first image classification result indicates that the fundus image to be tested is a color fundus image, then extract the optic disc area color features from the color fundus image; and based on the optic disc area color features, detect whether the color fundus image is a confocal laser fundus image based on a model or preset rules to obtain a first detection result.

[0045] S4, if the first detection result indicates that the color fundus image is not a confocal laser fundus image, then the color fundus image is subjected to interest extraction to generate a region of interest (ROI) image; the optic disc in the ROI image is segmented to obtain an optic disc segmentation region; the macula in the ROI image is located to obtain the macula center position; the circumscribed circle radius value corresponding to the ROI image is determined; the distance between the center position of the optic disc segmentation region and the center position of the macula is determined to obtain the optic disc-macula distance; the ratio between the optic disc-macula distance and the circumscribed circle radius value is determined as a first ratio; and the ratio between the diameter value of the optic disc segmentation region and the circumscribed circle radius value is determined as a second ratio; if the first ratio is less than a first preset threshold and / or the second ratio is less than a second preset threshold, then the color fundus image is determined to be a wide-angle fundus image; if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, then the color fundus image is determined to be a non-wide-angle fundus image.

[0046] S5, perform interest extraction on the non-wide-angle fundus image to generate a region of interest image; determine the image signal-to-noise ratio and sharpness distribution map corresponding to the region of interest image; determine the average sharpness value corresponding to the sharpness distribution map; if the image signal-to-noise ratio is less than a third preset threshold and the average sharpness value is less than a fourth preset threshold, then determine that the color fundus image is a handheld camera fundus image or a horizontal camera fundus image; if the image signal-to-noise ratio is not less than a third preset threshold and / or the average sharpness is not less than a fourth preset threshold, then determine that the color fundus image is a desktop camera fundus image.

[0047] An application method based on multimodal fundus image classification applies the image classification results generated by the multimodal fundus image classification method as described in the first aspect to the fields of medical research, image database construction, telemedicine, clinical trials, medical insurance claims, image processing algorithm development, education and training, or intelligent ophthalmic diagnostic systems.

[0048] Multimodal fundus image classification methods are applied in medical research to evaluate the performance and influencing factors of different camera types when capturing fundus images, providing support for the diagnosis and treatment of fundus diseases. In ophthalmic disease research, identifying the camera type helps ensure consistency in image quality, thereby improving the accuracy and reliability of research results. This method can be used to examine and validate the performance and accuracy of the fundus cameras used, thus improving the quality of fundus images and the accuracy of diagnosis.

[0049] Multimodal fundus image classification methods, applied to the construction of image databases, can be used to evaluate the imaging quality of different fundus cameras when purchasing them, serving as a reference for procurement decisions. Medical institutions can plan equipment maintenance, upgrades, or replacements based on camera type and image quality evaluation results, ensuring the continued provision of high-quality medical services.

[0050] Multimodal fundus image classification methods are applied in the field of image database construction: they can classify and manage fundus images acquired by different types of cameras, facilitating medical research, teaching, and clinical applications. Understanding the camera type used to capture each image helps in data classification and standardized management.

[0051] Multimodal fundus image classification methods are applied in the field of telemedicine: identifying the type of camera in fundus images can help doctors determine image quality and ensure the accuracy of remote diagnosis and treatment.

[0052] Multimodal fundus image classification methods are applied in the field of medical service quality evaluation: health regulatory authorities can use this method to assess the quality of fundus imaging services provided by medical institutions, including equipment performance and image acquisition standards.

[0053] Multimodal fundus image classification methods are applied in the field of clinical trials: in clinical trials of ophthalmic drugs or treatments, ensuring the use of consistent or specific levels of cameras can reduce the impact of image quality on trial results.

[0054] Multimodal fundus image classification methods are applied in the field of medical insurance claims: insurance companies may need to evaluate the camera type of submitted fundus images to verify the accuracy of medical examinations and process related claims.

[0055] Multimodal fundus image classification methods are applied to the field of image processing algorithm development: researchers can optimize image processing and analysis algorithms based on the characteristics of different camera types, thereby improving the accuracy of disease detection and diagnosis.

[0056] The classification method for multimodal fundus images can be applied in the field of education and training: In ophthalmology professional training, teachers and students can learn and compare the advantages and disadvantages of different devices by analyzing fundus images captured by different cameras. This method can be applied to ophthalmology education, helping students better understand and master fundus image acquisition techniques by evaluating the characteristics and performance of different types of cameras in fundus image acquisition.

[0057] The classification method for multimodal fundus images is applied to the field of intelligent ophthalmic diagnostic systems: This method can be applied to intelligent ophthalmic diagnostic systems to evaluate the camera type of the acquired fundus images, providing doctors with more accurate diagnostic and treatment suggestions.

[0058] like Figure 4 The diagram shown is a structural schematic of a multimodal fundus image classification device provided in an embodiment of the present invention.

[0059] A classification device 400 for multimodal fundus images includes: a category prediction module 401, used to perform category prediction based on a fundus image to be tested, and output a first image classification result; the first image classification result is used to characterize the image modality of the fundus image to be tested, the image modality including non-fundus image, OCT image, OCTA image, fundus angiography image and color fundus image.

[0060] In a preferred embodiment of this invention, the device further includes: a first detection module 402, configured to detect whether the color fundus image is a confocal laser fundus image if the first image classification result indicates that the fundus image to be tested is a color fundus image, and obtain a first detection result; a second detection module 403, configured to detect whether the color fundus image is a wide-angle fundus image if the first detection result indicates that the color fundus image is not a confocal laser fundus image, and obtain a second detection result; and an output module 404, configured to determine the camera type corresponding to the color fundus image based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image if the second detection result indicates that the color fundus image is not a wide-angle fundus image, and output a second image classification result corresponding to the camera type.

[0061] In a preferred embodiment of this example, the category prediction module includes: a first generation unit, used to perform size conversion processing on the fundus image to be tested based on a preset size standard to generate a quasi-fundus image; a second generation unit, used to perform color normalization processing on the quasi-fundus image to generate a normalized fundus image; a third generation unit, used to split the image channels of the normalized fundus image to generate several channel images; a fourth generation unit, used to extract the gray-level histogram corresponding to each channel image for any one of the several channel images to generate a channel gray-level histogram; a determination unit, used to take the channel gray-level histogram corresponding to each channel image and the normalized fundus image together as the test sample; and an output unit, used to perform category prediction on the test sample based on a classification model and output a first image classification result.

[0062] In a preferred embodiment of this example, the first detection module includes: an extraction unit, used to extract optic disc region color features from the color fundus image; and an acquisition unit, used to detect whether the color fundus image is a confocal laser fundus image based on the optic disc region color features and a model or preset rules, to obtain a first detection result.

[0063] In a preferred embodiment of this example, the second detection module includes: an extraction unit for extracting interest from the color fundus image to generate a region of interest image; a segmentation processing unit for segmenting the optic disc in the region of interest image to obtain a segmented optic disc region; a localization processing unit for localizing the macula in the region of interest image to obtain the center position of the macula; and a determination unit for determining whether the color fundus image is a wide-angle fundus image based on the region of interest image, the segmented optic disc region, and the center position of the macula, thereby obtaining a second detection result.

[0064] In a preferred embodiment of this example, the determining unit includes: a first determining subunit, configured to determine the radius of the circumscribed circle corresponding to the region of interest image; a second determining subunit, configured to determine the distance between the center of the optic disc segmentation region and the center of the macula, obtaining the optic disc-macula distance; a third determining subunit, configured to determine the ratio between the optic disc-macula distance and the radius of the circumscribed circle as a first ratio; and to determine the ratio between the diameter of the optic disc segmentation region and the radius of the circumscribed circle as a second ratio; a fourth determining subunit, configured to determine the color fundus image as a wide-angle fundus image if the first ratio is less than a first preset threshold and / or the second ratio is less than a second preset threshold; and a fifth determining subunit, configured to determine the color fundus image as a non-wide-angle fundus image if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold.

[0065] In a preferred embodiment of this example, the output module includes: a generation unit for extracting interest from the color fundus image to generate a region of interest image; a first determining unit for determining the image signal-to-noise ratio and sharpness distribution map corresponding to the region of interest image; a second determining unit for determining the average sharpness value corresponding to the sharpness distribution map; a third determining unit for determining that the color fundus image is a handheld camera fundus image or a horizontal camera fundus image if the image signal-to-noise ratio is less than a third preset threshold and the average sharpness value is less than a fourth preset threshold; and a fourth determining unit for determining that the color fundus image is a desktop camera fundus image if the image signal-to-noise ratio is not less than the third preset threshold and / or the average sharpness is not less than the fourth preset threshold.

[0066] In a preferred embodiment of this invention, the device further includes: an acquisition module, used to label the original fundus image to obtain a pre-training sample with labels; wherein the labels include: non-fundus labels, OCT labels, OCTA labels, fundus angiography labels, and color fundus labels; a first generation module, used to perform size conversion processing on the pre-training sample based on a preset size standard to generate a quasi-sample image; a normalization module, used to perform color normalization processing on the quasi-sample image to generate a normalized sample image; a second generation module, used to split the image channels of the normalized sample image to generate several channel sample images; a third generation module, used to extract the gray-level histogram corresponding to each of the several channel sample images for any one of the channel sample images to generate a sample gray-level histogram; a determination module, used to use the sample gray-level histogram corresponding to each channel sample image and the normalized sample image together as training samples; and a training module, used to train a model on several training samples using a classification network to generate a classification model.

[0067] The above-described apparatus can execute the multimodal fundus image classification method provided in an embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the fundus image classification method. Technical details not described in detail in this embodiment can be found in the multimodal fundus image classification method provided in an embodiment of the present invention.

[0068] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the multimodal fundus image classification method of the present invention.

[0069] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.

[0070] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0071] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to the following embodiments of this application described in the "Exemplary Methods" section above.

[0072] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0073] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0074] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0075] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0076] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0077] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

[0078] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A classification method for multimodal fundus images, characterized in that, include: Based on the fundus image to be tested, class prediction is performed, and the first image classification result is output. The first image classification result is used to characterize the image modality of the fundus image to be tested, and the image modality includes non-fundus images, OCT images, OCTA images, fundus angiography images, and color fundus images.

2. The classification method according to claim 1, characterized in that, Also includes: If the first image classification result indicates that the fundus image to be tested is a color fundus image, then it is detected whether the color fundus image is a confocal laser fundus image, and a first detection result is obtained; If the first detection result indicates that the color fundus image is not a confocal laser fundus image, then the second detection result is obtained by detecting whether the color fundus image is a wide-angle fundus image. If the second detection result indicates that the color fundus image is not a wide-angle fundus image, then based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, the camera type corresponding to the color fundus image is determined, and a second image classification result corresponding to the camera type is output.

3. The classification method according to claim 1, characterized in that, The step of performing category prediction based on the fundus image to be tested and outputting a first image classification result includes: Based on a preset size standard, the fundus image to be tested is resized to generate a quasi-fundus image; The quasi-fundus image is subjected to color normalization processing to generate a normalized fundus image; The image channels of the normalized fundus image are split to generate several channel images; For any one of the channel images: extract the grayscale histogram corresponding to each channel image to generate a channel grayscale histogram; The grayscale histogram of each channel image and the normalized fundus image are used together as the test sample; Based on the classification model, the category of the sample to be tested is predicted, and the first image classification result is output.

4. The classification method according to claim 2, characterized in that, The step of detecting whether the color fundus image is a confocal laser fundus image to obtain a first detection result includes: Extract color features of the optic disc region from the color fundus image; Based on the color features of the optic disc area, the system detects whether the color fundus image is a confocal laser fundus image based on a model or preset rules, and obtains a first detection result.

5. The classification method according to claim 2, characterized in that, The step of detecting whether the color fundus image is a wide-angle fundus image to obtain a second detection result includes: The color fundus image is subjected to region of interest extraction to generate a region of interest image; The visual disk in the region of interest image is segmented to obtain the visual disk segmentation region; The macula in the region of interest image is localized to obtain the center position of the macula; Based on the region of interest image, the segmented optic disc region, and the center position of the macula, it is determined whether the color fundus image is a wide-angle fundus image, thus obtaining a second detection result.

6. The classification method according to claim 5, characterized in that, The step of determining whether the color fundus image is a wide-angle fundus image based on the region of interest image, the segmented optic disc region, and the center position of the macula, to obtain a second detection result, includes: Determine the radius of the circumcircle corresponding to the region of interest image; The distance between the center of the optic disc segmentation region and the center of the macula is determined to obtain the optic disc-macula distance; The ratio between the optic disc-macular distance and the radius of the circumscribed circle is determined as the first ratio; and the ratio between the diameter of the optic disc segmentation region and the radius of the circumscribed circle is determined as the second ratio. If the first ratio is less than the first preset threshold and / or the second ratio is less than the second preset threshold, then the color fundus image is determined to be a wide-angle fundus image; If the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, then the color fundus image is determined to be a non-wide-angle fundus image.

7. The classification method according to claim 2, characterized in that, The step of determining the camera type corresponding to the color fundus image based on the signal-to-noise ratio and sharpness distribution map corresponding to the color fundus image, and outputting a second image classification result corresponding to the camera type, includes: The color fundus image is subjected to region of interest extraction to generate a region of interest image; Determine the image signal-to-noise ratio and sharpness distribution map corresponding to the region of interest; Determine the average sharpness value corresponding to the sharpness distribution map; If the image signal-to-noise ratio is less than a third preset threshold and the average sharpness value is less than a fourth preset threshold, then the color fundus image is determined to be a handheld camera fundus image or a horizontal camera fundus image. If the image signal-to-noise ratio is not less than a third preset threshold and / or the average sharpness is not less than a fourth preset threshold, then the color fundus image is determined to be a fundus image from a desktop camera.

8. The classification method according to claim 3, characterized in that, Also includes: The original fundus images are labeled to obtain labeled pre-training samples; wherein, the labels include: non-fundus labels, OCT labels, OCTA labels, fundus angiography labels, and color fundus labels; Based on a preset size standard, the pre-trained samples are subjected to size conversion processing to generate quasi-sample images; The quasi-sample image is subjected to color normalization processing to generate a normalized sample image; The image channels of the normalized sample image are split to generate several channel sample images; For any one of the channel sample images: extract the grayscale histogram corresponding to each of the sample channel images to generate a sample grayscale histogram; The grayscale histogram of each channel sample image and the normalized sample image are used together as training samples; A classification model is generated by training several training samples using a classification network.

9. A classification device for multimodal fundus images, characterized in that, include: The category prediction module is used to predict the category based on the fundus image to be tested and output the first image classification result; The first image classification result is used to characterize the image modality of the fundus image to be tested, and the image modality includes non-fundus images, OCT images, OCTA images, fundus angiography images, and color fundus images.

10. An application method based on multimodal fundus image classification, characterized in that, The image classification results generated by the multimodal fundus image classification method as described in any one of claims 1-8 may be applied to the fields of medical research, image database construction, telemedicine, clinical trials, medical insurance claims, image processing algorithm development, education and training, or intelligent ophthalmic diagnostic systems.

11. An electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the classification method as described in any one of claims 1-8.

12. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the classification method as described in any one of claims 1-8.

Images