Hand-held multispectral-based fundus imaging and exudate identification method and system

Abstract

The invention provides a fundus imaging and exudate identification method and system based on handheld multispectrum, and the method comprises the steps: constructing an optical imaging system and a handheld multispectral light source, and forming handheld multispectral equipment; collecting a multispectral fundus image; and quantizing and identifying the fundus image and the exudate by adopting an exudate identification quantization algorithm based on the multispectral image. According to the invention, multispectral fundus imaging can be realized in a portable environment, and exudate in each spectral image can be accurately identified and quantified, so that the early detection capability and diagnosis accuracy of fundus lesions and systemic lesions predictable by fundus are improved; by optimizing the optical path design, the mechanical structure and the exudate recognition quantification algorithm based on the multispectral image, multispectral imaging and exudate recognition quantification of the fundus are achieved, and the fundus imaging device has the advantages of being good in portability, multispectral imaging and image analyzable quantification, is suitable for being used in clinical and portable environments and has wide application prospects. The method is especially suitable for detection and diagnosis of early fundus lesions.

Description

Technical Field

[0001] The present invention belongs to the field of medical detection imaging and analysis, and specifically, relates to a handheld multispectral-based fundus imaging and exudate identification method and system, in particular, a portable medical imaging method for fundus exudate detection and a handheld multispectral fundus camera and a method for identifying and quantifying exudates in the resulting image. Background Art

[0002] In the prior art, fundus examination equipment mainly includes two categories: one is the traditional fundus camera based on white light imaging, and the other is the fundus camera based on multispectral imaging. Although the traditional white light fundus camera can provide high-resolution fundus images, its early detection capability of lesions is relatively limited, and it cannot provide multispectral information about fundus tissue. Multispectral imaging technology uses light of different wavelengths to illuminate the fundus, combined with image processing technology, to obtain fundus images under different spectra. These images can not only show the conventional fundus structure, but also reveal the physiological and pathological information of fundus tissue by analyzing the reflection spectrum at different wavelengths.

[0003] However, existing multispectral fundus cameras are usually bulky, not easy to carry, and complicated to operate. They do not have the function of accurately identifying and quantifying exudates in each multispectral image, and are not suitable for use by clinicians in a portable environment.

[0004] The patent document "Multi-spectral fundus imaging remote diagnosis and treatment system and its operation method" (CN106725292A) discloses a multi-spectral fundus imaging device and diagnosis and treatment system, but still requires junior medical personnel to input primary manual diagnosis results and generate preliminary manual diagnosis results. The recognition and quantification are not accurate and the operation is complicated.

[0005] Patent document "Multi-spectral light source and fundus imaging system" (CN109431458A) discloses a fundus imaging system with multiple laser light sources, but it is also unable to identify and quantify exudates and has poor portability.

[0006] Therefore, there is an urgent need for a handheld multispectral fundus imaging method with good portability, high imaging quality and exudate identification and detection to solve the above problems. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a fundus imaging and exudate identification method and system based on handheld multi-spectrum.

[0008] The handheld multispectral fundus imaging and exudate identification method provided by the present invention includes:

[0009] Step S1: construct an optical imaging system and a handheld multispectral light source to form a handheld multispectral device;

[0010] Step S2: collecting multispectral fundus images by a handheld multispectral device;

[0011] Step S3: using an exudate identification and quantification algorithm based on multispectral images to quantitatively identify the fundus image and exudate.

[0012] Preferably, the optical imaging system comprises: an eyepiece group 2, a multi-band light board 3, a dichroic prism group 5, a retina objective group 8, an alignment light group 6, a fixation light group 7, an infrared filter switch 9, a focusing motor 10 and a CCD imaging device 4.

[0013] The eyepiece group 2 is installed at the front end of the optical path and is aligned with the dichroic prism group 5; the dichroic prism group 5 is installed at a fixed angle on the imaging side behind the eyepiece group; the alignment lamp group 6, the fixation lamp group 7 and the multi-band lamp board 3 are installed on the same plane as the dichroic prism group 5; the retina objective lens group 8 is installed on the imaging side of the dichroic prism group 5; the CCD imaging device 4 is installed on the optical path behind the retina objective lens group 5; the infrared filter switch 9 is installed between the retina objective lens group 8 and the CCD imaging device; and the focus motor 10 is installed on the CCD imaging device;

[0014] The fixed angle range is 20 degrees to 30 degrees;

[0015] The eyepiece set is a pair of achromatic lenses with a focal length of 30 mm;

[0016] The multi-band light panel includes LED light sources of different wavelengths;

[0017] The LED light source includes white light, blue light, green light and red light;

[0018] The achromatic lens of the retina objective lens group has a focal length of 15 mm and a magnification ratio of 1:1.25;

[0019] The fundus image collected by the CCD imaging device is stored in a flash memory in the device and is connected to the outside for transmission via a USB interface.

[0020] Preferably, the handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch;

[0021] The number of the switched light sources is set according to demand;

[0022] The step S2 comprises:

[0023] Step S2.1: Aim at the object to be measured and adjust the focus by the focus motor until the fundus image is clear;

[0024] Step S2.2: collecting fundus images under different spectra, and saving and transmitting the multi-spectral fundus images to an external storage device;

[0025] The exudate identification and quantification algorithm based on multispectral images includes image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation;

[0026] The image preprocessing is multispectral image grayscale preprocessing;

[0027] The image segmentation based on deep learning adopts UNet neural network;

[0028] The exudate area quantification calculation is based on the segmentation mask, binarization and then using openCV to find the contour and obtain the maximum contour and its area.

[0029] Preferably, step S3 comprises:

[0030] Step S3.1: performing image preprocessing on the fundus multispectral image to obtain multispectral information;

[0031] Step S3.2: spectral mixing operation, effectively fusing multi-spectral information into feature maps;

[0032] Step S3.3: Load the feature map into the deep neural network, and perform the same image preprocessing on the paired images of fundus exudates manually annotated by experts in the public dataset, and use them as the supervised input network for model training to obtain a trained neural network;

[0033] Step S3.4: using the trained neural network to segment and identify the exudates in the multispectral image one by one to obtain the prediction results;

[0034] Step S3.5: Binarize the prediction results, find the contours through openCV and obtain the largest contour and its area.

[0035] Preferably, the step S3.2 comprises:

[0036] Step S3.2.1: Define the collected multispectral image as I, I = {I1, I2, ..., I N};

[0037] Among them, I i represents the i-th spectral channel, i = 1, 2, ..., N;

[0038] N represents the total number of spectral channels;

[0039] Step S3.2.2: Compress the spectral channels F = W·I and mix them. The mixed feature map

[0040] Among them, F represents the generated multi-channel feature map, the size of which is n×h×w;

[0041] h and w represent the height and width of the image respectively;

[0042] W represents the weight matrix, with a size of n×N;

[0043] n is the total number of channels after compression;

[0044] F j represents the weighted sum of all spectral channels;

[0045] w ji represents the element in the jth row and ith column of the weight matrix W;

[0046] j represents the jth output channel, j = 1, 2, ..., n);

[0047] Step S3.2.3: training and initializing the weight matrix W;

[0048] The weight matrix W is set to be uniformly distributed when initialized;

[0049] Step S3.2.4: Spectral feature recombination, mixed feature map F j Input to the encoder part of the neural network;

[0050] The total number n of channels after compression is 3, 5 or 7;

[0051] Step S3.2.5: Construct joint loss function

[0052] in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M;

[0053] M represents a multispectral image with M different spectra;

[0054] λ k represents the weight of the kth spectral input;

[0055] Y k represents the true label of the kth spectral input;

[0056] Y′ k represents the predicted output for the kth spectral input.

[0057] A handheld multispectral fundus imaging and exudate identification system provided by the present invention comprises: an optical imaging system, a handheld multispectral light source, and an exudate identification and quantification algorithm based on multispectral images;

[0058] The optical imaging system and the handheld multispectral light source constitute a handheld multispectral device;

[0059] Collect multispectral fundus images through handheld multispectral devices;

[0060] The exudate identification and quantification algorithm based on multispectral images is used to quantitatively identify fundus images and exudates.

[0061] Preferably, the optical imaging system comprises: an eyepiece group 2, a multi-band light board 3, a dichroic prism group 5, a retina objective group 8, an alignment light group 6, a fixation light group 7, an infrared filter switch 9, a focusing motor 10 and a CCD imaging device 4.

[0062] The eyepiece group 2 is installed at the front end of the optical path and is aligned with the dichroic prism group 5; the dichroic prism group 5 is installed at a fixed angle on the imaging side behind the eyepiece group; the alignment lamp group 6, the fixation lamp group 7 and the multi-band lamp board 3 are installed on the same plane as the dichroic prism group 5; the retina objective lens group 8 is installed on the imaging side of the dichroic prism group 5; the CCD imaging device 4 is installed on the optical path behind the retina objective lens group 5; the infrared filter switch 9 is installed between the retina objective lens group 8 and the CCD imaging device; and the focus motor 10 is installed on the CCD imaging device;

[0063] The fixed angle range is 20 degrees to 30 degrees;

[0064] The eyepiece set is a pair of achromatic lenses with a focal length of 30 mm;

[0065] The multi-band light panel includes LED light sources of different wavelengths;

[0066] The LED light source includes white light, blue light, green light and red light;

[0067] The achromatic lens of the retina objective lens group has a focal length of 15 mm and a magnification ratio of 1:1.25;

[0068] The fundus image collected by the CCD imaging device is stored in a flash memory in the device and is connected to the outside for transmission via a USB interface.

[0069] Preferably, the handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch;

[0070] The number of the switched light sources is set according to demand;

[0071] The collection includes:

[0072] Aim at the object to be measured and adjust the focus using the focus motor until the fundus image is clear;

[0073] Fundus images under different spectra are collected, and multi-spectral fundus images are saved and transmitted to external storage.

[0074] The exudate identification and quantification algorithm based on multispectral images includes image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation;

[0075] The image preprocessing is multispectral image grayscale preprocessing;

[0076] The image segmentation based on deep learning adopts UNet neural network;

[0077] The exudate area quantification calculation is based on the segmentation mask, binarization and then using openCV to find the contour and obtain the maximum contour and its area.

[0078] Preferably, the quantitative identification includes:

[0079] Perform image preprocessing on fundus multispectral images to obtain multispectral information;

[0080] Spectral mixing operation, effectively fusing multi-spectral information into feature maps;

[0081] The feature map is loaded into the deep neural network, and the paired images of fundus exudates manually annotated by experts in the public dataset are subjected to the same image preprocessing and used as the supervised input network for model training to obtain a trained neural network;

[0082] The trained neural network is used to segment and identify the exudates in the multispectral images one by one to obtain the prediction results;

[0083] Binarize the predicted results, find the contours through openCV and get the largest contour and its area.

[0084] Preferably, the spectral mixing operation comprises:

[0085] Define the collected multispectral image as I, I = {I1, I2, ..., I N};

[0086] Among them, I i represents the i-th spectral channel, i = 1, 2, ..., N;

[0087] N represents the total number of spectral channels;

[0088] Compress the spectral channel F = W·I and mix it. The feature map after mixing

[0089] Among them, F represents the generated multi-channel feature map, the size of which is n×h×w;

[0090] h and w represent the height and width of the image respectively;

[0091] W represents the weight matrix, with a size of n×N;

[0092] n is the total number of channels after compression;

[0093] F j represents the weighted sum of all spectral channels;

[0094] w ji represents the element in the jth row and ith column of the weight matrix W;

[0095] j represents the jth output channel, j = 1, 2, ..., n);

[0096] Training and initialization weight matrix W;

[0097] The weight matrix W is set to be uniformly distributed when initialized;

[0098] The mixed feature map F j Input to the encoder part of the neural network;

[0099] The total number n of channels after compression is 3, 5 or 7;

[0100] Constructing a joint loss function

[0101] in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M;

[0102] M represents a multispectral image with M different spectra;

[0103] λ k represents the weight of the kth spectral input;

[0104] Y k represents the true label of the kth spectral input;

[0105] Y′ k represents the predicted output for the kth spectral input.

[0106] Compared with the prior art, the present invention has the following beneficial effects:

[0107] 1. The present invention realizes multispectral imaging and exudate identification and quantification of the fundus by optimizing the optical path design, mechanical structure and exudate identification and quantification algorithm based on multispectral images.

[0108] 2. The present invention has the characteristics of good portability, multi-spectral imaging, and image analyzability and quantification, and is suitable for use in clinical and portable environments, and is particularly suitable for the detection and diagnosis of early fundus lesions.

[0109] 3. The present invention can realize multi-spectral fundus imaging in a portable environment and complete the accurate identification and quantification of exudates in each spectral image, thereby improving the early detection capability and diagnostic accuracy of fundus lesions and systemic lesions that can be predicted by the fundus. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0111] Figure 1 This is a flow chart of the handheld multispectral-based fundus imaging and exudate identification method.

[0112] Figure 2 Schematic diagram of the handheld multispectral fundus imaging optical system.

[0113] Figure 3 Schematic diagram of the supporting mechanical structure for multi-spectral light source switching.

[0114] Figure 4 Schematic diagram of the algorithm flow for exudate identification and quantification based on multispectral images.

[0115] The figure shows:

[0116] DETAILED DESCRIPTION

[0117] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0118] By optimizing the optical path design, mechanical structure, and artificial intelligence algorithm, the present invention can realize multi-spectral fundus imaging in a portable environment and complete the accurate identification and quantification of exudates in each spectral image, thereby improving the early detection capability and diagnostic accuracy of fundus lesions and systemic lesions that can be predicted by the fundus.

[0119] According to a handheld multispectral fundus imaging and exudate identification method provided by the present invention, Figure 1 As shown, including:

[0120] Step S1: construct an optical imaging system and a handheld multispectral light source to form a handheld multispectral device;

[0121] In more preferred embodiments, Figure 2As shown, the optical imaging system includes: an eyepiece group, a multi-band light board, a dichroic prism group, a retina objective group and a CCD imaging device, which can accurately capture fundus images under different spectra.

[0122] Eyepiece group: A pair of achromatic lenses with a focal length of 30mm is used to reduce chromatic aberration and improve image clarity. It is closest to the eyeball being imaged and is used for preliminary focusing of the light beam.

[0123] Multi-band light board: The light board contains multiple sets of LED light sources with different wavelengths, including white light, blue light, green light and red light, which are used to irradiate tissues at different depths of the fundus.

[0124] Beam splitter group: It is set on the imaging side of the eyepiece group and installed at an angle of 20 to 30 degrees. It is used to change the propagation direction of the light beam and optimize the path of the light beam entering the retina objective group so that the light beam can be accurately projected to the target area of ​​the fundus.

[0125] Retina objective lens group: Located on the imaging side of the beam splitter prism group, the achromatic lens of the retina objective lens group has a focal length of 15mm and a magnification ratio of 1:1.25. It is used to perform secondary precise focusing of the light beam reflected from the fundus to ensure image clarity.

[0126] CCD imaging device: As an imaging device of the optical system, it is connected to an external computer through a USB interface. The captured images can be stored in the flash memory inside the device and transferred to an external computer through the interface for further analysis. CCD generates high-resolution images by collecting reflected light processed by the optical system. The image data can be stored and analyzed later.

[0127] The eyepiece assembly is installed at the front end of the optical path, and the position is adjusted so as to be aligned with the beam splitter prism assembly;

[0128] The beam splitter prism group is installed behind the eyepiece group at an angle of 20 to 30 degrees to change the propagation direction of the light beam;

[0129] The retina objective lens group is installed on the imaging side of the beam splitter prism group to ensure that its focusing ability matches the optical design of the entire system;

[0130] The CCD imaging device is installed behind the retina objective lens group to collect the processed reflected light.

[0131] In more preferred embodiments, Figure 3 As shown, the handheld multi-spectrum light source adopts a push-pull light source switching device to achieve physical switching of the light source, which is simple and convenient to operate and more lightweight.

[0132] The handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch, which ensures that the light source switching is flexible and stable. The switching speed and reliability of the light source are tested after installation.

[0133] Among them, the number of switchable light sources is set according to demand.

[0134] Step S2: collecting multispectral fundus images by a handheld multispectral device;

[0135] In more preferred embodiments, step S2 includes step S2.1: aligning the eyeball to be inspected, observing through the eyepiece, and adjusting the focus using a focusing motor to obtain a clear image;

[0136] Step S2.2: Collect fundus images under different spectra, and save and transfer the multispectral images to a computer for subsequent analysis.

[0137] Step S3: using an exudate identification and quantification algorithm based on multispectral images to quantitatively identify the fundus image and exudate.

[0138] In more preferred embodiments, step S3 includes step S3.1: performing image preprocessing on the fundus multispectral image, including but not limited to grayscale conversion, to obtain multispectral information.

[0139] Step S3.2: Spectral mixing operation effectively fuses multi-spectral information into a more representative feature map, so that the deep neural network can show stronger generalization when processing images of different spectra.

[0140] Step S3.3: Load the feature map into the deep neural network UNet, and perform the same image preprocessing on the paired images of fundus exudates manually annotated by experts in the public dataset and use them as the supervised input network for model training.

[0141] Step S3.4: Use the trained neural network to segment and identify the exudates in the multispectral image one by one to obtain the prediction results.

[0142] Step S3.5: Binarize the prediction results, and then find the contour through openCV and obtain the maximum contour and its area.

[0143] In more preferred embodiments, Figure 4 As shown, the exudate identification and quantification algorithm based on multispectral images includes: image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation.

[0144] The image preprocessing is multispectral image grayscale preprocessing, the deep learning-based image segmentation uses the UNet neural network, and the exudate area quantification calculation is based on the segmentation mask and binarization and then uses openCV to find the contour and obtain the maximum contour and its area, all of which are current classic methods.

[0145] When processing multispectral images, in order to enhance the generalization of the model, spectral mixing is used to extract more representative features and generate feature maps similar to standard RGB channels. This process includes steps such as spectral channel selection, mixing, and feature recombination, which can expand the feature map after spectral mixing to multiple channels instead of limiting it to 3 channels. The spectral mixing feature extraction includes:

[0146] Step S3.2.1: define and represent spectral channels;

[0147] Multispectral images usually contain multiple spectral channels, each of which represents image information at a specific wavelength. The acquired multispectral image is defined as I, which consists of a total of N spectral channels, expressed as:

[0148] I={I1,I2,...,I N};

[0149] Among them, I i Represents the i-th spectral channel, i=1, 2, ..., N.

[0150] Step S3.2.2: compressing the mixed spectral channels;

[0151] In order to mix multi-spectral information to generate a three-channel feature map similar to RGB, a weight matrix W is introduced with a size of n×N to compress N spectral channels into n channels. This operation can be expressed as:

[0152] F=W·I

[0153] Among them, F represents the generated multi-channel feature map, the size of which is n×h×w (the same as the standard RGB image);

[0154] h and w represent the height and width of the image respectively.

[0155] For each output channel j (j = 1, 2, ..., n), the feature map F j Represents the weighted sum of all spectral channels, recorded as the mixed feature map:

[0156]

[0157] Among them, w ji The element in the j-th row and i-th column of the weight matrix W is the contribution of the i-th spectral channel to the j-th output channel.

[0158] Step S3.2.3: training and initializing the weight matrix;

[0159] The weight matrix W is a trainable parameter of the model and is optimized by back propagation during training. The goal is to make the mixed feature map F jIn the subsequent convolution operation, more representative features can be extracted to adapt to the input of different spectra. During initialization, W is set to a uniform distribution.

[0160] Step S3.2.4: spectral feature reconstruction;

[0161] The mixed feature map F j The input is sent to the encoder part of the neural network for further feature extraction and segmentation. At this point, when the model processes images with different spectra, it is equivalent to operating in the "RGB" space, making its results for different spectral imaging more robust.

[0162] When the number of spectra is large, F is expanded to multiple channels instead of being limited to 3 channels. For example, feature maps with 5 or 7 channels are generated, that is, n is 3, 5 or 7, so as to provide richer spectral information for subsequent convolution operations.

[0163] Step S3.2.5: construct a joint loss function;

[0164] In order to further enhance the generalization of the model, it is necessary to construct a joint loss function to supervise the segmentation results of multiple spectral inputs at the same time. Assuming there are M multispectral images with different spectra, the joint loss function It can be expressed as:

[0165]

[0166] in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M;

[0167] λ k represents the weight of the kth spectral input;

[0168] Y k represents the true label of the kth spectral input;

[0169] Y′ k represents the predicted output for the kth spectral input.

[0170] A handheld multispectral fundus imaging and exudate identification system provided by the present invention comprises: an optical imaging system, a handheld multispectral light source, and an exudate identification and quantification algorithm based on multispectral images.

[0171] The optical imaging system and the handheld multispectral light source constitute a handheld multispectral device;

[0172] Collect multispectral fundus images through handheld multispectral devices;

[0173] The exudate identification and quantification algorithm based on multispectral images is used to quantitatively identify fundus images and exudates.

[0174] In more preferred examples, the optical imaging system includes: an eyepiece group 2, a multi-band light board 3, a dichroic prism group 5, a retina objective group 8, an alignment light group 6, a fixation light group 7, an infrared filter switch 9, a focusing motor 10 and a CCD imaging device 4.

[0175] The eyepiece group 2 is installed at the front end of the optical path and is aligned with the dichroic prism group 5; the dichroic prism group 5 is installed at a fixed angle on the imaging side behind the eyepiece group; the alignment lamp group 6, the fixation lamp group 7 and the multi-band lamp board 3 are installed on the same plane as the dichroic prism group 5; the retina objective lens group 8 is installed on the imaging side of the dichroic prism group 5; the CCD imaging device 4 is installed on the optical path behind the retina objective lens group 5; the infrared filter switch 9 is installed between the retina objective lens group 8 and the CCD imaging device; and the focus motor 10 is installed on the CCD imaging device;

[0176] The fixed angle range is 20 degrees to 30 degrees;

[0177] The eyepiece set is a pair of achromatic lenses with a focal length of 30 mm;

[0178] The multi-band light panel includes LED light sources of different wavelengths;

[0179] The LED light source includes white light, blue light, green light and red light;

[0180] The achromatic lens of the retina objective lens group has a focal length of 15 mm and a magnification ratio of 1:1.25;

[0181] The fundus image collected by the CCD imaging device is stored in a flash memory in the device and is connected to the outside for transmission via a USB interface.

[0182] In more preferred embodiments, the handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch;

[0183] The number of the switched light sources is set according to demand;

[0184] The collection includes:

[0185] Aim at the object to be measured and adjust the focus using the focus motor until the fundus image is clear;

[0186] Collect fundus images under different spectra, and save and transmit multi-spectral fundus images to external storage;

[0187] The exudate identification and quantification algorithm based on multispectral images includes image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation;

[0188] The image preprocessing is multispectral image grayscale preprocessing;

[0189] The image segmentation based on deep learning adopts UNet neural network;

[0190] The exudate area quantification calculation is based on the segmentation mask, binarization and then using openCV to find the contour and obtain the maximum contour and its area.

[0191] In more preferred embodiments, the quantitative identification includes:

[0192] The fundus multispectral image is preprocessed to obtain multispectral information.

[0193] Spectral mixing operation, effectively fusing multi-spectral information into feature maps;

[0194] The feature map is loaded into the deep neural network, and the paired images of fundus exudates manually annotated by experts in the public dataset are subjected to the same image preprocessing and used as the supervised input network for model training to obtain a trained neural network;

[0195] The trained neural network is used to segment and identify the exudates in the multispectral images one by one to obtain the prediction results;

[0196] Binarize the predicted results, find the contours through openCV and get the largest contour and its area.

[0197] In more preferred embodiments, the spectral mixing operation includes:

[0198] Define the collected multispectral image as I, I = {I1, I2, ..., I N};

[0199] Among them, I i represents the i-th spectral channel, i = 1, 2, ..., N;

[0200] N represents the total number of spectral channels;

[0201] Compress the spectral channel F = W·I and mix it. The feature map after mixing

[0202] Among them, F represents the generated multi-channel feature map, the size of which is n×h×w;

[0203] h and w represent the height and width of the image respectively;

[0204] W represents the weight matrix, with a size of n×N;

[0205] n is the total number of channels after compression;

[0206] F jrepresents the weighted sum of all spectral channels;

[0207] w ji represents the element in the jth row and ith column of the weight matrix W;

[0208] j represents the jth output channel, j = 1, 2, ..., n);

[0209] Training and initialization weight matrix W;

[0210] The weight matrix W is set to be uniformly distributed when initialized;

[0211] The mixed feature map F j Input to the encoder part of the neural network;

[0212] The total number n of channels after compression is 3, 5 or 7;

[0213] Constructing a joint loss function

[0214] in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M;

[0215] M represents a multispectral image with M different spectra;

[0216] λ k represents the weight of the kth spectral input;

[0217] Y k represents the true label of the kth spectral input;

[0218] Y′ k represents the predicted output for the kth spectral input.

[0219] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0220] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A handheld multispectral fundus imaging and exudate identification method, characterized in that: include: Step S1: construct an optical imaging system and a handheld multispectral light source to form a handheld multispectral device; Step S2: collecting multispectral fundus images by a handheld multispectral device; Step S3: using an exudate identification and quantification algorithm based on multispectral images to quantitatively identify the fundus image and exudate.

2. The handheld multispectral fundus imaging and exudate identification method according to claim 1 is characterized in that: The optical imaging system comprises: an eyepiece group 2, a multi-band light board 3, a dichroic prism group 5, a retina objective group 8, an alignment light group 6, a fixation light group 7, an infrared filter switch 9, a focus motor 10 and a CCD imaging device 4; The eyepiece group 2 is installed at the front end of the optical path and is aligned with the dichroic prism group 5; the dichroic prism group 5 is installed at a fixed angle on the imaging side behind the eyepiece group; the alignment lamp group 6, the fixation lamp group 7 and the multi-band lamp board 3 are installed on the same plane as the dichroic prism group 5; the retina objective lens group 8 is installed on the imaging side of the dichroic prism group 5; the CCD imaging device 4 is installed on the optical path behind the retina objective lens group 5; the infrared filter switch 9 is installed between the retina objective lens group 8 and the CCD imaging device; and the focus motor 10 is installed on the CCD imaging device; The fixed angle range is 20 degrees to 30 degrees; The eyepiece set is a pair of achromatic lenses with a focal length of 30 mm; The multi-band light panel includes LED light sources of different wavelengths; The LED light source includes white light, blue light, green light and red light; The achromatic lens of the retina objective lens group has a focal length of 15 mm and a magnification ratio of 1:1.25; The fundus image collected by the CCD imaging device is stored in a flash memory in the device and is connected to the outside for transmission via a USB interface.

3. The handheld multispectral fundus imaging and exudate identification method according to claim 1 is characterized in that: The handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch; The number of the switched light sources is set according to demand; The step S2 comprises: Step S2.1: Aim at the object to be measured and adjust the focus by the focus motor until the fundus image is clear; Step S2.2: collecting fundus images under different spectra, and saving and transmitting the multi-spectral fundus images to an external storage device; The exudate identification and quantification algorithm based on multispectral images includes image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation; The image preprocessing is multispectral image grayscale preprocessing; The image segmentation based on deep learning adopts UNet neural network; The exudate area quantification calculation is based on the segmentation mask, binarization and then using openCV to find the contour and obtain the maximum contour and its area.

4. The handheld multispectral fundus imaging and exudate identification method according to claim 1 is characterized in that: The step S3 comprises: Step S3.1: performing image preprocessing on the fundus multispectral image to obtain multispectral information; Step S3.2: spectral mixing operation, effectively fusing multi-spectral information into feature maps; Step S3.3: Load the feature map into the deep neural network, and perform the same image preprocessing on the paired images of fundus exudates manually annotated by experts in the public dataset, and use them as the supervised input network for model training to obtain a trained neural network; Step S3.4: using the trained neural network to segment and identify the exudates in the multispectral image one by one to obtain the prediction results; Step S3.5: Binarize the prediction results, find the contours through openCV and obtain the largest contour and its area.

5. The handheld multispectral fundus imaging and exudate identification method according to claim 4 is characterized in that: The step S3.2 comprises: Step S3.2.1: Define the collected multispectral image as I, I = {I1, I2, ..., I N }; Among them, I i represents the i-th spectral channel, i = 1, 2, ..., N; N represents the total number of spectral channels; Step S3.2.2: Compress the spectral channels F = W·I and mix them. The mixed feature map Among them, F represents the generated multi-channel feature map, the size of which is n×h×w; h and w represent the height and width of the image respectively; W represents the weight matrix, with a size of n×N; n is the total number of channels after compression; F j represents the weighted sum of all spectral channels; w ji represents the element in the jth row and ith column of the weight matrix W; j represents the jth output channel, j = 1, 2, ..., n); Step S3.2.3: training and initializing the weight matrix W; The weight matrix W is set to be uniformly distributed when initialized; Step S3.2.4: Spectral feature recombination, mixed feature map F j Input to the encoder part of the neural network; The total number n of channels after compression is 3, 5 or 7; Step S3.2.5: Construct joint loss function in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M; M represents a multispectral image with M different spectra; λ k represents the weight of the kth spectral input; Y k represents the true label of the kth spectral input; Y′ k represents the predicted output for the kth spectral input.

6. A handheld multispectral fundus imaging and exudate identification system, characterized in that: include: Optical imaging system, handheld multispectral light source, and exudate identification and quantification algorithm based on multispectral images; The optical imaging system and the handheld multispectral light source constitute a handheld multispectral device; Collect multispectral fundus images through handheld multispectral devices; The exudate identification and quantification algorithm based on multispectral images is used to quantitatively identify fundus images and exudates.

7. The handheld multi-spectral fundus imaging and exudate identification system according to claim 6 is characterized in that: The optical imaging system comprises: an eyepiece group 2, a multi-band light board 3, a dichroic prism group 5, a retina objective group 8, an alignment light group 6, a fixation light group 7, an infrared filter switch 9, a focus motor 10 and a CCD imaging device 4; The eyepiece group 2 is installed at the front end of the optical path and is aligned with the dichroic prism group 5; the dichroic prism group 5 is installed at a fixed angle on the imaging side behind the eyepiece group; the alignment lamp group 6, the fixation lamp group 7 and the multi-band lamp board 3 are installed on the same plane as the dichroic prism group 5; the retina objective lens group 8 is installed on the imaging side of the dichroic prism group 5; the CCD imaging device 4 is installed on the optical path behind the retina objective lens group 5; the infrared filter switch 9 is installed between the retina objective lens group 8 and the CCD imaging device; and the focus motor 10 is installed on the CCD imaging device; The fixed angle range is 20 degrees to 30 degrees; The eyepiece set is a pair of achromatic lenses with a focal length of 30 mm; The multi-band light panel includes LED light sources of different wavelengths; The LED light source includes white light, blue light, green light and red light; The achromatic lens of the retina objective lens group has a focal length of 15 mm and a magnification ratio of 1:1.25; The fundus image collected by the CCD imaging device is stored in a flash memory in the device and is connected to the outside for transmission via a USB interface.

8. The handheld multi-spectral fundus imaging and exudate identification system according to claim 6 is characterized in that: The handheld multi-spectrum light source is composed of a housing, a push-pull light source switching device and a multi-way power switching switch; The number of the switched light sources is set according to demand; The collection includes: Aim at the object to be measured and adjust the focus using the focus motor until the fundus image is clear; Collect fundus images under different spectra, and save and transmit multi-spectral fundus images to external storage; The exudate identification and quantification algorithm based on multispectral images includes image preprocessing, spectral mixing feature extraction, exudate segmentation and identification based on deep learning, and exudate area quantification calculation; The image preprocessing is multispectral image grayscale preprocessing; The image segmentation based on deep learning adopts UNet neural network; The exudate area quantification calculation is based on the segmentation mask, binarization and then using openCV to find the contour and obtain the maximum contour and its area.

9. The handheld multi-spectral fundus imaging and exudate identification system according to claim 6, characterized in that: The quantitative identification includes: Perform image preprocessing on fundus multispectral images to obtain multispectral information; Spectral mixing operation, effectively fusing multi-spectral information into feature maps; The feature map is loaded into the deep neural network, and the paired images of fundus exudates manually annotated by experts in the public dataset are subjected to the same image preprocessing and used as the supervised input network for model training to obtain a trained neural network; The trained neural network is used to segment and identify the exudates in the multispectral images one by one to obtain the prediction results; Binarize the predicted results, find the contours through openCV and get the largest contour and its area.

10. The handheld multi-spectral fundus imaging and exudate identification system according to claim 9, characterized in that: The spectral mixing operation includes: Define the collected multispectral image as I, I = {I1, I2, ..., I N }; Among them, I i represents the i-th spectral channel, i = 1, 2, ..., N; N represents the total number of spectral channels; Compress the spectral channel F = W·I and mix it. The feature map after mixing Among them, F represents the generated multi-channel feature map, the size of which is n×h×w; h and w represent the height and width of the image respectively; W represents the weight matrix, with a size of n×N; n is the total number of channels after compression; F j represents the weighted sum of all spectral channels; w ji represents the element in the jth row and ith column of the weight matrix W; j represents the jth output channel, j = 1, 2, ..., n); Training and initialization weight matrix W; The weight matrix W is set to be uniformly distributed when initialized; The mixed feature map F j Input to the encoder part of the neural network; The total number n of channels after compression is 3, 5 or 7; Constructing a joint loss function in, represents the multispectral image loss function of the kth spectral input, k = 1, 2, ..., M; M represents a multispectral image with M different spectra; λ k represents the weight of the kth spectral input; Y k represents the true label of the kth spectral input; Y′ k represents the predicted output for the kth spectral input.

Images