Fundus image processing method, fundus image processing device, fundus image processing program, and storage medium storing the program

Abstract

An ocular fundus image processing apparatus (1) includes: an image input unit (3) that acquires a first ocular fundus image generated by irradiating an ocular fundus of a subject with excitation light of a blue wavelength and a second ocular fundus image generated by irradiating the ocular fundus of the subject with excitation light of a green wavelength; a model generation unit (7) that generates three completed learning models for predicting correction coefficients for calculating a macular pigment amount of the subject from input images including the first ocular fundus image and the second ocular fundus image by training using three different initial values; a coefficient prediction unit (9) that predicts three correction coefficients by inputting the input images including the first ocular fundus image and the second ocular fundus image to the three completed learning models; a coefficient derivation unit (11) that calculates a statistical value using the three correction coefficients as a target and derives the correction coefficients of the subject as a statistical value; and a pigment amount calculation unit (13) that calculates the macular pigment amount of the subject based on the first ocular fundus image and the correction coefficients of the subject.

Description

Technical Field

[0001] One aspect of the implementation relates to a fundus image processing method, a fundus image processing apparatus, a fundus image processing program, and a storage medium storing the program. Background Technology

[0002] Devices for measuring macular pigment optical density (MPOD) using autofluorescence images of the fundus of a subject have been used for a long time. Measurement of MPOD, representing macular pigment density, is important for the prevention of age-related macular degeneration (AMD). However, when MPOD is measured in cataract patients, the MPOD value is underestimated due to reduced image quality of the autofluorescence image. Therefore, it is known that the measured MPOD value needs to be corrected to obtain an accurate value (see Non-Patent Literature 1 below).

[0003] Existing technical documents

[0004] Non-patent literature

[0005] Non-patent literature 1: A.Obana et al., "Grade of Cataract and Its Influence on Measurement of Macular Pigment Optical Density Using AutofluorescenceImaging", Investigative Ophthalmology&Visual Science, June 2018, Vol.59, 3011-3019 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Existing correction methods, such as those described above, utilize methods that reflect correction coefficients derived from subjective evaluations in the measured values ​​of MPOD, and methods based on multiple regression. However, these methods suffer from complex correction processes and insufficient reliability of the corrected MPOD.

[0008] Therefore, one aspect of the implementation is made in view of this problem, with the technical problem being to provide a fundus image processing method, fundus image processing apparatus, fundus image processing program, and storage medium for storing the program that can produce highly reliable MPODs through simple processing.

[0009] Technical means to solve the problem

[0010] One aspect of the implementation relates to a fundus image processing method. It includes: acquiring a first image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a first wavelength; acquiring a second image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength; using multiple different initial values, training to generate multiple trained deep learning models for predicting correction coefficients for calculating the amount of macular pigment in a subject from an input image containing at least the first and second images; predicting multiple correction coefficients by inputting the input image containing at least the first and second images into the multiple trained deep learning models; calculating statistical values ​​for the multiple correction coefficients and deriving the statistical values ​​as correction coefficients for the subject; and calculating the amount of macular pigment in the subject based on at least one of the first or second image and the subject's correction coefficients.

[0011] Alternatively, another aspect of the embodiment relates to a fundus image processing apparatus comprising: a first acquisition unit that acquires a first image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a first wavelength; a second acquisition unit that acquires a second image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength; a generation unit that uses multiple different initial values ​​to generate multiple learned deep learning models for predicting correction coefficients for calculating the amount of macular pigment in a subject from an input image containing at least the first image and the second image; a prediction unit that predicts multiple correction coefficients by inputting the input image containing at least the first image and the second image to the multiple learned deep learning models; a derivation unit that calculates statistical values ​​for the multiple correction coefficients and derives the statistical values ​​as the correction coefficients for the subject; and a calculation unit that calculates the amount of macular pigment in the subject based on at least one of the first image or the second image and the correction coefficients for the subject.

[0012] Alternatively, another aspect of the implementation involves a fundus image processing program in which the processor functions as follows: a first acquisition unit that acquires a first image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a first wavelength; a second acquisition unit that acquires a second image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength; a generation unit that uses multiple different initial values ​​to generate multiple learned deep learning models for predicting correction coefficients for calculating the amount of macular pigment in a subject from an input image containing at least the first image and the second image; a prediction unit that predicts multiple correction coefficients by inputting the input image containing at least the first image and the second image into the multiple learned deep learning models; a derivation unit that calculates statistical values ​​for the multiple correction coefficients and derives the statistical values ​​as the correction coefficients for the subject; and a calculation unit that calculates the amount of macular pigment in the subject based on at least one of the first image or the second image and the correction coefficients for the subject.

[0013] Alternatively, another aspect of the implementation is a computer-readable storage medium that stores fundus image processing programs.

[0014] According to one or the other of the above aspects, a fundus fluorescence image (i.e., a first image) obtained by excitation light of a first wavelength and a fundus fluorescence image (i.e., a second image) obtained by excitation light of a second wavelength are obtained. Using multiple different initial values, multiple trained deep learning models are generated to predict correction coefficients from an input image containing the first and second images. Then, by inputting the first and second images into the multiple trained deep learning models, multiple correction coefficients are predicted, and the statistical values ​​of the multiple correction coefficients are derived as correction coefficients for the subject. The macular pigment level of the subject is calculated based on the first or second image and its correction coefficients. Thus, multiple deep learning models are constructed using the first and second images as training data, and the first and second images of the subject are input into the multiple deep learning models to predict multiple correction coefficients. The macular pigment level is calculated by statistically evaluating the multiple correction coefficients. As a result, a highly reliable macular pigment level that reflects the trend of image quality in images of multiple subjects can be calculated through simple processing.

[0015] The effects of the invention

[0016] According to the implementation method, a highly reliable macular pigment density can be derived through simple processing. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating the functional structure of the fundus image processing apparatus 1 according to the embodiment.

[0018] Figure 2 It means as Figure 1 A block diagram of an example of the hardware structure of the computer 20 used in the fundus image processing device 1.

[0019] Figure 3 It means Figure 1 The diagram shows the network structure of the completed learning model LM generated by the model generation unit 7.

[0020] Figure 4 It means Figure 3 The diagram shows the coupling structure of the two fully coupled layers in the subsequent structure L2 of the completed learning model LM.

[0021] Figure 5 This is a flowchart illustrating the process of the training phase performed by the fundus image processing device 1.

[0022] Figure 6 This is a flowchart illustrating the process of the prediction phase performed by the fundus image processing device 1.

[0023] Figure 7 This is a diagram illustrating the structure of the fundus image processing program involved in the implementation method.

[0024] Figure 8 This is a graph showing the experimental results of the accuracy of the MPOD numerical series predicted according to this embodiment.

[0025] Figure 9 This is a graph showing the experimental results of the accuracy of the MPOD numerical series predicted according to this embodiment.

[0026] Figure 10 This is a diagram representing the network structure of the fully learned LM model used in the variant example.

[0027] Figure 11 It means Figure 10 The diagram shows the two-layer coupling structure in the subsequent stage L5 of the completed learning model LM. Detailed Implementation

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Furthermore, in the description, the same symbols are used for the same elements or elements having the same function, and repeated descriptions are omitted.

[0029] Figure 1 This is a block diagram illustrating the functional structure of the fundus image processing apparatus 1 according to this embodiment. For example... Figure 1As shown, the fundus image processing device 1 is a computing device that processes fundus images of a subject obtained using an external fundus analysis device 50 and calculates the macular pigment optical density (MPOD), which represents the amount of macular pigment in the subject. The fundus image processing device 1 can acquire fundus images from the fundus analysis device 50 via a wired or wireless communication network.

[0030] The fundus analysis device 50 is a known optical device for acquiring fundus images of a subject. It includes a built-in light source and imaging element (not shown), and has the function of irradiating the fundus of a subject with excitation light and acquiring the resulting autofluorescence image of the fundus as the fundus image. In this embodiment, the fundus analysis device 50 acquires a first fundus image obtained by irradiating the subject with blue excitation light of 486 nm and a second fundus image obtained by irradiating the subject with green excitation light of 518 nm. The wavelength of the excitation light used to acquire the first fundus image only needs to be within the range of 450–495 nm, which is the blue wavelength region, and the wavelength of the excitation light used to acquire the second fundus image only needs to be within the range of 495–570 nm, which is the green wavelength region.

[0031] Generally, the macula, located at the center of the retina, has the property of absorbing blue light. Therefore, in the fundus images of the subject obtained using the fundus resolving device 50, there are changes in the brightness distribution corresponding to the amount of macular pigment. MPOD has traditionally been calculated by comparing the average brightness value near the center (fovea) of the retina with that of the surrounding circular region based on the autofluorescence image obtained in response to blue light excitation.

[0032] The fundus image processing apparatus 1 calculates MPOD based on a first fundus image and a second fundus image obtained by the fundus analysis apparatus 50 on a subject. That is, the fundus image processing apparatus 1, as a functional structural element, is configured to include an image input unit (first acquisition unit, second acquisition unit) 3, a coefficient calculation unit 5, a model generation unit 7, a coefficient prediction unit 9, a coefficient derivation unit 11, a pigment quantity calculation unit 13, and a model storage unit 15.

[0033] Figure 2 This describes the hardware structure of the computer 20 that implements the fundus image processing apparatus 1 of this embodiment. For example... Figure 2As shown, computer 20 physically includes a CPU (Central Processing Unit) 101 as a processor, RAM (Random Access Memory) 102 or ROM (Read Only Memory) 103 as storage media, a communication module 104, and an input / output module 106, all electrically connected. Furthermore, computer 20 may include a display, keyboard, mouse, touch panel display, etc., as part of the input / output module 106, and may also include data storage devices such as hard disk drives and semiconductor memory. Moreover, computer 20 may be composed of multiple computers. Additionally, computer 20 may further include a GPU (Graphics Processing Unit) as a dedicated processor for image processing, and may be configured to perform parallel processing within the CPU 101 and GPU.

[0034] Figure 1 The functional units of the fundus image processing apparatus 1 shown are implemented as follows: a program (the fundus image processing program of this embodiment) is read into hardware such as the CPU 101 and RAM 102; and based on the control of the CPU 101, the communication module 104 and the input / output module 106 are operated, and data is read from and written to the RAM 102. The CPU 101 of the computer 20 executes this computer program, thereby enabling the computer 20 to function as a computer. Figure 1 Each functional unit performs its function, sequentially executing the processing corresponding to the fundus image processing method described below. All data required for the execution of this computer program, as well as all data generated through the execution of this computer program, are stored in built-in memory such as ROM103 and RAM102, or storage media such as hard disk drives.

[0035] Here, the functions of each functional unit of the fundus image processing device 1 will be explained in detail.

[0036] In the prediction phase for estimating the MPOD of a subject, the image input unit 3 acquires a first fundus image and a second fundus image (hereinafter, the fundus image set of the prediction subject) of the subject before cataract surgery. Furthermore, in the training phase for generating multiple learned deep learning models, the image input unit 3 acquires multiple sets (e.g., 148 sets) of first and second fundus images (hereinafter also referred to as the preoperative fundus image set) of the subject before cataract surgery, used as training data for various subjects or under various photographic conditions, combined with first and second fundus images (hereinafter also referred to as the postoperative fundus image set) of the same subject after cataract surgery. Furthermore, when the image input unit 3 obtains the fundus image set of the prediction object and the preoperative fundus image set, it performs differential analysis on the corresponding pixels between the first fundus image and the second fundus image contained in each fundus image set, generates a differential image by shifting the brightness value of all pixels in such a way that the minimum value of the differential brightness value becomes 0, and attaches the generated differential image to each fundus image set.

[0037] During the training phase, the coefficient calculation unit 5 calculates the correction factor (CF) value for correcting MPOD based on multiple combinations of preoperative and postoperative fundus image sets.

[0038] Specifically, the coefficient calculation unit 5 calculates MPOD by referring to at least one fundus image (preferably the first fundus image) from the preoperative fundus image set. The MPOD calculation is based on the brightness value I at a pre-specified pixel location near the center of the retina. min The average brightness value I of the pre-defined circular areas around it max(ave) The ratio is calculated according to the following formula.

[0039] MPOD = -1.4·log{I min / I max(ave)}

[0040] The coefficient calculation unit 5 calculates the MPOD at various positions with eccentricity of 0.23 degrees, 0.51 degrees, 0.98 degrees, and 1.99 degrees from the center of the retina (fovea), and sets it as the local MPOD. 0.23 Local MPOD 0.51 Local MPOD 0.98 Local MPOD 1.99Furthermore, the coefficient calculation unit 5 also calculates the total MPOD value for the region within an eccentric angle of 8.98 degrees from the center of the retina (fovea), and sets it as an MPOD storage volume. The coefficient calculation unit 5 groups the calculated MPOD values ​​into {local MPOD} 0.23 Local MPOD 0.51 Local MPOD 0.98 Local MPOD1 .99 The MPOD storage volume is set as an MPOD value column.

[0041] Similarly, the coefficient calculation unit 5 calculates the MPOD value column by referring to at least one fundus image (preferably the first fundus image) in the postoperative fundus image set.

[0042] Furthermore, the coefficient calculation unit 5 calculates the CF value for each correction from the preoperative MPOD value set to the postoperative MPOD value set, based on the MPOD value series calculated from the preoperative fundus image set (preoperative MPOD value series) and the MPOD value series calculated from the postoperative fundus image set (postoperative MPOD value series). For example, the CF value is calculated by dividing each postoperative MPOD value series by the preoperative MPOD value series, resulting in a group {CF...}. 0.23 CF 0.51 CF 0.98 CF 1.99 CF TOTAL (Hereinafter referred to as CF numeric column.)

[0043] The coefficient calculation unit 5 appends multiple sets of CF numerical values, calculated as described above, to multiple corresponding preoperative fundus image sets used as training data in the model generation unit 7, and then submits them to the model generation unit 7. These CF numerical values ​​are also used as teacher data (labels) during the training (supervised learning) in the model generation unit 7.

[0044] During the training phase, the model generation unit 7 generates multiple learned deep learning models by training on multiple preoperative fundus image sets. These models predict CF (Cyclic Fibre Channel) values ​​representing correction coefficients used to calculate the macular pigmentation of a subject using at least one image from the subject's fundus image set as input. Specifically, the model generation unit 7 generates three learned models with a Convolutional Neural Network (CNN) structure that uses a first fundus image, a second fundus image, and a difference image from the subject's fundus image set as input images. The model generation unit 7 stores data MD1, MD2, and MD3, including parameters used to enable the three learned models to perform actions, in the model storage unit 15.

[0045] exist Figure 3 In the diagram, the network structure of the learned model LM generated by the model generation unit 7 is represented. The learned model LM consists of an input layer L0 with a set of fundus images of the target object as input in 3ch; a pre-stage structure L1 after the input layer L0, which alternately connects convolutional layers and pooling layers in multiple groups; a post-stage structure L2 with two fully coupled layers sequentially connected to the post-stage structure L1; and an output layer L3 that converts the data of each node from the post-stage structure L2 onwards into a final CF numerical column as a vector output. Such a learned model LM can use a large-scale image database such as ImageNet, based on the structure of a pre-trained CNN, and change the number of nodes in the output layer L3 to the desired number (5 in this embodiment). In the parameters of the learned model LM, except for the last fully coupled layer, the pre-trained parameters (such as the parameters of the weight filter) are used as initial values ​​(assuming the application of transfer learning).

[0046] The model generation unit 7 uses multiple preoperative fundus image sets as training data and the corresponding CF value columns as teacher data, training based on three different initial values ​​to generate three fully learned models. Specifically, before training, the model generation unit 7 generates a pseudo-random number series by inputting three random number seeds (pre-set initial values) into a pseudo-random number generator within the computer 20, and initializes the parameters of the final fully coupled layer in the fully learned LM model of the CNN based on this pseudo-random number series. For example, as... Figure 4 As shown, the coupling weight w between nodes i and j (where i and j are arbitrary integers) in the two fully coupled layers of the subsequent structure L2 of the learned model LM is... i，j The initial values ​​are set based on a series of pseudo-random numbers. Through this random initialization of the learned model LM, the optimization results of the three learned models generated through training depend on the initial values; therefore, the prediction results using the three learned models will be slightly different.

[0047] Furthermore, the model generation unit 7 can replace the parameter initialization of the learned model LM based on a pseudo-random number series as described above, or, above the parameter initialization, randomly change the order of the training data prompts based on a pseudo-random number series in each loop (training round) where all training data is prompted (input) to the learned model LM. This prevents the parameters of the learned model from getting trapped in local solutions during training. Moreover, when training the learned model as a CNN, data augmentation (data argumentation) can be performed by cropping random positions near the center of the prompted input image to increase the variation of the input image. The aforementioned function of the model generation unit 7 can also be used to vary the data augmentation effect among the three learned models.

[0048] The coefficient prediction unit 9 predicts three sets of CF values ​​for the subject by using three completed learning models generated during the training phase, respectively, as input from a set of preoperative fundus images of the subject. At this time, the coefficient prediction unit 9 reads the data MD1, MD2, and MD3 required to perform the prediction processing using the three completed learning models from the model storage unit 15.

[0049] In the prediction phase, the coefficient derivation unit 11 calculates statistical values ​​from the three sets of CF value columns predicted by the coefficient prediction unit 9 with the subjects as the subjects, and derives the statistical values ​​as the final CF value columns for the subjects. For example, the coefficient derivation unit 11 calculates three sets of CF value columns {CF... 0.23 CF 0.51 CF 0.98 CF 1.99 CF TOTAL The average of their respective values ​​{CFA} 0.23 CFA 0.51 CFA 0.98 CFA 1.99 CFA TOTAL The average values ​​of these values ​​are used as the final CF (Computed Tomography) values. By using the average of the predicted values ​​as the final predicted values ​​(through ensemble processing), the dependence of the initial values ​​on the optimization results of deep learning can be mitigated, thereby improving the prediction accuracy of the corrected values.

[0050] In the prediction phase, the pigment quantity calculation unit 13 uses the final CF value column derived by the coefficient derivation unit 11 to calculate the subject's MPOD value column and outputs it to the input / output module 106. Figure 2 ) output. Furthermore, the pigment quantity calculation unit 13 can also output the calculated MPOD values ​​via the communication module 104. Figure 2The pigment quantity calculation unit 13 sends the data to an external device. Specifically, it refers to the fundus image (preferably the first fundus image) of at least one of the fundus images in the set of fundus images of the subject, and calculates the MPOD value column using the same calculation method as the coefficient calculation unit 5. Then, the pigment quantity calculation unit 13 calculates the subject's MPOD value column by correcting the calculated MPOD value column using the CF value column. At this time, the pigment quantity calculation unit 13 calculates the subject's MPOD value column by correcting the calculated MPOD value column {local MPOD...} 0.23 Local MPOD 0.51 Local MPOD 0.98 Local MPOD 1.99 Each value contained in the MPOD storage volume is multiplied by the CF value column {CFA}. 0.23 CFA 0.51 CFA 0.98 CFA 1.99 CFA TOTAL The corresponding correction values ​​included in} are used to calculate the MPOD value column of the subject.

[0051] Next, the process of predicting MPOD in a subject using the fundus image processing device 1 according to this embodiment, i.e., the process of the fundus image processing method according to this embodiment, will be described. Figure 5 This is a flowchart illustrating the process of the training phase performed by the fundus image processing device 1. Figure 6 This is a flowchart illustrating the process of the prediction phase performed by the fundus image processing device 1.

[0052] First, at the start of the training phase, corresponding to operator input instructions from the fundus image processing device 1, the image input unit 3 acquires multiple sets of preoperative and postoperative fundus image sets (step S101). Correspondingly, the coefficient calculation unit 5 calculates MPOD value columns for both the preoperative and postoperative fundus image sets, and calculates CF value columns based on these value columns (step S102). The calculation of CF value columns is repeatedly performed on all sets of both the preoperative and postoperative fundus image sets, and the calculated CF value columns are appended to the corresponding preoperative fundus image set as teacher data.

[0053] Next, the model generation unit 7 generates pseudo-random numbers using a pre-set random number seed (step S103). Then, the model generation unit 7 sets the initial values ​​of the CNN based on the pseudo-random numbers and changes the order of the training data prompts based on the pseudo-random numbers, thereby generating a deep learning model that has completed learning through training (step S104). Furthermore, the model generation unit 7 stores the data of the generated deep learning model that has completed learning in the model storage unit 15 (step S105).

[0054] The above steps S103 to S105 are repeated 3 times while setting 3 random number seeds (step S106). As a result, 3 deep learning models that have completed learning are generated and stored.

[0055] Next, when the prediction phase for the subject begins in accordance with the operator's instructions to the fundus image processing device 1, the image input unit 3 acquires a set of fundus images of the subject to be predicted (step S201). Then, the coefficient prediction unit 9 inputs the set of fundus images of the subject to be predicted as input images to three learned deep learning models, thereby predicting three sets of CF value sequences (step S202).

[0056] Next, the coefficient derivation unit 11 derives the subject's final CF value series by calculating the statistical values ​​of the three predicted CF value series (step S203). Then, the pigment quantity calculation unit 13 calculates the MPOD value series based on the first and second fundus images included in the subject's fundus image set (step S204). Finally, the pigment quantity calculation unit 13 corrects the MPOD value series using the CF value series and outputs the corrected MPOD value series (step S205).

[0057] Next, refer to Figure 7 This describes the structure of the fundus image processing program that enables the computer 20 to function as the fundus image processing device 1 described above.

[0058] The fundus image processing program P1 includes a main module P10, an image input module P15, a coefficient calculation module P16, a model generation module P17, a coefficient prediction module P18, a coefficient export module P19, and a pigment amount calculation module P20.

[0059] The main module P10 is the part that comprehensively controls the processing of fundus images. The functions implemented by executing the image input module P15, coefficient calculation module P16, model generation module P17, coefficient prediction module P18, coefficient derivation module P19, and pigment amount calculation module P20 are the same as the functions of the image input unit 3, coefficient calculation unit 5, model generation unit 7, coefficient prediction unit 9, coefficient derivation unit 11, and pigment amount calculation unit 13 of the fundus image processing device 1.

[0060] The fundus image processing program P1 is provided, for example, by a storage medium such as a CD-ROM, DVD, or ROM, or by a semiconductor memory. Alternatively, the fundus image processing program P1 can also be provided via a network as a computer data signal superimposed on a carrier wave.

[0061] According to the fundus image processing apparatus 1 described above, multiple sets of first fundus images obtained by excitation light containing a blue wavelength region and second fundus images obtained by excitation light containing a green wavelength region are acquired. Using three different initial values, three learned deep learning models are generated to predict CF value columns from input images containing the first and second fundus images. Then, input images containing the first and second fundus images, obtained from subjects before cataract surgery, are input to the three learned deep learning models, thereby predicting three CF value columns. The average of the statistical values ​​of the three CF value columns is derived as the subject's final CF value columns. The subject's MPOD value column is calculated based on the MPOD value column calculated from the first and second fundus images and the CF value columns. Therefore, three deep learning models were constructed using the first and second fundus images as training data. The first and second fundus images of the subjects were then input into the three deep learning models to predict three CF (figurine flow) value columns. The macular pigment level was calculated by statistically evaluating these three CF value columns. The result is a highly reliable calculation of macular pigment levels that reflects the trend of image quality changes across multiple subjects, using simple processing.

[0062] In this embodiment, in particular, the training phase is performed while the random number seed is changed. In this case, the initialization of the CNN parameters, the order of the training data prompts, and the effect of data augmentation will vary depending on the random number seed. Since the optimal initial values ​​of the CNN parameters, the optimal order of the training data prompts, or the optimal data augmentation method are unknown, it is possible to obtain a prediction value close to the optimal training result of the CNN by averaging the prediction values ​​of the learned CNN trained with various training seeds.

[0063] Furthermore, in this embodiment, excitation light with wavelengths within the blue wavelength region is used to acquire the first fundus image, and excitation light with wavelengths within the green wavelength region is used to acquire the second fundus image. Since the macula has the property of absorbing blue light, the amount of macular pigment can be calculated with high precision by using the two images generated by excitation light of such wavelengths as input images.

[0064] Furthermore, in this embodiment, the input image to the CNN includes a difference image. In this case, the amount of macular pigment can be calculated with higher accuracy.

[0065] Furthermore, in this embodiment, the average of three predicted values ​​from three learned deep learning models is used as the statistical value. In this case, the predicted values ​​from the three learned deep learning models can be reflected equally when calculating the amount of macular pigment, thus enabling a more reliable calculation of the amount of macular pigment.

[0066] Furthermore, in this embodiment, pseudo-random numbers generated from three random number seeds are used to train the deep learning model. Based on this structure, even with training using a limited number of input images, it is possible to comprehensively generate three deep learning models for predicting CF numerical columns.

[0067] Furthermore, in this embodiment, pseudo-random numbers are used to initialize the parameters of the deep learning model or to change the order of the input images fed into the deep learning model. Thus, even when training with a limited number of input images, multiple deep learning models for predicting CF numerical columns can be comprehensively generated. As a result, the accuracy of predicting macular pigmentation based on statistical values ​​can be improved.

[0068] Here, experimental results are shown on the accuracy of the MPOD numerical series predicted using this embodiment.

[0069] exist Figure 8 In the middle, the MPOD value column predicted using this embodiment will contain each value {local MPOD}. 0.23 Local MPOD 0.51 Local MPOD 0.98 Local MPOD 1.99 The average error of each MPOD storage volume is compared with the case without correction using the CF numerical series and the case corrected using an existing multiple regression method (described in "A. Obana et al., 'Grade of Cataract and Its Influence on Measurement of Macular Pigment Optical Density Using Autofluorescence Imaging', Investigative Ophthalmology & Visual Science, June 2018, Vol. 59, 3011-3019"). Thus, compared with the multiple regression-based method, it can be seen that the overall error is improved in this embodiment. In particular, the error of the MPOD storage volume is significantly improved from 14.81% to 7.81%.

[0070] exist Figure 9In this study, the following scenarios were considered: using only the second fundus image as the input image for 1ch without any ensemble processing (Example 1); using only the difference image as the input image for 1ch without any ensemble processing (Example 2); using only the first fundus image as the input image for 1ch without any ensemble processing (Example 3); using 3ch input images without any ensemble processing (Example 4); using only the second fundus image as the input image for 1ch with ensemble processing (Example 5); using only the difference image as the input image for 1ch with ensemble processing (Example 6); using only the first fundus image as the input image for 1ch with ensemble processing (Example 7); and using 3ch input images with ensemble processing (Example 8). These scenarios represent the average error of each value in the MPOD numerical column. The experimental results show that the error is reduced with 3ch input regardless of whether ensemble processing is used. Furthermore, the error is minimized when ensemble processing is used with 3ch input. These results demonstrate that both 3ch input and ensemble processing are important in reducing the error of the predicted values.

[0071] The various embodiments of the present invention have been described above, but the present invention is not limited to the above embodiments. It can also be modified or applied to other ways without changing its spirit.

[0072] The image input unit 3 of the fundus image processing apparatus 1 described above acquires a difference image using a set of fundus images of the prediction target and a set of preoperative fundus images as the target, and uses the difference image in the training and prediction phases. As a variation, an addition image can also be acquired using these fundus image sets, in which the brightness values ​​of each pixel are added between the first and second fundus images, and used in the training and prediction phases.

[0073] Furthermore, the coefficient derivation unit 11 of the fundus image processing apparatus 1 in the above embodiment calculates the average value as a statistical value of the three CF value columns, but it can also calculate the median value. In this case, it is also possible to perform high-precision prediction of macular pigmentation.

[0074] Furthermore, the structure of the deep learning model used for predicting CF numerical columns in the fundus image processing apparatus 1 of the above embodiments is not limited to... Figure 3 The structure shown. For example, it could also be... Figure 10 The structure of the completed learning model LM is shown.

[0075] Figure 10The completed learning model LM shown, as a subsequent structure L4 connected to the preceding structure L1, has a structure with convolutional layers and global average pooling layers connected in sequence. In the parameters of the completed learning model LM with such a structure, the parameters (such as the parameters of the weight filters) are used as initial values ​​outside the convolutional layers of the subsequent structure L4.

[0076] In Adoption Figure 10 In a modified example of the structure of the learned model LM shown, the model generation unit 7 randomly changes the initial values ​​of the parameters of the convolutional layer of the subsequent structure L4 based on a series of pseudo-random numbers. Specifically, as... Figure 11 As shown, the model generation unit 7 sets the weight coefficients w of the filters applied in the convolutional layer of the subsequent structure L4 according to a pseudo-random number series. p The initial value. Based on this variation, it is also possible to comprehensively generate multiple deep learning models for predicting CF numerical columns.

[0077] In the above-described embodiment, preferably, one of the first wavelength and the second wavelength is a wavelength within the blue region, and the other of the first wavelength and the second wavelength is a wavelength within the green region. Since the macula has the property of absorbing blue light, by using two images of excitation light with such wavelengths as input images, the amount of macular pigment can be calculated with high precision.

[0078] Furthermore, it is also preferable that the input image also includes a difference image or an additive image based on the first image and the second image. In this case, by including a difference image or an additive image based on the two images in the input image, the amount of macular pigment can be calculated with higher accuracy.

[0079] Furthermore, the preferred statistical value is the average or median of multiple correction coefficients. In this case, multiple correction coefficients predicted using multiple learned deep learning models can be reflected equally, and the amount of macular pigment can be calculated, thus enabling a more reliable calculation of macular pigment.

[0080] Furthermore, in the above embodiments, it is also preferable to train the deep learning model using pseudo-random numbers generated based on multiple different initial values. According to this structure, even when training with a limited number of input images, it is possible to comprehensively generate multiple deep learning models for predicting correction coefficients.

[0081] Furthermore, in the above embodiments, it is preferable to use pseudo-random numbers in the initialization of the parameters of the deep learning model, and it is also preferable to change the order of the input images to the deep learning model based on pseudo-random numbers. In this way, even when training with a limited number of input images, multiple deep learning models for predicting correction coefficients can be comprehensively generated.

[0082] Industrial availability

[0083] The implementation method can be used to process fundus image processing method, fundus image processing device, fundus image processing program and storage medium storing the program, and can produce highly reliable MPOD through simple processing.

[0084] Explanation of symbols

[0085] 1…Fundus image processing device, P1…Fundus image processing program, 3…Image input unit (first acquisition unit, second acquisition unit), 7…Model generation unit, 9…Coefficient prediction unit, 11…Coefficient derivation unit, 13…Pigment quantity calculation unit, 20…Computer, 101…CPU (processor), LM…Completed learning model.

Claims

1. A method for processing fundus images, characterized in that, include: The step of obtaining a first image of a fluorescein image of the fundus produced by irradiating the fundus of a subject with excitation light of a first wavelength; The step of obtaining a second image of the fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength; The steps of obtaining a set of multiple preoperative fundus images, including a first fundus image generated by irradiating the fundus of multiple subjects before cataract surgery with excitation light of the first wavelength and a second fundus image generated by irradiating the fundus of the multiple subjects before cataract surgery with excitation light of the second wavelength. The steps of obtaining a set of multiple postoperative fundus images, including a first fundus image generated by irradiating the fundus of the multiple subjects after cataract surgery with excitation light of the first wavelength and a second fundus image generated by irradiating the fundus of the multiple subjects after cataract surgery with excitation light of the second wavelength. The step of calculating correction coefficients for each combination of the plurality of preoperative fundus image sets and the plurality of postoperative fundus image sets; Using multiple different initial values ​​and the correction coefficients as teacher data, the steps involve training multiple learned deep learning models to predict correction coefficients for calculating the macular pigmentation of the subject from an input image containing at least the first image and the second image. The step of predicting multiple correction coefficients by inputting an input image containing at least the first image and the second image into the plurality of learned deep learning models; The steps of calculating statistical values ​​using the plurality of correction coefficients as objects, and deriving the statistical values ​​as the correction coefficients for the subjects; and The step of calculating the macular pigmentation of the subject based on at least one of the first image or the second image, and the correction coefficient of the subject. One of the first wavelength and the second wavelength is a wavelength within the blue region, and the other of the first wavelength and the second wavelength is a wavelength within the green region.

2. The fundus image processing method as described in claim 1, characterized in that, The input image also includes a difference image or an addition operation image based on the first image and the second image.

3. The fundus image processing method as described in claim 1, characterized in that, The statistical value is the average or median value of the plurality of correction coefficients.

4. The fundus image processing method as described in claim 2, characterized in that, The statistical value is the average or median value of the plurality of correction coefficients.

5. The fundus image processing method according to any one of claims 1 to 4, characterized in that, In the generation step, a deep learning model is trained using pseudo-random numbers generated based on the multiple different initial values.

6. The fundus image processing method as described in claim 5, characterized in that, In the generation step, the pseudo-random number is used to initialize the parameters of the deep learning model.

7. The fundus image processing method as described in claim 5, characterized in that, In the generation step, the order of the input images to the deep learning model is changed based on the pseudo-random number.

8. The fundus image processing method as described in claim 6, characterized in that, In the generation step, the order of the input images to the deep learning model is changed based on the pseudo-random number.

9. A fundus image processing device, characterized in that, include: The image input unit acquires a first image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a first wavelength, acquires a second image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength, and acquires a plurality of preoperative fundus image sets consisting of a first fundus image of a fundus fluorescence image generated by irradiating the fundus of a plurality of subjects before cataract surgery with excitation light of the first wavelength and a second fundus image of a fundus fluorescence image generated by irradiating the fundus of the plurality of subjects before cataract surgery with excitation light of the second wavelength, and acquires a plurality of postoperative fundus image sets consisting of a first fundus image of a fundus fluorescence image generated by irradiating the fundus of a plurality of subjects after cataract surgery with excitation light of the first wavelength and a second fundus image of a fundus fluorescence image generated by irradiating the fundus of the plurality of subjects after cataract surgery with excitation light of the second wavelength. The coefficient calculation unit calculates correction coefficients for each combination of the plurality of preoperative fundus image sets and the plurality of postoperative fundus image sets. The generation unit, using multiple different initial values ​​and the correction coefficients as teacher data, generates multiple learned deep learning models by training to predict correction coefficients for calculating the amount of macular pigment in the subject from an input image containing at least the first image and the second image. The prediction unit predicts multiple correction coefficients by inputting an input image containing at least the first image and the second image into the plurality of learned deep learning models. The derivation unit calculates statistical values ​​using the plurality of correction coefficients as objects, and derives the statistical values ​​as the correction coefficients for the subject; and The computing unit calculates the amount of macular pigment in the subject based on at least one of the first image or the second image, and the correction coefficient of the subject. One of the first wavelength and the second wavelength is a wavelength within the blue region, and the other of the first wavelength and the second wavelength is a wavelength within the green region.

10. The fundus image processing apparatus as described in claim 9, characterized in that, The input image also includes a difference image or an addition operation image based on the first image and the second image.

11. The fundus image processing apparatus as described in claim 9, characterized in that, The statistical value is the average or median value of the plurality of correction coefficients.

12. The fundus image processing apparatus as described in claim 10, characterized in that, The statistical value is the average or median value of the plurality of correction coefficients.

13. The fundus image processing apparatus according to any one of claims 9 to 12, characterized in that, The generation unit uses pseudo-random numbers generated based on the multiple different initial values ​​to train a deep learning model.

14. The fundus image processing apparatus as described in claim 13, characterized in that, The generation unit uses the pseudo-random numbers to initialize the parameters of the deep learning model.

15. The fundus image processing apparatus as described in claim 13, characterized in that, The generation unit changes the order of the input images fed into the deep learning model based on the pseudo-random numbers.

16. The fundus image processing apparatus as described in claim 14, characterized in that, The generation unit changes the order of the input images fed into the deep learning model based on the pseudo-random numbers.

17. A computer-readable storage medium, characterized in that, Store fundus image processing program, The fundus image processing program enables the processor to function as the following components: The image input unit acquires a first image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a first wavelength, acquires a second image of a fundus fluorescence image generated by irradiating the fundus of a subject with excitation light of a second wavelength different from the first wavelength, and acquires a plurality of preoperative fundus image sets consisting of a first fundus image of a fundus fluorescence image generated by irradiating the fundus of a plurality of subjects before cataract surgery with excitation light of the first wavelength and a second fundus image of a fundus fluorescence image generated by irradiating the fundus of the plurality of subjects before cataract surgery with excitation light of the second wavelength, and acquires a plurality of postoperative fundus image sets consisting of a first fundus image of a fundus fluorescence image generated by irradiating the fundus of a plurality of subjects after cataract surgery with excitation light of the first wavelength and a second fundus image of a fundus fluorescence image generated by irradiating the fundus of the plurality of subjects after cataract surgery with excitation light of the second wavelength. The coefficient calculation unit calculates correction coefficients for each combination of the plurality of preoperative fundus image sets and the plurality of postoperative fundus image sets. The generation unit, using multiple different initial values ​​and the correction coefficients as teacher data, generates multiple learned deep learning models by training to predict correction coefficients for calculating the amount of macular pigment in the subject from an input image containing at least the first image and the second image. The prediction unit predicts multiple correction coefficients by inputting an input image containing at least the first image and the second image into the plurality of learned deep learning models. The derivation unit calculates statistical values ​​using the plurality of correction coefficients as objects, and derives the statistical values ​​as the correction coefficients of the subject; and The computing unit calculates the amount of macular pigment in the subject based on at least one of the first image or the second image, and the correction coefficient of the subject. One of the first wavelength and the second wavelength is a wavelength within the blue region, and the other of the first wavelength and the second wavelength is a wavelength within the green region.

Images