Eye disease prediction method and device, electronic equipment and readable storage medium

Abstract

The invention provides an eye disease prediction method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring an eye fundus image and an anterior segment optical coherence tomography image of the eyes of a user; inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model; wherein the trained eye disease prediction model is used for extracting a first eye feature from the fundus image and extracting a second eye feature from the anterior segment optical coherence tomography image, fusing the first eye feature and the second eye feature to obtain an eye fusion feature, and outputting an eye disease prediction result according to the eye fusion feature. According to the method, the neural network obtains the complementary information of the multi-modal data, the eye disease is predicted according to the complementary information of the multi-modal data, and the accuracy of judging the eye disease risk is improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to an eye disease prediction method, device, electronic device and readable storage medium. Background Art

[0002] Currently, the quality of eye care services varies due to a shortage of labour, as well as an aging population and changing lifestyles (less time spent outdoors, increasingly sedentary lifestyles and unhealthy eating habits), which are leading to an increase in the number of people suffering from eye diseases and vision impairment.

[0003] Doctors diagnose eye diseases through fundus examination or optical coherence tomography (OCT) examination. However, this diagnostic method may misjudge the patient's condition and miss patients with eye diseases. Summary of the invention

[0004] The embodiments of the present application provide an eye disease prediction method, device, electronic device, readable storage medium and computer program product, which can solve the problem of low accuracy in diagnosing eye diseases.

[0005] In a first aspect, an embodiment of the present application provides an eye disease prediction method, comprising:

[0006] Acquire a fundus image and an anterior segment optical coherence tomography image of the user's eye;

[0007] Inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model;

[0008] Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature.

[0009] In one embodiment, the trained eye disease prediction model includes a trained first extraction network, a trained second extraction network, and a trained prediction network;

[0010] Inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model, comprising:

[0011] Inputting the fundus image into the trained first extraction network to obtain the first eye feature output by the trained first extraction network;

[0012] Inputting the anterior segment optical coherence tomography image into the trained second extraction network to obtain the second eye feature output by the trained second extraction network;

[0013] The first eye feature and the second eye feature are input into the trained prediction network to obtain the prediction result output by the trained prediction network, the trained prediction network is used to convert the first eye feature into a first eye feature vector and the second eye feature into a second eye feature vector, then fuse the first eye feature vector and the second eye feature vector to obtain an eye fusion feature vector, and output the eye disease prediction result according to the eye fusion feature.

[0014] In one embodiment, the trained prediction network includes a trained multi-head attention mechanism network;

[0015] The trained multi-head attention mechanism network is used to: map the first eye feature vector to obtain a first mapping vector; map the second eye feature vector to obtain a second mapping vector and a third mapping vector; and based on the number of attention heads, fuse the first mapping vector, the second mapping vector and the third mapping vector to obtain the eye fusion feature vector.

[0016] In one embodiment, before inputting the fundus image and the anterior segment optical coherence tomography image into the trained eye disease prediction model, the method further includes:

[0017] Acquire image training samples, wherein the image training samples include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, wherein the first label is used to characterize the eye disease condition of the sample;

[0018] Inputting the fundus image sample into a first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into a prediction network;

[0019] Inputting the anterior segment optical coherence tomography image sample into a second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network;

[0020] After converting the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through the prediction network, fusing the first sample feature vector and the second sample feature vector to obtain a sample fusion feature vector, and predicting and outputting a sample result according to the sample fusion feature vector to obtain the sample result;

[0021] Determining a loss value of a first loss function of the eye disease prediction model according to the sample result and the first label;

[0022] If the loss value of the first loss function is greater than a first preset loss value, then after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value;

[0023] If the loss value of the first loss function is less than the first preset loss value, obtaining the trained eye disease prediction model;

[0024] The eye disease prediction model includes the first extraction network, the second extraction network and the prediction network.

[0025] In one embodiment, the image training sample further includes a second label of the fundus image sample, and the second label is used to represent the true segmentation result of the optic cup and optic disc;

[0026] After converting the first sample feature into a first sample feature vector through the prediction network, the method further includes:

[0027] Segmenting the fundus image sample according to the first sample feature vector to obtain a segmentation prediction result of the fundus image sample;

[0028] Determining a loss value of a second preset loss function of the eye disease prediction model according to the segmentation prediction result and the second label;

[0029] If the loss value of the total loss function is greater than the second preset loss value, then after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the total loss function is less than the second preset loss value;

[0030] If the loss value of the total loss function is less than the second preset loss value, obtaining the trained eye disease prediction model;

[0031] The total loss function includes the first preset loss function and the second preset loss function.

[0032] In one embodiment, the first eye feature is an appearance feature of an optic cup and optic disc region in the fundus image, and the second eye feature is an appearance feature of an anterior chamber angle region in the anterior segment optical coherence tomography image.

[0033] In a second aspect, an embodiment of the present application provides an eye disease prediction device, comprising:

[0034] An acquisition module, used to acquire a fundus image and anterior segment optical coherence tomography image of a user's eye;

[0035] A prediction module, used for inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model;

[0036] Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature.

[0037] In one embodiment, the apparatus further comprises a training module;

[0038] The acquisition module is further used to acquire image training samples, wherein the image training samples include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, wherein the first label is used to characterize the eye disease condition of the sample;

[0039] A training module, used for inputting the fundus image sample into a first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into a prediction network;

[0040] Also used for inputting the anterior segment optical coherence tomography image sample into a second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network;

[0041] The method is further configured to, after converting the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through the prediction network, fuse the first sample feature vector and the second sample feature vector to obtain a sample fusion feature vector, and predict and output a sample result according to the sample fusion feature vector to obtain the sample result;

[0042] It is also used to determine the loss value of the first loss function of the eye disease prediction model according to the sample result and the first label;

[0043] Also used for, if the loss value of the first loss function is greater than a first preset loss value, adjusting the parameters of the eye disease prediction model, and continuing to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value;

[0044] Also used for obtaining the trained eye disease prediction model if the loss value of the first loss function is less than the first preset loss value;

[0045] The eye disease prediction model includes the first extraction network, the second extraction network and the prediction network.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in any one of the above-mentioned first aspects when executing the computer program.

[0047] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0048] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] The embodiment of the present application obtains a fundus image and an anterior segment optical coherence tomography image of the user's eye; inputs the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model; wherein the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature, thereby enabling a neural network to obtain complementary information of multimodal data, and predict eye diseases based on the complementary information of multimodal data, thereby improving the accuracy of judging eye disease risks.

[0051] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a first flow chart of an eye disease prediction method provided by an embodiment of the present application;

[0054] Figure 2 This is a second flow chart of the eye disease prediction method provided by an embodiment of the present application;

[0055] Figure 3 This is a third flow chart of the eye disease prediction method provided in one embodiment of the present application;

[0056] Figure 4 is a schematic diagram of the structure of an eye disease prediction device provided in one embodiment of the present application;

[0057] Figure 5 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0058] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0059] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0060] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0061] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0062] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0063] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0064] Doctors diagnose eye diseases through fundus examination or optical coherence tomography (OCT) examination. Fundus images can be taken by a fundus camera, which can clearly show physiological structures such as the optic cup, optic disc, and blood vessels. Optical coherence tomography images can be obtained by cross-sectional scanning with near-infrared light. The optical coherence tomography images include structures at different levels of the retinal cross section. Among them, anterior segment optical coherence tomography is an effective means of diagnosing eye diseases.

[0065] In one embodiment, Figure 1 This is a schematic diagram of the first flow chart of the eye disease prediction method provided by an embodiment of the present application. Figure 1 As shown, the method comprises:

[0066] S11: Acquire a fundus image and an anterior segment optical coherence tomography image of the user's eyes.

[0067] In the application, the user is the subject to be diagnosed, and the fundus image and anterior segment optical coherence tomography image of the user's current eye are obtained.

[0068] S12: Inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model.

[0069] Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output an eye disease prediction result based on the eye fusion feature.

[0070] In the application, the eye disease prediction model mines the complementary information between the two images to obtain the first eye feature and the second eye feature. The two features are fused to obtain the eye fusion feature. Prediction is performed based on the eye fusion feature to obtain accurate eye disease prediction results.

[0071] The eye disease prediction result may be an eye disease risk, which is expressed as a probability, or the eye disease prediction result may be an eye disease result, which is expressed as a numerical value.

[0072] For example, glaucoma is diagnosed based on fundus images and anterior segment optical coherence tomography images. Glaucoma usually causes structural changes in the optic disc region of the eye, characterized by a larger ratio of the vertical optic disc and optic cup diameters in the fundus image. Glaucoma usually causes structural changes in the anterior segment of the eye, characterized by the structure and density of the anterior chamber angle. The trained eye disease prediction model extracts features related to glaucoma and in a complementary relationship from two images, namely, a first eye feature and a second eye feature, and then fuses the features to obtain an eye fusion feature; and makes predictions based on the eye fusion feature to obtain an eye disease prediction result.

[0073] The principles of diagnosing cataracts based on fundus images and anterior segment optical coherence tomography images are similar and will not be elaborated here.

[0074] It is understandable that accurate prediction results can be obtained by using multimodal data: fundus images and anterior segment optical coherence tomography images for prediction. The eye disease prediction model obtains complementary information between the two images for fusion, and makes predictions based on the fused features to improve the accuracy of the prediction and obtain more accurate prediction results. The accuracy of the prediction can be improved through appropriate fusion methods.

[0075] This embodiment obtains a fundus image and an anterior segment optical coherence tomography image of the user's eyes; inputs the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model; wherein the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature, thereby enabling a neural network to obtain complementary information of multimodal data, and predict eye diseases based on the complementary information of multimodal data, thereby improving the accuracy of judging eye disease risks.

[0076] In one embodiment, the trained eye disease prediction model includes a trained first extraction network, a trained second extraction network, and a trained prediction network.

[0077] Step S12 includes:

[0078] S121: Inputting the fundus image into the trained first extraction network to obtain the first eye feature output by the trained first extraction network.

[0079] In a possible implementation manner, the first eye feature is an appearance feature of an optic cup and optic disc region in a fundus image.

[0080] In the application, an image of the optic cup and optic disc region in the fundus image is input into the trained first extraction network. The trained first extraction network extracts detailed appearance features, namely, first eye features, from the image, and then the first eye features are input into the trained prediction network.

[0081] S122: Input the anterior segment optical coherence tomography image into the trained second extraction network to obtain a second eye feature output by the trained second extraction network.

[0082] In a possible implementation manner, the second eye feature is an appearance feature of the anterior chamber angle region in an anterior segment optical coherence tomography image.

[0083] In the application, an image of the anterior chamber angle region in an anterior segment optical coherence tomography image is input into the trained second extraction network. The trained second extraction network extracts detailed appearance features, i.e., second eye features, from the image, and then the second eye features are input into the trained prediction network.

[0084] The first extraction network and the second extraction network may be convolutional neural networks. The appearance features rich in details are features that the network can learn rich information.

[0085] S123: Inputting the first eye feature and the second eye feature into the trained prediction network to obtain a prediction result output by the trained prediction network.

[0086] Among them, the trained prediction network is used to convert the first eye feature into a first eye feature vector and the second eye feature into a second eye feature vector, fuse the first eye feature vector and the second eye feature vector to obtain an eye fusion feature vector, and output an eye disease prediction result based on the eye fusion feature.

[0087] In a possible implementation, the trained prediction model includes a trained first conversion network layer and a trained second conversion network layer. The trained first conversion network layer is used to convert the first eye feature and output a first eye feature vector. The trained second conversion network layer is used to convert the second eye feature and output a second eye feature vector.

[0088] In a possible implementation, the trained prediction model includes a trained multi-head attention mechanism network. The trained multi-head attention mechanism network is used to: map the first eye feature vector to obtain a first mapping vector; map the second eye feature vector to obtain a second mapping vector and a third mapping vector; and fuse the first mapping vector, the second mapping vector, and the third mapping vector based on the number of attention heads to obtain an eye fusion feature vector.

[0089] In the application, the formula for obtaining the first mapping vector is: fundus =F fundus W1. The formula for obtaining the second mapping vector is: k oct =F oct W2. The formula for obtaining the third mapping vector is v oct =F oct W3. Among them, W1, W2, and W3 are the mapping matrices of the trained multi-head attention mechanism network, and F fundus is the first eye feature vector, F oct is the second eye feature vector.

[0090] For the eye fusion feature vector, the relationship between the first eye feature and the second eye feature can be expressed as: Where n is the number of attention heads and D is the dimension of the first eye feature and the second eye feature.

[0091] Among them, the multi-head attention mechanism network can be a Transformer model (a model that uses an attention mechanism to accelerate model training).

[0092] In a possible implementation, the trained prediction model includes a trained fully connected network layer. The trained fully connected network layer is used to make predictions based on the eye fusion feature vector and output eye disease prediction results.

[0093] Figure 2 FIG. 2 is a second flow chart of the eye disease prediction method provided by an embodiment of the present application. Figure 2 As shown in the figure, the dotted box above is the branch for processing the fundus image. The image of the optic cup and optic disc area is input into the first extraction network trained in the fundus image branch to obtain the first eye feature of D dimension, and input it into the first conversion network layer trained. The first eye feature is converted by the first conversion network layer (patch embedding, image encoding) trained in the fundus image branch to obtain the first eye feature vector, and input it into the trained multi-head attention mechanism network (located in Figure 2 middle dashed box).

[0094] The dotted box below is the branch for processing the anterior segment optical coherence tomography image. The image of the anterior chamber angle area is input into the trained second extraction network in the anterior segment optical coherence tomography image branch to obtain the D-dimensional second eye feature, which is then input into the trained second conversion network layer. The second eye feature is converted by the trained second conversion network layer in the anterior segment optical coherence tomography image branch to obtain the second eye feature vector, which is then input into the trained multi-head attention mechanism network.

[0095] The trained eye disease prediction model includes L trained multi-head attention mechanism networks. These trained multi-head attention mechanism networks are connected in sequence. The output of the previous multi-head attention mechanism network is the input of the next multi-head attention mechanism network, and the feature fusion is repeated L times. Specifically, the first eye feature vector is mapped through a norm layer (normalization layer) to obtain a first mapping vector (q in the figure). The second eye feature vector is mapped through a norm layer to obtain a second mapping vector (k in the figure) and a third mapping vector (v in the figure). Then, based on the number of attention heads, the first mapping vector, the second mapping vector, and the third mapping vector are fused through an MSA layer (multi-head attention layer) to obtain a fused feature vector. Then through After a norm layer and an MLP layer (multi-layer perceptron) process the fused feature vector, the eye fusion feature vector is obtained.

[0096] The eye fusion feature vector output by the last multi-head attention mechanism network is input into the trained fully connected layer. The trained fully connected layer is used to make predictions based on the eye fusion feature vector to obtain the eye disease prediction results.

[0097] in, is the position code, To seek peace.

[0098] It can be understood that the local details of the image are extracted through the extraction network. The global information is learned through the prediction network, and the relationship between pixels with long distances in the image is learned. The eye disease prediction model combines the above advantages of the two networks.

[0099] This embodiment inputs the fundus image into a trained first extraction network, extracts the first eye feature through the trained first extraction network, and inputs the first eye feature into the trained prediction network; inputs the anterior segment optical coherence tomography image into a trained second extraction network, extracts the second eye feature through the trained second extraction network, and inputs the second eye feature into the trained prediction network; after converting the first eye feature into a first eye feature vector and the second eye feature into a second eye feature vector through the trained prediction network, the first eye feature vector and the second eye feature vector are fused to obtain an eye fusion feature vector, and an eye disease prediction result is determined and output based on the eye fusion feature, so as to fully exploit the complementary information between the fundus image and the anterior segment optical coherence tomography image to obtain eye features, and make predictions based on the fused eye features, thereby improving the accuracy of eye disease risk prediction.

[0100] In one embodiment, the eye disease prediction model includes a first extraction network, a second extraction network and a prediction network.

[0101] Before step S12, the method further includes:

[0102] S21: Obtain image training samples.

[0103] The image training samples include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, and the first label is used to characterize the eye disease condition of the sample.

[0104] The plurality of sets of samples may include a plurality of sets of images of one eye, or may include images of a plurality of eyes. The eye disease condition refers to whether the eye is diseased.

[0105] S22: Inputting the fundus image sample into the first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into the prediction network.

[0106] S23: inputting the anterior segment optical coherence tomography image sample into the second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network.

[0107] S24: After converting the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through the prediction network, the first sample feature vector and the second sample feature vector are fused to obtain a sample fusion feature vector, and a sample result is predicted and output according to the sample fusion feature vector to obtain a sample result.

[0108] The sample result may be an eye disease risk, expressed as a probability, or the sample result may be an eye disease result, expressed as a numerical value.

[0109] S25: Determine a loss value of a first loss function of the eye disease prediction model according to the sample result and the first label.

[0110] The first loss function may be a cross entropy loss function. The formula of the first loss function is L1=L out (P o ,G o ). o is the sample result, G o is the first label.

[0111] S26: If the loss value of the first loss function is greater than the first preset loss value, after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value.

[0112] In the application, if the loss value of the first loss function is greater than the first preset loss value, it means that the eye disease prediction model has not yet performed the task. After adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model and return to execute steps S22 to S25.

[0113] S27: If the loss value of the first loss function is less than the first preset loss value, a trained eye disease prediction model is obtained.

[0114] In the application, if the loss value of the first loss function is less than the first preset loss value, it means that the eye disease prediction model has been trained, the task can be executed, and the trained eye disease prediction model can be obtained.

[0115] This embodiment trains the eye disease prediction model so that the eye disease prediction model can fully exploit the complementary information between the fundus image and the anterior segment optical coherence tomography image, obtain a suitable fusion method, and improve the accuracy of predicting eye diseases.

[0116] In one embodiment, the image training sample further includes a second label of the fundus image sample, and the second label is used to characterize the true segmentation result of the optic cup and optic disc. Correspondingly, the total loss function includes a first preset loss function and a second preset loss function.

[0117] After the first sample feature is converted into the first sample feature vector through the prediction network, it also includes:

[0118] S31: Segment the fundus image sample according to the first sample feature vector to obtain a segmentation prediction result of the fundus image sample.

[0119] In the application, after the first extraction network outputs the first sample feature vector, the first sample feature vector is obtained from the first extraction network, and the optic cup and optic disc of the fundus image sample are segmented using the first sample feature vector to obtain the segmentation prediction result of the fundus image sample.

[0120] S32: Determine a loss value of a second preset loss function of the eye disease prediction model according to the segmentation prediction result and the second label.

[0121] The second loss function may be a cross entropy loss function. The formula of the first loss function is L2=L fundus (P f ,G f ). f is the segmentation prediction result, G f For the second label.

[0122] S33: If the loss value of the total loss function is greater than the second preset loss value, after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the total loss function is less than the second preset loss value.

[0123] Among them, the total loss function is L=L1+L2.

[0124] In the application, if the loss value of the total loss function is greater than the second preset loss value, it means that the eye disease prediction model has not yet performed the task. After adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model and return to execute steps S22 to S25, and S31 to S32.

[0125] S34: If the loss value of the total loss function is less than the second preset loss value, a trained eye disease prediction model is obtained.

[0126] in, Figure 3 FIG. 2 is a third flow chart of the eye disease prediction method provided in an embodiment of the present application. Figure 3 As shown, in the branch for processing the fundus image, the image of the optic cup and optic disc area is input into the first extraction network in the fundus image branch to obtain the first sample feature of D dimension, and input into the trained first conversion network layer. The first sample feature is converted by the trained first conversion network layer in the fundus image branch to obtain the first sample feature vector. The first sample feature vector is obtained, and the fundus image sample is segmented according to the first sample feature vector to obtain the segmentation prediction result of the fundus image sample (the rightmost figure in the fundus image branch). Other identical parts are not repeated here.

[0127] This embodiment obtains a segmentation prediction result of the fundus image sample by segmenting the fundus image sample according to the first sample feature vector; determines the loss value of the second preset loss function of the eye disease prediction model according to the segmentation prediction result and the second label; if the loss value of the total loss function is greater than the second preset loss value, adjusts the parameters of the eye disease prediction model, and continues to train the eye disease prediction model until the loss value of the total loss function is less than the second preset loss value; if the loss value of the total loss function is less than the second preset loss value, obtains a trained eye disease prediction model; introduces the segmentation result of the fundus image, adds more supervision information, better trains the eye disease prediction model, and improves the prediction ability of the model.

[0128] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0129] Corresponding to the method described in the above embodiment, for the sake of convenience of explanation, only the part related to the embodiment of the present application is shown.

[0130] In one embodiment, Figure 4 Schematic diagram of the structure of an eye disease prediction device provided by an embodiment of the present application. Figure 4 As shown, the device comprises:

[0131] An acquisition module 10 is used to acquire a fundus image and an anterior segment optical coherence tomography image of a user's eye;

[0132] The prediction module 11 is used to input the fundus image and the anterior segment optical coherence tomography image into the trained eye disease prediction model to obtain the eye disease prediction result output by the trained eye disease prediction model.

[0133] Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output an eye disease prediction result based on the eye fusion feature.

[0134] In one embodiment, the trained eye disease prediction model includes a trained first extraction network, a trained second extraction network, and a trained prediction network.

[0135] The prediction module is specifically used to input the fundus image into a trained first extraction network to obtain a first eye feature output by the trained first extraction network; input the anterior segment optical coherence tomography image into a trained second extraction network to obtain a second eye feature output by the trained second extraction network; and input the first eye feature and the second eye feature into the trained prediction network to obtain a prediction result output by the trained prediction network.

[0136] Among them, the trained eye disease prediction model includes a trained first extraction network, a trained second extraction network and a trained prediction network. The trained prediction network is used to convert the first eye feature into a first eye feature vector and the second eye feature into a second eye feature vector, then fuse the first eye feature vector and the second eye feature vector to obtain an eye fusion feature vector, and output an eye disease prediction result based on the eye fusion feature.

[0137] In one embodiment, the apparatus further comprises a training module;

[0138] The acquisition module is also used to acquire image training samples, which include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, and the first label is used to characterize the eye disease condition of the sample.

[0139] A training module, used for inputting the fundus image sample into the first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into the prediction network;

[0140] It is also used for inputting the anterior segment optical coherence tomography image sample into the second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network;

[0141] It is also used to convert the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through a prediction network, fuse the first sample feature vector and the second sample feature vector to obtain a sample fusion feature vector, and predict and output a sample result according to the sample fusion feature vector to obtain a sample result;

[0142] It is also used to determine the loss value of the first loss function of the eye disease prediction model according to the sample result and the first label;

[0143] It is also used for adjusting the parameters of the eye disease prediction model if the loss value of the first loss function is greater than the first preset loss value, and continuing to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value;

[0144] Also used for obtaining a trained eye disease prediction model if the loss value of the first loss function is less than a first preset loss value;

[0145] Among them, the eye disease prediction model includes a first extraction network, a second extraction network and a prediction network.

[0146] In one embodiment, the image training sample further includes a second label of the fundus image sample, and the second label is used to represent the true segmentation result of the optic cup and optic disc.

[0147] The training module is further used to segment the fundus image sample according to the first sample feature vector to obtain a segmentation prediction result of the fundus image sample;

[0148] It is also used to determine the loss value of a second preset loss function of the eye disease prediction model according to the segmentation prediction result and the second label;

[0149] It is also used for adjusting the parameters of the eye disease prediction model if the loss value of the total loss function is greater than the second preset loss value, and continuing to train the eye disease prediction model until the loss value of the total loss function is less than the second preset loss value;

[0150] Also used for obtaining a trained eye disease prediction model if the loss value of the total loss function is less than a second preset loss value;

[0151] The total loss function includes a first preset loss function and a second preset loss function.

[0152] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 2 of this embodiment includes: at least one processor 20 ( Figure 5Only one is shown in the figure), a memory 21, and a computer program 22 stored in the memory 21 and executable on the at least one processor 20, wherein the processor 20 implements the steps of any of the above-mentioned method embodiments when executing the computer program 22.

[0153] The electronic device 2 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device 2 may include, but is not limited to, a processor 20 and a memory 21. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 2 and does not constitute a limitation on the electronic device 2. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0154] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0155] In some embodiments, the memory 21 may be an internal storage unit of the electronic device 2, such as a hard disk or memory of the electronic device 2. In other embodiments, the memory 21 may also be an external storage device of the electronic device 2, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 2. Further, the memory 21 may also include both an internal storage unit of the electronic device 2 and an external storage device. The memory 21 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 21 may also be used to temporarily store data that has been output or is to be output.

[0156] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0157] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0158] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0159] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some cases, the computer-readable medium cannot be an electric carrier signal and a telecommunication signal.

[0161] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0163] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for predicting eye diseases, characterized in that: include: Acquire a fundus image and an anterior segment optical coherence tomography image of the user's eye; Inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model; Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature.

2. The method according to claim 1, characterized in that The trained eye disease prediction model includes a trained first extraction network, a trained second extraction network and a trained prediction network; Inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model, comprising: Inputting the fundus image into the trained first extraction network to obtain the first eye feature output by the trained first extraction network; Inputting the anterior segment optical coherence tomography image into the trained second extraction network to obtain the second eye feature output by the trained second extraction network; The first eye feature and the second eye feature are input into the trained prediction network to obtain the prediction result output by the trained prediction network, the trained prediction network is used to convert the first eye feature into a first eye feature vector and the second eye feature into a second eye feature vector, then fuse the first eye feature vector and the second eye feature vector to obtain an eye fusion feature vector, and output the eye disease prediction result according to the eye fusion feature.

3. The method according to claim 2, characterized in that: The trained prediction network includes a trained multi-head attention mechanism network; The trained multi-head attention mechanism network is used to: map the first eye feature vector to obtain a first mapping vector; map the second eye feature vector to obtain a second mapping vector and a third mapping vector; and based on the number of attention heads, fuse the first mapping vector, the second mapping vector and the third mapping vector to obtain the eye fusion feature vector.

4. The method according to any one of claims 1 to 3, characterized in that: Before inputting the fundus image and the anterior segment optical coherence tomography image into the trained eye disease prediction model, the method further includes: Acquire image training samples, wherein the image training samples include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, wherein the first label is used to characterize the eye disease condition of the sample; Inputting the fundus image sample into a first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into a prediction network; Inputting the anterior segment optical coherence tomography image sample into a second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network; After converting the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through the prediction network, fusing the first sample feature vector and the second sample feature vector to obtain a sample fusion feature vector, and predicting and outputting a sample result according to the sample fusion feature vector to obtain the sample result; Determining a loss value of a first loss function of the eye disease prediction model according to the sample result and the first label; If the loss value of the first loss function is greater than a first preset loss value, then after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value; If the loss value of the first loss function is less than the first preset loss value, obtaining the trained eye disease prediction model; The eye disease prediction model includes the first extraction network, the second extraction network and the prediction network.

5. The method according to claim 4, characterized in that The image training sample also includes a second label of the fundus image sample, and the second label is used to represent the true segmentation result of the optic cup and optic disc; After converting the first sample feature into a first sample feature vector through the prediction network, the method further includes: Segmenting the fundus image sample according to the first sample feature vector to obtain a segmentation prediction result of the fundus image sample; Determining a loss value of a second preset loss function of the eye disease prediction model according to the segmentation prediction result and the second label; If the loss value of the total loss function is greater than the second preset loss value, then after adjusting the parameters of the eye disease prediction model, continue to train the eye disease prediction model until the loss value of the total loss function is less than the second preset loss value; If the loss value of the total loss function is less than the second preset loss value, obtaining the trained eye disease prediction model; The total loss function includes the first preset loss function and the second preset loss function.

6. The method according to claim 5, characterized in that The first eye feature is an appearance feature of an optic cup and optic disc region in the fundus image, and the second eye feature is an appearance feature of an anterior chamber angle region in the anterior segment optical coherence tomography image.

7. An eye disease prediction device, characterized in that: include: An acquisition module, used to acquire a fundus image and anterior segment optical coherence tomography image of a user's eye; A prediction module, used for inputting the fundus image and the anterior segment optical coherence tomography image into a trained eye disease prediction model to obtain an eye disease prediction result output by the trained eye disease prediction model; Among them, the trained eye disease prediction model is used to extract a first eye feature from the fundus image and a second eye feature from the anterior segment optical coherence tomography image, fuse the first eye feature and the second eye feature to obtain an eye fusion feature, and output the eye disease prediction result according to the eye fusion feature.

8. The device according to claim 7, characterized in that Also includes training modules; The acquisition module is further used to acquire image training samples, wherein the image training samples include multiple groups of fundus image samples, anterior segment optical coherence tomography image samples, and a first label, wherein the first label is used to characterize the eye disease condition of the sample; A training module, used for inputting the fundus image sample into a first extraction network, extracting first sample features through the first extraction network, and inputting the first sample features into a prediction network; Also used for inputting the anterior segment optical coherence tomography image sample into a second extraction network, extracting second sample features through the second extraction network, and inputting the second sample features into the prediction network; The method is further configured to, after converting the first sample feature into a first sample feature vector and the second sample feature into a second sample feature vector through the prediction network, fuse the first sample feature vector and the second sample feature vector to obtain a sample fusion feature vector, and predict and output a sample result according to the sample fusion feature vector to obtain the sample result; It is also used to determine the loss value of the first loss function of the eye disease prediction model according to the sample result and the first label; Also used for, if the loss value of the first loss function is greater than a first preset loss value, adjusting the parameters of the eye disease prediction model, and continuing to train the eye disease prediction model until the loss value of the first loss function is less than the first preset loss value; Also used for obtaining the trained eye disease prediction model if the loss value of the first loss function is less than the first preset loss value; The eye disease prediction model includes the first extraction network, the second extraction network and the prediction network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images