Image-based eye lesion grading analysis method and device, medium, program product and terminal

Abstract

The invention provides an image-based eye lesion grading analysis method and device, a medium, a program product and a terminal, and the method comprises the steps: obtaining one or more eye lesion images, inputting the one or more eye lesion images to a pre-trained first model, and generating a corresponding eye feature vector; for a single image, calculating the similarity between the feature vector and each eye disease category, and outputting a prediction category with the highest similarity; and for a plurality of images, respectively calculating the similarity between the feature vectors and each category, and realizing adaptive weighted integration by using the second model to obtain a final prediction category. Sub-model subdivision and risk level layering are carried out according to prediction categories, and a structured risk layering report is output. The problem that in the prior art, the diagnostic value of multi-image information in complex cases is difficult to effectively evaluate is solved, the accuracy of ultrasound in serious vitreous lesion and intraocular tumor recognition is remarkably improved through a self-adaptive weighted integration mechanism, high-risk patient screening and clinical decision making are supported, and the application scene of ultrasound in ophthalmic diagnosis is expanded.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image-based eye lesion grading analysis method, device, medium, program product and terminal. BACKGROUND

[0002] There are many technical defects in the analysis of ophthalmic ultrasound images, which affect the clinical application effect. First, although artificial intelligence models are applied, these models cannot simultaneously identify multiple eye features presented in the images, resulting in low accuracy of feature recognition and classification. In addition, the limitations of current technology in image feature level assessment result in an inability to accurately classify the severity of features, which limits the refinement level of analysis reports.

[0003] Secondly, ophthalmic ultrasound examination usually involves multiple images, and existing methods are not efficient in integrating information from different perspectives, affecting the comprehensive understanding and analysis of image information. The eyeball structure is complex, and the subjectivity is strong during manual analysis. The lack of standardized procedures results in differences in interpretation results of the same image information by different analysts, which may lead to inconsistent analysis results for the same subject.

[0004] Although deep learning technology is introduced, the accuracy and interpretability of the model are still insufficient, which leads to the difficulty of analysts in understanding the basis of the model output, further affecting the trust in the analysis results. Finally, the text of the ultrasound report is less structured, and the terminology is not uniform, increasing the complexity of information extraction and subsequent processing. Therefore, these factors together limit the ability of AI models in automatic image analysis and structured report generation, ultimately affecting the efficiency and accuracy of image feature analysis. SUMMARY

[0005] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide an image-based eye lesion grading analysis method, device, medium, program product and terminal, which solves the problem of low accuracy in the identification and evaluation process in the existing ophthalmic ultrasound image analysis.

[0006] To achieve the above object and other related objects, the first aspect of the present application provides an image-based eye lesion grading analysis method, comprising: acquiring one or more eye disease images; inputting the eye disease images into a first encoder of a pre-trained first model to generate one or more corresponding eye feature vectors; when the eye disease images are multiple images, inputting all eye feature vectors into a second model to generate an attention weight vector of all eye disease images; calculating an aggregated feature vector based on the eye feature vectors and the attention weight vector; based on the first model, calculating the similarity between the eye feature vector or the aggregated feature model and each preset eye disease category, and extracting one or more disease categories with the highest similarity as a prediction category; performing a risk grading mapping operation according to the prediction category to generate a risk stratification mapping result; and outputting a structured report according to the risk stratification mapping result.

[0007] In some embodiments of the first aspect of the present application, the first model comprises a first encoder and a second encoder, and the pre-training process of the first model comprises: inputting an eye training image into the first encoder to generate an eye image feature; inputting a corresponding plurality of eye disease category texts into the second encoder to map the eye disease category texts to a feature space with the same dimension as the eye image feature, to generate an eye disease text feature of each category; calculating the similarity between each eye image feature and each eye disease text feature; based on the similarity between each eye image feature and the eye disease text feature corresponding to its true category, and the similarity of other category text features, constructing a contrast loss function; and performing a parameter update operation on the first encoder and the second encoder according to the contrast loss function, so as to maximize the similarity between the eye image feature and the eye disease text feature corresponding to its true category, and minimize the similarity with the eye disease text features of non-corresponding categories.

[0008] In some embodiments of the first aspect of the present application, the process of inputting all similarities into the second model to perform adaptive weighted integration operation to generate an aggregated similarity score comprises: inputting all similarities into an attention weight calculation model to generate an attention weight coefficient corresponding to the similarity of each eye disease image; performing a weighting operation on the similarity of each eye disease image and the corresponding attention weight coefficient, and performing a summation operation on the similarity vector after the weighting operation in the image dimension to generate an aggregated similarity score vector.

[0009] In some embodiments of the first aspect of the present application, the process of performing a risk classification mapping operation according to the predicted category to generate a risk classification mapping result includes: determining whether the predicted category is vitreous opacity or substantial space-occupying; if the predicted category is vitreous opacity, inputting the eye feature vector into a third model to perform a turbidity degree identification operation and outputting a corresponding turbidity grade; if the predicted category is substantial space-occupying, inputting the eye feature vector or the aggregated feature model into a fourth model to perform a tumor subtype identification operation and outputting a corresponding tumor subtype; and classifying the eye disease image into a plurality of risk levels according to the predicted category, the turbidity grade, and the tumor subtype.

[0010] In some embodiments of the first aspect of the present application, the risk level includes one of normal, low-risk vision, high-risk vision, and intraocular tumor.

[0011] In some embodiments of the first aspect of the present application, the method further includes the step of: mapping the gradient-weighted feature map activation intensity corresponding to the predicted category back to the spatial position of the eye disease image, thereby generating a Grad-CAM heat map for each eye disease image to highlight the important regions that the model focuses on in the classification decision. To achieve the above object and other related objects, the second aspect of the present application provides an image-based eye lesion classification analysis device, which includes: a data acquisition module configured to acquire one or more eye disease images; a feature extraction module configured to input the eye disease image into a first encoder of a pre-trained first model to generate one or more eye feature vectors; when the eye disease image is one image, based on the first model, calculating the similarity of the eye feature vector to each preset eye disease category, and extracting one or more disease categories with the highest similarity as the predicted category; when the eye disease image is multiple images, based on the first model, calculating the similarity of each eye feature vector to each preset eye disease category, inputting all similarities into a second model to perform an adaptive weighted integration operation on all similarities to generate an aggregated similarity score, and extracting one or more disease categories with the highest similarity in the aggregated similarity score as the predicted category; a classification analysis module configured to calculate the similarity of the eye feature vector or the aggregated feature model to each preset eye disease category based on the first model, and extract one or more disease categories with the highest similarity as the predicted category; perform a risk classification mapping operation according to the predicted category to generate a risk classification mapping result; and output a structured report according to the risk classification mapping result.

[0012] To achieve the above object and other related objects, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the image-based eye lesion classification analysis method.

[0013] To achieve the above object and other related objects, the fourth aspect of the present application provides a computer program product, which comprises computer program codes, and when the computer program codes are run on a computer, the computer is caused to implement the image-based eye lesion grading analysis method.

[0014] To achieve the above object and other related objects, the fifth aspect of the present application provides an electronic terminal, which comprises a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the image-based eye lesion grading analysis method.

[0015] As described above, the image-based eye lesion grading analysis method, device, medium, program product and terminal of the present application have the following beneficial effects: by inputting one or more eye disease images into the pre-trained first model, a corresponding eye feature vector is generated. For a single image, the similarity of the feature vector to each eye disease category is calculated, and the prediction category with the highest similarity is output; for multiple images, the similarity of the feature vector to each category is calculated respectively, and the second model is used to realize adaptive weighted integration to obtain the final prediction category. According to the prediction category, the sub-model is subdivided and the risk level is layered, and a structured risk stratification report is output. The present application solves the problem that the prior art cannot effectively evaluate the diagnostic value of multiple image information in complex cases, significantly improves the accuracy of ultrasound in identifying severe vitreous lesions and intraocular tumors through an adaptive weighted integration mechanism, supports high-risk patient screening and clinical decision-making, and expands the application scenarios of ultrasound in ophthalmic diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of an embodiment of the image-based eye lesion grading analysis method of the present application is shown.

[0017] Figure 2 A flowchart of risk stratification mapping in an embodiment of the image-based eye lesion grading analysis method of the present application is shown.

[0018] Figure 3 A structural diagram of an embodiment of the image-based eye lesion grading analysis device of the present application is shown.

[0019] Figure 4 A structural diagram of an embodiment of the image-based eye lesion grading analysis terminal of the present application is shown. DETAILED DESCRIPTION

[0020] The present application can be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details of the present application based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.

[0021] Before the present application is further described, the terms and terminology used in the embodiments of the present application are explained, and the terms and terminology used in the embodiments of the present application are applicable to the following explanations:

[0022] <1> Ocular ultrasound image: Ocular ultrasound image is a kind of real-time, non-invasive imaging examination using high-frequency sound waves to penetrate the eyeball and orbital tissue, and forming through the reflection and echo of sound waves. It can show the morphology, position, size and blood flow of the internal structure of the eyeball (such as lens, vitreous body, retina, choroid) and orbital lesions (such as orbital tumor, ocular muscle lesions). It has important value in the diagnosis of ocular diseases such as vitreous opacity, retinal detachment and intraocular tumor, and is especially suitable for cases where direct ophthalmoscopy examination is not possible due to corneal opacity or pupil dilation.

[0023] <2> CLIP model: CLIP model is a pre-trained model developed by OpenAI, which can learn the correspondence between images and texts. Its core idea is to train on a large number of image-text pairs through contrastive learning, so that matching images and texts are closer in embedding space, while non-matching ones are farther apart. This makes the CLIP model have strong zero-shot learning ability, that is, it can directly recognize new visual concepts without additional training, and can complete various cross-modal tasks such as image classification, image retrieval, and text-to-image generation.

[0024] <3> Swin Transformer v2 model: Swin Transformer v2 is a visual Transformer model improved by Microsoft Research based on SwinTransformer. Swin Transformer introduces a shifted window mechanism to limit self-attention calculation within a local window, effectively reducing computational complexity, making it able to handle high-resolution images and becoming one of the mainstream backbone networks in the field of computer vision. SwinTransformer v2 further optimizes it on this basis, including improving the position encoding, normalization strategy and large-scale training stability, making it achieve better performance in various visual tasks such as image classification, object detection and semantic segmentation.

[0025] <4>Chinese Bert model: Chinese Bert model is a pre-trained version of BERT model developed by Google on Chinese corpus. BERT is a bidirectional encoder based on Transformer architecture, which is pre-trained on large-scale unlabeled text through two self-supervised tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP), so as to learn rich language representation. Chinese Bert is optimized for the characteristics of Chinese text (such as unclear word boundaries, importance of character-level features), making it perform well in Chinese natural language processing (NLP) tasks (such as text classification, named entity recognition, question answering system, machine translation).

[0026] <5> Vitreous opacity: Vitreous opacity refers to the presence of any form of turbidity in the vitreous cavity of the eyeball, causing light scattering or obstruction when passing through the vitreous, thereby affecting vision. Common causes of vitreous opacity include physiological floaters and pathological opacity. Physiological floaters are characterized by the formation of point-like, linear or sheet-like floating objects due to vitreous liquefaction and condensation; pathological opacity is caused by vitreous hemorrhage (such as diabetic retinopathy, retinal hole), inflammation (such as uveitis), infection, trauma, posterior vitreous detachment (PVD), etc. Patients often feel a black shadow floating in front of their eyes, and severe cases can cause visual impairment.

[0027] <6> Solid mass: Solid mass refers to the lesion formed by abnormal proliferation or accumulation of tissue or cells with certain morphology and density found in medical imaging (such as ultrasound, CT, MRI). It is opposite to cystic mass (such as cyst, internal liquid). Solid mass can be benign (such as hyperplasia, fibroma) or malignant (such as tumor), and its nature needs to be determined in combination with clinical manifestations, other imaging features and pathological biopsy. In ophthalmology, solid mass found by ocular ultrasound may indicate intraocular tumor (such as choroidal melanoma, retinoblastoma) or other solid lesions.

[0028] <7> Grad-CAM heat map: Grad-CAM heat map is a visualization technique commonly used in deep learning models to explain where the model's attention or focus is mainly concentrated when making predictions. In computer vision, Grad-CAM heat map is usually presented in the form of a color overlay image, with brighter or warmer colors indicating higher attention from the model to that area, and being considered to contribute more to the final prediction result. This helps to understand the model decision-making process and assess whether the model focuses on the correct features in the image.

[0029] <8> Prototype matching graph: Prototype matching graph is commonly used in the field of explainable artificial intelligence (XAI), especially in models based on prototype learning. Such models learn a set of prototypes to represent the core features or typical samples of different classes. The prototype matching graph shows the similarity or matching degree between a given input sample and each prototype learned by the model. Through visualization, it can be understood which prototype or prototypes the input sample is closest to, thus explaining the basis on which the model classifies it into a certain specific class. It helps to reveal the decision-making logic inside the model, enhancing the explainability of the model.

[0030] <9> Structured report: Structured report refers to medical reports written in a pre-set, standardized template and format, rather than in free text form. It usually contains clear headings, subheadings, fields and pre-defined options or standard terms, facilitating quick entry, retrieval, analysis and sharing of information. In medical image diagnosis, structured report can improve the accuracy, completeness and consistency of the report, reduce omissions, and facilitate communication among doctors and subsequent data mining and research. For example, the structured report of eye ultrasound may include fixed items such as examination site, scanning method, measurement data of each structure of the eyeball, lesion description, diagnostic impression and recommendations.

[0031] To facilitate understanding of the embodiments of the present application, first, the Figure 1 details are explained. Figure 1 A flowchart of an image-based eye lesion grading analysis method according to an embodiment of the present application is shown. The image-based eye lesion grading analysis method in this embodiment mainly includes the following steps:

[0032] Step S11: Obtain one or more eye disease images.

[0033] In an embodiment of the present application, the eye disease images are eye ultrasound images. The types of lesions included in the eye images are: retinal detachment, retinal hole, choroidal detachment, solid space-occupying lesion, globe wall abnormality, vitreous opacity, no obvious abnormal echo, posterior vitreous detachment, posterior scleral staphyloma, optic disc edema, silicone oil, eyeball atrophy, choroidal defect, suprachoroidal hemorrhage, optic disc calcification, foreign body, lens dislocation, vitreous star-shaped degeneration.

[0034] Among them, the retinal detachment is manifested as a high echo membrane structure from the eye wall peeling, the retinal hole shows local fracture and low echo section; choroidal detachment is an irregular echo increase area, the substantial occupation is a low to medium echo mass with clear or blurred boundary; the abnormal ball wall reflects the thickening of the eye wall or local echo change, vitreous opacity is scattered or punctate strong echo; posterior vitreous detachment shows that the vitreous structure is separated from the retina, the posterior sclera aneurysm is a cystic echo area; optic disc edema is manifested as blurred outline of the optic disc, silicone oil shows a uniform high echo area; eyeball atrophy is the shrinkage of the eye content, choroidal defect is manifested as local echo loss, choroidal hemorrhage is a mixed echo area; optic disc calcification has strong echo with acoustic shadow, foreign body shows abnormal strong echo with acoustic shadow, lens dislocation shows abnormal lens position and echo change, vitreous star-shaped degeneration shows star-shaped echo structure.

[0035] Step S12: inputting the eye disease image into a first encoder of the pre-trained first model to generate one or more corresponding eye feature vectors.

[0036] In an embodiment of the present application, after the eye disease image is input into the first encoder of the pre-trained first model, the first encoder extracts features at multiple levels through a deep neural network structure. The first encoder includes but is not limited to a multi-layer convolutional neural network, a transformer or the like. By extracting visual features at different levels from the image through the first encoder, the input image is decomposed into multiple local regions, and the features of each region are gradually encoded, fused and optimized to capture important information and details in the image. After layer-by-layer processing of the network, multi-dimensional feature vectors are finally generated, which can reflect the key visual features in the eye disease image for subsequent analysis and classification.

[0037] In an embodiment of the present application, the first model includes a first encoder and a second encoder, and the pre-training process of the first model includes: inputting an eye training image into the first encoder to generate an eye image feature; inputting a plurality of eye disease category texts corresponding to the eye image feature into the second encoder to map the eye disease category texts to a feature space with the same dimension as the eye image feature, and generate an eye disease text feature of each category; calculating the similarity between each eye image feature and each eye disease text feature; constructing a contrast loss function based on the similarity between each eye image feature and the eye disease text feature corresponding to its true category, and the similarity of other category text features; and performing a parameter update operation on the first encoder and the second encoder according to the contrast loss function, so as to maximize the similarity between the eye image feature and the eye disease text feature corresponding to its true category, and minimize the similarity between the eye image feature and the eye disease text features of non-corresponding categories.

[0038] In this embodiment, the first model includes a first encoder and a second encoder. The first encoder is used to receive the input eye training images and perform feature extraction, generating a vector representing the image features. The second encoder is used to process text information related to multiple eye disease categories, including disease names and descriptions for disease categories. After inputting the text, the output is a category semantic feature with the same dimension as the image feature vector.

[0039] It should be noted that during the training stage of the first model, the eye images are encoded by the first encoder to obtain the corresponding image features; at the same time, the second encoder encodes the text description related to the corresponding category and other categories of the image to obtain the semantic features of different categories. Then, the similarity between each group of eye image features and the text features of the corresponding category is calculated, and the similarity between the image features and the text features of other categories is also calculated. Based on the above similarity, a contrast loss function is designed and constructed to distinguish between real categories and non-real categories. The loss function aims to maximize the similarity between the image features and their real category text features, while minimizing the similarity between the image features and the text features of non-corresponding categories. During the model training process, the first encoder and the second encoder are optimized based on the contrast loss function, so as to promote the model to learn more discriminative feature representations. Exemplarily, the first encoder adopts the SwinTransformer v2 model; the text encoder used by the second encoder is Chinese Bert, which is used to receive direct Chinese input of disease reports.

[0040] Further, the contrast loss function aims to maximize the similarity between the eye image features and their real eye disease category text features, while minimizing the similarity between the image features and all non-corresponding eye disease category text features. It is usually calculated by calculating the similarity between an eye image feature and all eye disease text features (including real corresponding and non-corresponding categories), and then transformed by using logarithmic and exponential functions to ensure effective discrimination between positive and negative samples during model learning.

[0041] Exemplarily, a pre-trained CLIP model structure is used, in which the first encoder is a Swin Transformer v2 architecture, which is used to automatically extract image features from medical images and output the probability distribution of the image belonging to each category of 18 different eye diseases. The second encoder is specifically used to process a large amount of information related to the disease type, including but not limited to the name of the disease category and its description, to generate the semantic features corresponding to the category. This embodiment achieves effective alignment of medical image features and disease category semantic features in the same feature space by synchronously training the two encoders.

[0042] Step S13: When the eye disease image is one image, based on the first model, the similarity of the eye feature vector to each preset eye disease category is calculated, and one or more disease categories with the highest similarity are extracted as the predicted category.

[0043] In an embodiment of the present application, different feature extraction and disease category recognition operations are performed for multiple input images and single input images. Specifically, for a single eye disease image, the image is directly input into the pre-trained first encoder, and an eye feature vector representing the eye feature is extracted. Then, the eye feature vector is mapped to the feature space, and the cosine similarity between the eye feature vector and the text feature vector corresponding to each preset eye disease category is calculated one by one. By comparing the similarity of each disease category text vector and the eye feature vector, one or more disease categories with the highest similarity are selected as the prediction result of the single image, so as to realize accurate classification and recognition of the single eye disease image.

[0044] In an embodiment of the present application, in specific implementation, in order to improve the fault tolerance and robustness of the model, several disease candidates with the highest similarity ranking are output. In order to facilitate subsequent processing and decision-making, the number of candidates is usually an odd number (such as three, five, etc.), so as to ensure that the selected result is representative and easy to classify and decide by majority voting mechanism. This strategy not only improves the accuracy of model prediction, but also provides more reference options for complex cases with multiple possible diseases.

[0045] Step S14: When the eye disease image is multiple images, based on the first model, the similarity of each eye feature vector to each preset eye disease category is calculated, and all similarities are input into the second model to perform adaptive weighted integration operation on all similarities, generate an aggregated similarity score, and extract one or more disease categories with the highest similarity in the aggregated similarity score as the predicted category.

[0046] In an embodiment of the present application, the process of inputting all similarities into the second model to perform adaptive weighted integration operation to generate an aggregated similarity score includes: inputting all similarities into an attention weight calculation model to generate an attention weight coefficient corresponding to the similarity of each eye disease image; performing weighted operation on the similarity of each eye disease image and the corresponding attention weight coefficient, and performing summation operation on the similarity vector after the weighted operation in the image dimension to generate an aggregated similarity score vector.

[0047] In the present embodiment, the eye feature vector of each image is extracted by the pre-trained first model. For the feature vector of each image, the similarity between the feature vector and the text feature vector of each preset eye disease category is calculated to form a matrix containing multiple groups of similarity data. The similarity data matrix is input into the second model to perform adaptive prediction result aggregation operation.

[0048] It is worth noting that the second model employs an attention mechanism. By calculating the attention weight corresponding to the similarity of each image, it adaptively measures the importance of different images in the final judgment, thereby generating the final prediction result for multiple input images of the same lesion. The attention weight represents the contribution of each image to the differentiation of disease categories during the diagnostic process. Subsequently, the similarity of each image is multiplied by its corresponding attention weight to complete the weighting operation. By summing all weighted similarity values ​​along the image dimension, an aggregated similarity score that integrates information from multiple images is generated. This aggregated similarity score reflects the overall correlation between all input images of the same patient and each disease category. At the same time, through adaptive adjustment of the attention weight, it highlights key lesion image features and suppresses irrelevant interference information. Finally, the disease category with the highest score is selected as the prediction result for multiple eye disease images, thus achieving effective fusion of features from multiple images and improving the accuracy and robustness of eye disease identification.

[0049] Step S15: For the predicted category, perform a risk grading mapping operation to generate a risk stratification mapping result; output a structured report based on the risk stratification mapping result.

[0050] In one embodiment of this application, the process of performing a risk grading mapping operation and generating a risk stratification mapping result based on the predicted category includes: determining whether the predicted category is vitreous opacity or substantial lesion; if the predicted category is vitreous opacity, inputting the ocular feature vector into a third model to perform an opacity degree identification operation and outputting the corresponding opacity level; if the predicted category is substantial lesion, inputting the ocular feature vector or the aggregated feature model into a fourth model to perform a tumor subtype identification operation and outputting the corresponding tumor subtype.

[0051] like Figure 2 As shown, for the predicted category of eye features, the prediction results are transformed into different risk levels by performing a risk grading mapping operation. Specifically, it is first determined whether the initial predicted category is "vitreous opacity" or "solid mass". If it is "vitreous opacity", the current eye feature vector is input into a pre-trained third model for identifying the degree of vitreous opacity, and the corresponding opacity level is output. If it is "solid mass", the feature vector or aggregated feature model is input into a fourth model to perform further tumor subtype identification and identify the specific tumor type. These sub-models for detailed identification are all pre-trained based on the sub-CLIP model. This embodiment further refines the preliminary prediction results.

[0052] In an embodiment of the present application, the eye disease image is classified into multiple risk levels according to the predicted category, the turbidity level and the tumor subtype, wherein the risk level includes one of normal, low-risk vision, high-risk vision and intraocular tumor.

[0053] In the embodiment, the process of performing risk classification mapping operation to generate risk stratification mapping result includes: combining the preset lookup rule, and mapping different predicted categories to specific risk levels. For example, “no obvious abnormal echo”, “vitreous star degeneration” are classified as normal category, “vitreous posterior detachment”, “slight vitreous opacity” belong to low-risk vision, “retinal detachment”, “obvious vitreous opacity” are classified as high-risk vision, and “substantial space-occupying” indicates high-risk state of intraocular tumor. Through this way of combining sub-model to perform lesion identification and performing risk stratification mapping according to the preset lookup rule, the vision risk level and potential life threat of the patient can be dynamically judged according to the multi-image information and diagnosis result. The stratification system fully embodies the unique advantages of ultrasound in identifying serious vitreous lesions and intraocular tumors, and makes it show higher application value in high-risk patient screening, preoperative decision and follow-up recommendation scenarios.

[0054] In an embodiment of the present application, the method further includes the step of: mapping the similarity distribution of the predicted category back to each region of the eye disease image to generate a Grad-CAM heat map of each eye disease image.

[0055] In the embodiment, the generated heat map is based on Grad-CAM technology and is used to explain the spatial attention area of the model in the eye disease image classification task. This method calculates the weighted sum of each channel by extracting the feature map of a specific convolutional layer in the model and combining the gradient information of the layer to the predicted category, so as to obtain the importance of each spatial position to the classification result. Since the spatial resolution of the feature map of the convolutional neural network is usually lower than that of the original image, the importance distribution needs to be up-sampled to align with the original image, forming a two-dimensional heat map. The value of each position in the heat map reflects the contribution degree of the region to the final classification decision of the model, and the higher the value, the more attention the model pays to the region. Finally, by superimposing the heat map on the original image, the region that the model pays attention to when making a prediction can be intuitively displayed, thereby enhancing the explainability of the model output.

[0056] In the embodiment, the method further includes the process of generating a prototype matching map based on the prediction result, which includes selecting the test images from the multiple prediction results, calculating the similarity score of each test image with all images in the gold standard image library, and finding the gold standard image with the highest similarity for each test image as its prototype matching map.

[0057] In an embodiment of the present application, a structured report is output according to the risk stratification mapping result, a structured diagnosis report is automatically generated, and the content includes image interpretation results, main lesion categories, risk level judgments, visual interpretation (such as a prototype matching diagram), and personalized follow-up / treatment recommendations. The lesion category refers to, under the premise of being interpreted as a "substantial space-occupying", the lesion is further output as benign / malignant and a specific tumor category. The disease type is further obtained by manual input of a doctor. The personalized follow-up / treatment recommendation is a suggestion given by a professional doctor after judging the disease, and the doctor's suggestion is spliced into a treatment suggestion for the patient.

[0058] In an embodiment of the present application, the method further comprises: the process of automatically generating a structured diagnosis report according to the risk stratification mapping result comprises: integrating the preliminary predicted lesion categories in a standard format, and detailing the abnormal area description and lesion property information according to the prediction result, and combining the risk stratification model result to determine the final risk degree of the disease, and then automatically generating visual interpretation content matching the interpretation result. The visual interpretation content includes but is not limited to a prototype matching diagram or a risk heat map to enhance the interpretability of the report.

[0059] Further, according to the risk level and the lesion category, a corresponding personalized follow-up and treatment plan is automatically matched from a preset doctor professional suggestion text library, and when multiple diseases exist at the same time, multiple suggestions are integrated according to a preset rule to form a comprehensive treatment plan exclusive to the patient; finally, all the contents are integrated and output in a structured data format (such as JSON or XML) to ensure that it is compatible with the calling of an electronic medical record system, while supporting multiple forms such as text and PDF, facilitating the viewing and application of doctors and patients. Through this process, efficient, professional and personalized structured report generation based on existing diagnosis data is realized, significantly improving the quality and efficiency of clinical decision support and patient management.

[0060] In the embodiments of the present application, the terms "first", "second", and the like are used to distinguish items or similar items with basically the same function and effect. For example, the first model and the second model are only used to distinguish different models, and do not limit the order. Those skilled in the art can understand that the terms "first", "second", and the like do not limit the quantity and execution order, and the terms "first", "second", and the like do not necessarily mean different.

[0061] It should be noted that in the embodiments of the present application, the words "exemplary" or "for example" indicate an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0062] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of "and / or" describing the associated objects indicates that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent a, b, c, a-b, a-c, b-c or a-b-c, where a, b and c can be single or multiple.

[0063] Figure 3 is a schematic block diagram of an image-based eye lesion grading analysis device 300 provided by the embodiments of the present application. As shown in the figure, the device comprises a data acquisition module 301, a feature extraction module 302 and a grading analysis module 303. Figure 3

[0064] The data acquisition module 301 is configured to acquire one or more eye disease images.

[0065] The feature extraction module 302 is configured to input the eye disease image into a first encoder of a pre-trained first model to generate one or more corresponding eye feature vectors; when the eye disease image is one image, the feature extraction module 302 is configured to calculate the similarity between the eye feature vector and each preset eye disease category based on the first model, and extract one or more disease categories with the highest similarity as a prediction category; when the eye disease image is multiple images, the feature extraction module 302 is configured to calculate the similarity between each eye feature vector and each preset eye disease category based on the first model, input all the similarities into a second model to perform an adaptive weighted integration operation on all the similarities, generate an aggregated similarity score, and extract one or more disease categories with the highest similarity in the aggregated similarity score as a prediction category.

[0066] The grading analysis module 303 is configured to calculate the similarity between the eye feature vector or the aggregated feature model and each preset eye disease category based on the first model, and extract one or more disease categories with the highest similarity as a prediction category; perform a risk grading mapping operation according to the prediction category to generate a risk stratification mapping result; and output a structured report according to the risk stratification mapping result.

[0067] It should be understood that the specific processes of each module for performing the corresponding steps described above have been described in detail in the above method embodiments, and for the sake of brevity, will not be described here.

[0068] ​It should also be appreciated that the division of tasks between the modules in the embodiments of the present application is illustrative only and that other divisions of tasks between the modules are possible. In addition, the various functional modules in the various embodiments of the present application can be integrated in one processor, or can be physically present separately, or two or more modules can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module.

[0069] Figure 4 is a schematic block diagram of an electronic terminal provided by an embodiment of the present application. As shown in Figure 4 , the electronic terminal includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the apparatus are coupled together by a bus system 404. It can be appreciated that the bus system 404 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all the buses are marked as the bus system in Figure 4 .

[0070] The user interface 405 can include a display, a keyboard, a mouse, a trackball, a click gun, a key, a button, a touchpad, or a touch screen, etc.

[0071] It can be appreciated that the memory 402 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present application is intended to include but not limited to these and any other suitable categories of memory.

[0072] The memory 402 in the embodiment of the present application is configured to store various types of data to support the operation of the electronic terminal 400. Examples of the data include any executable program for operating on the electronic terminal 400, such as an operating system 4021 and an application program 4022. The operating system 4021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks. The application program 4022 can include various application programs, such as a media player (Media Player), a browser (Browser), and the like, for implementing various application services. The image-based eye lesion grading analysis method provided by the embodiment of the present application can be included in the application program 4022.

[0073] The method disclosed in the above embodiment of the present application can be applied to the processor 401 or implemented by the processor 401. The processor 401 can be an integrated circuit chip having a processing capability of signals. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 401 or an instruction in the form of software. The above processor 401 can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, and the like. The processor 401 can implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application. The general-purpose processor 401 can be a microprocessor or any conventional processor, and the like. The steps of the accessory optimization method provided in conjunction with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium, which is located in the memory. The processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0074] In the exemplary embodiments, the electronic terminal 400 can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), and the like, for executing the above method.

[0075] According to the method provided by the embodiment of the present application, the present application further provides a computer program product, which comprises computer program code, when the computer program code is executed on a computer, so that the computer executes the image-based eye lesion grading analysis method in any one of the above embodiments.

[0076] According to the method provided in the embodiments of the present application, the present application further provides a computer readable storage medium, which stores program codes, and when the program codes are run on a computer, the computer is caused to execute the image-based eye lesion grading analysis method in any one of the above-mentioned embodiments.

[0077] The terms "component," "module," "system," and the like as used herein are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, partially localized, and / or distributed across two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal).

[0078] Those of skill in the art would understand that the various illustrative logical blocks and steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. The choice of whether to implement the described functionality in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0080] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0082] In addition, the functional units in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0083] In the above embodiments, the functions of the functional units can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the whole or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as high-density digital video disc (Digital Video Disc, DVD), or semiconductor media (such as solid state disk (Solid State Disk, SSD), etc.

[0084] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0086] In summary, the present application provides an image-based eye lesion grading analysis method, device, medium, program product and terminal. One or more eye disease images are input into a pre-trained first model to generate corresponding eye feature vectors. For a single image, the similarity of the feature vector to each eye disease category is calculated, and the highest similarity of one or more predicted categories is output. For multiple images, the similarity of the feature vector to each category is calculated, and a second model is used to realize adaptive weighted integration to obtain the final predicted category. According to the predicted category, the sub-model is subdivided and the risk level is layered, and a structured risk stratification report is output. The present application solves the problem that the prior art cannot effectively evaluate the diagnostic value of multiple image information in complex cases. Through the adaptive weighted integration mechanism, the accuracy of ultrasound in identifying severe vitreous lesions and intraocular tumors is significantly improved, supporting high-risk patient screening and clinical decision-making, and expanding the application scenarios of ultrasound in ophthalmic diagnosis. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has high industrial utilization value.

[0087] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not intended to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical idea disclosed in the present application should be covered by the claims of the present application.

Claims

1. An image-based grading analysis method for ocular pathology, characterized in that, The method comprises the following steps: obtaining one or more eye disease images; inputting the eye disease image into a first encoder of a pre-trained first model to generate one or more eye feature vectors; when the eye disease image is one image, calculating the similarity between the eye feature vector and each preset eye disease category based on the first model, and extracting one or more disease categories with the highest similarity as the predicted category; when the eye disease image is multiple images, calculating the similarity between each eye feature vector and each preset eye disease category based on the first model, inputting all the similarities into a second model to perform adaptive weighted integration operation on all the similarities, generating an aggregated similarity score, and extracting one or more disease categories with the highest similarity in the aggregated similarity score as the predicted category; performing risk classification mapping operation according to the predicted category to generate a risk stratification mapping result; outputting a structured report according to the risk stratification mapping result.

2. The image-based grading analysis method of ocular lesions according to claim 1, characterized in that, The first model comprises a first encoder and a second encoder, and the pre-training process of the first model comprises the following steps: inputting an eye training image into the first encoder to generate an eye image feature; inputting corresponding multiple eye disease category texts into the second encoder to map the eye disease category texts to a feature space with the same dimension as the eye image feature to generate an eye disease text feature of each category; calculating the similarity between each eye image feature and each eye disease text feature; constructing a contrast loss function based on the similarity between each eye image feature and the eye disease text feature corresponding to its true category, and the similarity of other category text features; performing parameter update operation on the first encoder and the second encoder according to the contrast loss function to maximize the similarity between the eye image feature and the eye disease text feature corresponding to its true category, and minimize the similarity between the eye image feature and the eye disease text feature of non-corresponding categories.

3. The image-based grading analysis method of ocular pathology according to claim 2, wherein, The process of inputting all the similarities into the second model to perform adaptive weighted integration operation on all the similarities to generate an aggregated similarity score comprises the following steps: inputting all the similarities into an attention weight calculation model to generate an attention weight coefficient corresponding to the similarity of each eye disease image; performing weighting operation on the similarity of each eye disease image and the corresponding attention weight coefficient, and performing summation operation on the similarity vector after the weighting operation in the image dimension to generate an aggregated similarity score vector. 4.The image-based grading analysis method of eye lesions according to claim 1, wherein, The process of performing risk classification mapping operation according to the predicted category to generate a risk stratification mapping result comprises the following steps: determining whether the predicted category is vitreous opacity or substantial space-occupying; if the predicted category is vitreous opacity, inputting the eye feature vector into a third model to perform opacity degree recognition operation and outputting the corresponding opacity level; if the predicted category is substantial space-occupying, inputting the eye feature vector or the aggregated feature model into a fourth model to perform tumor subtype recognition operation and output the corresponding tumor subtype; classifying the eye disease image into multiple risk levels according to the predicted category, the opacity level and the tumor subtype.

5. The image-based grading analysis method of ocular pathology according to claim 4, wherein, The risk level includes one of normal, low-risk vision, high-risk vision, and intraocular tumor. 6.The image-based grading analysis method of eye lesions according to claim 1, wherein, The method further comprises the following steps: mapping the gradient-weighted feature map activation intensity corresponding to the predicted category back to the spatial position of the eye disease image, thereby generating a Grad-CAM heat map for each eye disease image to highlight the important regions that the model focuses on in the classification decision.

7. An image-based ocular lesion grading analysis apparatus, characterized by, The method comprises the following steps: a data acquisition module for acquiring one or more eye disease images; a feature extraction module for inputting the eye disease image into a first encoder of a pre-trained first model to generate one or more eye feature vectors; when the eye disease image is one image, calculating the similarity between the eye feature vector and each preset eye disease category based on the first model, and extracting one or more disease categories with the highest similarity as the predicted category; when the eye disease image is multiple images, calculating the similarity between each eye feature vector and each preset eye disease category based on the first model, inputting all similarities into a second model to perform adaptive weighted integration operation on all similarities to generate an aggregated similarity score, and extracting one or more disease categories with the highest similarity in the aggregated similarity score as the predicted category; a hierarchical analysis module for calculating the similarity between the eye feature vector or the aggregated feature model and each preset eye disease category based on the first model, and extracting one or more disease categories with the highest similarity as the predicted category; performing risk classification mapping operation according to the predicted category to generate risk stratification mapping result; outputting a structured report according to the risk stratification mapping result.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the image-based eye lesion hierarchical analysis method of any one of claims 1-6.

9. A computer program product, characterised in that, The computer program product comprises computer program code, which, when executed on a computer, causes the computer to implement the image-based eye lesion hierarchical analysis method of any one of claims 1-6.

10. An electronic terminal comprising a memory, a processor and a computer program stored on the memory, characterized in that, The processor executes the computer program to realize the image-based eye lesion hierarchical analysis method of any one of claims 1-6.

Images