Report generation and clinical evaluation method and device for fundus fluorescent angiography image

Abstract

The invention discloses a report generation and clinical evaluation method and device for fundus fluorescent angiography images, and belongs to the technical field of medical artificial intelligence, and the method comprises the steps: obtaining multi-modal clinical FFA data, and constructing the multi-modal clinical FFA data into a data set; constructing an FFA report generation model for generating an FFA report corresponding to an input FFA sequence image, and guiding model training by using the data set, loss based on a report generation task and comparative learning loss based on disease diagnosis supervision; inputting an FFA sequence image to be analyzed into the trained FFA report generation model to obtain a generated FFA report; and constructing a comprehensive clinical evaluation system comprising a natural language generation index, a clinical efficacy index taking a key focus as a core, an efficiency index assisted by artificial intelligence and a Like scale index, and carrying out precise clinical evaluation on the generated FFA report. According to the method, the FFA report can be automatically generated and comprehensively evaluated, and the generation quality and reliability of the ophthalmology report are improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical artificial intelligence technology, and specifically relates to a method and device for report generation and clinical evaluation of fundus fluorescein angiography images. Background Art

[0002] Fundus Fluorescein Angiogram (FFA) is the "gold standard" for retinal vascular visualization. It visualizes the fundus vascular system by intravenous injection of fluorescent dyes, provides hemodynamic information of the retina and choroid, and is the key to diagnosing various chorioretinal diseases. Compared with other conventional retinal examinations, FFA takes multiple FFA sequence images at different phases and locations based on its development characteristics in clinical practice, providing more dynamic and rich fundus vascular information, but it is also accompanied by more complex and time-consuming interpretation, which requires the ophthalmologist's rich clinical experience and professional level. Given that it takes a lot of time for doctors to diagnose and write high-quality FFA reports, and there is a relative shortage of ophthalmologists with rich clinical experience, especially in remote or densely populated areas, this situation urgently requires a reliable and scalable artificial intelligence-assisted system for FFA sequence image interpretation.

[0003] In recent years, with the rapid development of artificial intelligence technology, medical report generation models have gradually become a research hotspot. However, most of the current medical report generation models are for chest X-ray images and the corresponding report text is in English, which is mainly limited by the difficulty of collecting and obtaining private clinical data. In the field of ophthalmology, some studies have begun to develop automatic report generation methods for eye images, such as OCT, ultrasound, color fundus photography, and FFA, but the overall research is still in its infancy, and the accuracy and reliability of the report need to be further improved. For example, a Chinese patent document with publication number CN118136200A discloses a diagnostic report generation method, device and medium, including: obtaining actual color fundus images, actual fundus fluorescein angiography images, and corresponding actual clinical features, on the one hand, by constructing a multi-label diagnostic classification model for fundus diseases to achieve diagnostic classification, on the other hand, by constructing a report generation model based on a knowledge memory network and a ternary edge loss function to achieve intelligent generation of diagnostic reports. This method comprehensively utilizes multimodal information to more comprehensively capture the characteristics of fundus diseases, thereby improving the accuracy of diagnosis and report generation.

[0004] However, the above methods and other existing medical report generation studies have the problem of incomplete model performance evaluation. These studies have focused on improving the textual similarity between generated reports and real reports, namely the natural language generation (NLG) index, while the NLG index cannot directly reflect the clinical value of the generated reports. At present, there is no comprehensive evaluation method and evaluation system for the quality of generated medical reports, which seriously hinders the reliability and clinical applicability of medical report generation models. Therefore, in order to further promote the effective application of artificial intelligence-assisted systems in FFA interpretation, it is urgent to further design automatic report generation and accurate clinical evaluation methods for fundus fluorescein angiography images. Summary of the invention

[0005] In view of the above, the purpose of the present invention is to provide a method and device for report generation and clinical evaluation of fundus fluorescein angiography images, by training an FFA report generation model through a comparative learning method based on diagnostic supervision to achieve automatic generation of FFA reports, and construct a comprehensive evaluation method to evaluate the generated FFA reports, thereby improving the generation quality and reliability of ophthalmic reports, and providing clinical decision support with more clinical value.

[0006] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows: An embodiment of the present invention provides a method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images, comprising the following steps: Obtain multimodal clinical FFA data including FFA sequence images and their corresponding real FFA reports and disease diagnosis labels and construct them into a dataset after preprocessing; Construct an FFA report generation model to generate FFA reports corresponding to the input FFA sequence images, and use the dataset and the loss based on the report generation task and the contrastive learning loss based on disease diagnosis supervision to guide the training of the FFA report generation model. Input the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report; A comprehensive clinical evaluation system is constructed, which includes an automatic evaluation method based on natural language generation, an automatic clinical efficacy evaluation method with key lesions as the core, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale, to conduct accurate clinical evaluation of the generated FFA reports.

[0007] Preferably, the multimodal clinical FFA data including FFA sequence images and their corresponding real FFA reports and disease diagnosis labels are obtained and constructed into a data set after preprocessing, including: Obtain multimodal clinical FFA data generated by a clinical patient's FFA examination, including FFA sequence images of different phases and a corresponding real FFA report and the patient's eye disease diagnosis label, and construct the multimodal clinical FFA data of several patients into an original multimodal FFA dataset; The original multimodal FFA dataset was organized into a monocular multimodal FFA dataset. The FFA sequence images and real FFA reports in each monocular multimodal FFA dataset were subjected to image preprocessing and text preprocessing respectively, and then constructed into a dataset together with the patient's monocular disease diagnosis label.

[0008] Preferably, the FFA report generation model includes an input layer, an image feature extractor, an encoder, a decoder and an output layer connected in sequence. The FFA sequence image is input into the image feature extractor through the input layer to extract the image features. The image features are further encoded into a high-dimensional feature representation through the encoder to capture the key lesion information and its association in the image. The high-dimensional feature representation is then decoded by the decoder and converted into a report text in natural language form. Finally, the complete FFA report is output through the output layer.

[0009] Preferably, the method of using a dataset and a loss based on a report generation task and a contrastive learning loss based on disease diagnosis supervision to guide the training of the FFA report generation model comprises: The FFA sequence images in the dataset are input into the FFA report generation model and the corresponding FFA reports are generated. The loss based on the report generation task is calculated based on the real FFA reports in the dataset and the FFA reports generated by the FFA report generation model. ; According to the disease diagnosis labels in the dataset, positive image pairs and negative image pairs are divided, where the positive sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to the same category of disease diagnosis labels, and the negative sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to different categories of disease diagnosis labels, and a contrastive learning loss based on disease diagnosis supervision is constructed. Used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs; Will and Combined into total loss Used to guide the training of the FFA report generation model.

[0010] Preferably, the automatic evaluation method based on natural language generation includes: The new FFA reports generated by the FFA report generation model are evaluated using natural language generation indicators, including the BLEU-n indicator, the METEOR indicator, and the ROUGE-L indicator.

[0011] Preferably, the automatic clinical efficacy evaluation method with key lesions as the core includes: The key lesion identifier is used to automatically identify and standardize the key lesion descriptions pre-defined by experts from the real FFA reports and the FFA reports generated by the model. If the real FFA report and the FFA report generated by the model contain the same key lesion descriptions, it is considered that the FFA report generation model has accurately learned the key lesions presented in the FFA sequence images, and the clinical efficacy indicators including sensitivity, specificity, F1 score and AUC are calculated.

[0012] Preferably, the key lesion descriptions include: hemorrhage, microaneurysm, non-perfused area, new blood vessels, vascular occlusion, laser spots, vascular deformation, fluorescence leakage, and no obvious abnormality.

[0013] Preferably, the report writing and diagnostic testing method assisted by artificial intelligence includes: In the AI-assisted mode, the FFA report generation model automatically generates an FFA report. The doctor can judge the quality of the FFA report generated by the model based on the corresponding FFA sequence images, and then cite, modify or abandon it, and then obtain a report reviewed by the doctor, and finally give a disease diagnosis; In the conventional mode, the FFA report generated by the model is not provided, and the doctor is solely responsible for writing the report and finally giving the disease diagnosis; By recording the time required to write a report for each FFA sequence image and the disease diagnosis results under the two modes, the time efficiency and diagnostic performance improvement effect of the FFA report generation model in assisting doctors in report writing can be determined.

[0014] Preferably, the expert scoring method based on the Likert scale includes: By reading FFA sequence images and real FFA reports, experts used Likert scale to score the FFA reports written by doctors themselves, the FFA reports generated by the FFA report generation model, and the FFA reports reviewed by doctors with the assistance of the FFA report generation model, and evaluated the report generation quality based on the scoring results.

[0015] To achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a report generation and clinical evaluation device for fundus fluorescein angiography images, which is implemented by the above-mentioned report generation and clinical evaluation method for fundus fluorescein angiography images, and includes: a data set construction module, a model training module, a model prediction module and a clinical evaluation module; The data set construction module is used to obtain multimodal clinical FFA data including FFA sequence images and their corresponding FFA reports and disease diagnosis labels and construct them into a data set after preprocessing; The model training module is used to construct an FFA report generation model for generating an FFA report corresponding to an input FFA sequence image, and uses a data set and a loss based on a report generation task and a contrastive learning loss based on disease diagnosis supervision to guide the FFA report generation model to be trained; The model prediction module is used to input the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report; The clinical evaluation module is used to construct a comprehensive clinical evaluation system including an automatic evaluation method based on natural language generation, an automatic clinical efficacy evaluation method centered on key lesions, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale, so as to perform accurate clinical evaluation on the generated FFA report.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses the loss of the report generation task and the contrastive learning loss based on disease diagnosis supervision to guide the training of the FFA report generation model, which not only ensures the grammatical correctness and completeness of the pathological description of the generated text, but also strengthens the semantic alignment of images with the same disease diagnosis through disease diagnosis supervised contrastive learning, ensuring that the clinical diagnostic features learned by the model are highly consistent with the real medical labels in the feature space, thereby improving the accuracy of report generation.

[0017] (2) Based on the automatic evaluation method based on natural language generation, the present invention further constructs an automatic evaluation method for clinical efficacy with key lesions as the core, which can make up for the deficiency of only evaluating the clinical value of FFA reports through text quality evaluation, focus on the substantial impact of reports on clinical decision-making, and improve the clinical value and clinical applicability of reports. It is further combined with the report writing and diagnostic test methods assisted by artificial intelligence, and the comprehensive clinical evaluation system of the expert scoring method based on the Likert scale, which can verify the clinical equivalence and reliability of the report generated by the model from multiple dimensions and multiple levels, filling the gap in the clinical value evaluation method of the medical report generation model, and helping to promote the clinical application research and development of the medical report generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1It is a flowchart of a method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a model training framework of an FFA report generation model based on disease diagnosis supervised contrast learning provided by an embodiment of the present invention; Figure 3 is a distribution diagram of expert scoring results based on a five-level Likert scale provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of a report generation and clinical evaluation device for fundus fluorescein angiography images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0021] The inventive concept of the present invention is: in view of the problems of low accuracy and quality of ophthalmic report generation in the prior art and lack of clinical value and applicability, the embodiment of the present invention provides a report generation and clinical evaluation method and device for fundus fluorescein angiography images, simulates the decision-making process of ophthalmologists in clinical practice, and proposes a method based on diagnostic supervised comparative learning to guide model training and generate FFA reports. In addition, based on traditional natural language generation indicators, not only a clinical efficacy indicator with key lesions as the core is proposed, but also a method for measuring the impact of report generation on clinicians' FFA report writing time and diagnostic accuracy is proposed, and expert scoring is performed, which can realize the automatic generation of FFA reports and realize a comprehensive evaluation of the clinical value of the generated reports, thereby improving the quality of ophthalmic report generation and using them for clinical decision support.

[0022] Figure 1 FIG. 1 is a flow chart of a method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a method for generating a report and clinically evaluating fundus fluorescein angiography images, comprising the following steps: S1, obtain multimodal clinical FFA data including FFA sequence images and their corresponding real FFA reports and disease diagnosis labels and construct them into a dataset after preprocessing.

[0023] S11, collected multimodal clinical FFA data generated by an FFA examination of 1,164 clinical patients in an ophthalmology hospital. Each data set includes: 5 to 16 FFA sequence images in different phases, a corresponding real FFA text report, and the patient's eye disease diagnosis as a label. Disease diagnoses include: normal, proliferative / non-proliferative diabetic retinopathy (DR), wet / dry age-related macular degeneration (AMD), branch retinal vein occlusion (BRVO), central retinal vein occlusion (CRVO), central serous chorioretinopathy (CSC) or Koyanagi-Harada syndrome (VKH). These collected multimodal FFA datasets were constructed into original multimodal FFA datasets after data review by doctors. .

[0024] S12, original multimodal FFA dataset Organized into a monocular multimodal FFA dataset That is, for patients who have undergone FFA examinations on both eyes, the original multimodal FFA data of both eyes are split into one left eye data and one right eye data. Finally, 14,684 FFA sequence images and 1,453 FFA reports and their corresponding disease diagnoses are obtained.

[0025] S13, multimodal FFA dataset for each monocular 5 to 16 FFA sequence images of different phases are preprocessed, sorted and named according to the imaging shooting time, irrelevant information around the image is cut off, and the image size is kept at 768×768, thereby obtaining the FFA sequence image set. .

[0026] S14, multimodal FFA dataset for each monocular The real FFA text reports in the dataset were preprocessed, including removing words without clinical significance, correcting typos, unifying the expression of multiple words with the same meaning, and analyzing word frequency, and then the FFA text report set was obtained. .

[0027] S15: Use the patient's medical number and eye type as unique identifiers and merge the FFA sequence image set based on the unique identifiers. , FFA Text Report Collection , and the corresponding eye disease diagnosis labels, constitute the data set of the FFA report generation model .

[0028] S16, divide the data set into training sets according to a certain ratio , test set and validation set In the embodiment, the division ratio is 3:1:1.

[0029] S2, construct an FFA report generation model to generate FFA reports corresponding to the input FFA sequence images, and use the dataset and the loss based on the report generation task as well as the contrastive learning loss based on disease diagnosis supervision to guide the training of the FFA report generation model.

[0030] S21, construct the FFA report generation model, such as Figure 2 As shown, it includes an input layer, an image feature extractor, an encoder, a decoder and an output layer connected in sequence, the FFA sequence image is input into the image feature extractor through the input layer to extract the image features, the image features are further encoded into a high-dimensional feature representation through the encoder to capture the key lesion information and its association in the image, and then the high-dimensional feature representation is decoded by the decoder to convert it into a report text in natural language form, and finally the complete FFA report is output through the output layer. Among them, the image feature extractor includes a feature extraction module and a pooling layer of a pre-trained model (the ResNet101 model pre-trained on ImageNet-1k is used in the embodiment); the encoder includes a plurality of identical encoding layers stacked (six are used in the embodiment), each encoding layer includes a multi-head attention mechanism, a feedforward neural network, layer normalization and a residual connection; the decoder includes a relational memory network and a plurality of identical decoding layers stacked (six are used in the embodiment), each decoding layer also includes a multi-head attention mechanism, a feedforward neural network, a layer normalization and a residual connection.

[0031] S22, initializing the FFA report generation model.

[0032] S23, the training set The FFA sequence images and corresponding disease diagnosis labels in the are input into the FFA report generation model for forward propagation calculation to generate the corresponding FFA report.

[0033] S24, calculate the loss based on the report generation task based on the real FFA reports in the dataset and the FFA reports generated by the FFA report generation model , the standard cross entropy loss is used in the embodiment. And the positive image pairs and negative image pairs are divided according to the disease diagnosis labels in the dataset, where the positive sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to the same category of disease diagnosis labels, and the negative sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to different categories of disease diagnosis labels, and the contrastive learning loss based on disease diagnosis supervision is constructed. It is used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, that is, to simulate the decision-making process of ophthalmologists in real clinical practice and conduct supervised learning based on disease diagnosis, so that FFA cases with the same disease diagnosis labels are closer in clinical content and more in line with the characteristics of FFA cases.

[0034] Will and Combined into total loss Used to guide FFA report generation model training, total loss It is expressed as: , in, It represents the weight parameter used to balance the two losses and is set to 0.2 in the embodiment.

[0035] S25, the total loss of the generated model by FFA report Back propagation is performed to update the network parameters, where Only used to update the parameters of the image feature extractor. Take the maximum number of iterations (In the embodiment, 30 rounds of iterations are performed) in the validation set The best performing model is used as the trained optimal FFA report generation model.

[0036] S3, inputting the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report.

[0037] S31, the test set The FFA sequence images in the training report generation model are input to generate the corresponding FFA report. The FFA report includes the description of the lesions in the FFA sequence images, for example: the right eye shows inferotemporal branch retinal vein occlusion, venous circulation disorder in the blocked area, large non-perfused areas, vascular leakage and macular edema.

[0038] S4, build a comprehensive clinical evaluation system that includes an automatic evaluation method based on natural language generation, an automatic evaluation method for clinical efficacy with key lesions as the core, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale to conduct accurate clinical evaluation of the generated FFA report.

[0039] S41, construct an automatic evaluation method based on natural language generation (NLG), and use NLG indicators to evaluate the new FFA reports generated by the FFA report generation model. NLG indicators include BLEU-n indicators (including BLEU-1, BLEU-2, BLEU-3, BLEU-4), METEOR indicators and ROUGE-L indicators. Among them, BLEU-n measures the overlap of n word sequences between generated text and real text, and is widely used in the quality of translated text and reference text. METEOR enhances BLEU-n by considering synonyms and word form changes, allowing more flexible evaluation of generated FFA reports. ROUGE-L emphasizes recall and is suitable for evaluating the coherent description of complex clinical information.

[0040] S42, build an automatic clinical efficacy evaluation method with key lesions as the core, and use the key lesion identifier to automatically identify and standardize the key lesion descriptions (including the positive and negative nature of the lesions) pre-defined by experts from the real FFA reports and the FFA reports generated by the model, so as to evaluate whether the report generation model has accurately learned the key lesions presented in the FFA sequence images (recorded in the real text report). The key lesion descriptions include: hemorrhage, microaneurysm, non-perfused area, new blood vessels, vascular occlusion, laser spots, vascular deformation, fluorescence leakage, and no obvious abnormalities. If the real FFA report and the FFA report generated by the model contain the same key lesion descriptions, it is considered that the FFA report generation model has accurately learned the key lesions presented in the FFA sequence images. Therefore, unlike the NLG indicator, the CE indicator reflects the accuracy of the clinical content more. The CE indicator includes classification indicators such as sensitivity, specificity, F1 score, and AUC.

[0041] S43, construct an AI-assisted report writing and diagnosis test method, by establishing a report writing and diagnosis tool and inviting two residents who were not involved in the study design to evaluate it in AI-assisted mode and conventional mode. In AI-assisted mode, the FFA report is automatically generated by the FFA report generation model. The doctor judges the quality of the FFA report generated by the model based on the corresponding FFA sequence images, and performs citation, modification or abandonment operations to obtain a report reviewed by the doctor, and finally gives a disease diagnosis. In conventional mode, the FFA report generated by the model is not provided, and the doctor is solely responsible for writing the report independently, and finally giving a disease diagnosis. By recording the time required for report writing and the disease diagnosis results for each FFA sequence image in the two modes, the time efficiency of the FFA report generation model in assisting doctors in report writing and the improvement effect of diagnostic accuracy can be judged.

[0042] S44, construct an expert scoring method based on the Likert scale, using a five-level Likert scale, that is, score the report quality according to five options: 1 very inconsistent, 2 not quite consistent, 3 basically consistent, 4 relatively consistent, 5 very consistent. In the embodiment, a total of 6 senior ophthalmologists are invited as experts. Each senior ophthalmologist can view the FFA sequence images and corresponding diagnoses of the FFA case data in the test set, but knows nothing about the report type information (FFA reports written by doctors themselves, FFA reports generated by the FFA report generation model, and FFA reports obtained after doctor review with the assistance of the FFA report generation model). Each senior ophthalmologist scores the three types of reports in a separate random order using the Likert scale, and the quality of the three reports can be evaluated based on the expert scoring results.

[0043] S45, based on the four evaluation methods proposed in steps S41 to S44, the FFA report generated by the model in step S31 is accurately clinically evaluated, thereby providing support for the clinical application of the FFA report generation model based on diagnostic supervised comparative learning.

[0044] In order to further verify the effectiveness of the report generation and clinical evaluation method for fundus fluorescein angiography images provided by the embodiment of the present invention, the method of the present invention is compared with three existing typical medical report generation models: R2Gen (reference title: Generating Radiology Reports via Memory-driven Transformer), R2GenRL (reference title: Reinforced Cross-Modal Alignment for Radiology Report Generation), and M2KT (reference title: Radiology report generation with a learned knowledge base and multi-modal alignment). Table 1 shows the performance of these models on NLG and CE indicators. It can be seen that the method proposed in the present invention has achieved the best performance: in NLG indicators, BLEU-1=0.549, BLEU-2=0.443, BLEU-3=0.373, BLEU-4=0.321, METEOR=0.342, ROUGE-L=0.540; in CE indicators, sensitivity=0.826, specificity=0.837, F1 score=0.790, AUC=0.831. Since clinical data sets usually present imbalance problems, this affects sensitivity and specificity, making these two indicators unable to fully reflect the true performance of the model, while the F1 score and AUC, which combine the two indicators of sensitivity and specificity, can better comprehensively evaluate the performance of the model. The method of the present invention performs best in F1 score and AUC, which fully demonstrates that the method of the present invention can effectively process unbalanced clinical data, has the best overall effect, and has clinical effectiveness.

[0045] Table 1 Performance comparison of FFA report generation model and existing models on NLG and CE indicators

[0046] In terms of report writing and diagnostic testing assisted by artificial intelligence, with the assistance of the method of the present invention, on the one hand, the average writing time of each report for two residents was shortened from 153.93±14.67 seconds to 108.08±19.29 seconds, which improved the efficiency of the residents by about 30%; on the other hand, the average diagnostic accuracy was improved from 85.55% to 90.34%, which shows that the method of the present invention significantly improved the average diagnostic accuracy of residents for FFA sequence images.

[0047] In terms of expert scoring based on the five-level Likert scale, Table 2 shows the scoring results of six senior ophthalmologists on three types of FFA reports: the score of the FFA report generated by the method of the present invention was 4.12±1.29, the score of the report written by the resident doctor was 4.38±1.09, and the score of the report reviewed by the resident doctor with the assistance of the method of the present invention was 4.46±0.99. This shows that the FFA report generated by the method of the present invention is more consistent with the real FFA report, close to the level of the resident doctor, and the method of the present invention effectively assists the resident doctor and can improve the quality of his report writing.

[0048] Table 2 Scoring results of three FFA reports by senior ophthalmologists

[0049] Based on the same inventive concept, Figure 4 As shown, an embodiment of the present invention further provides a report generation and clinical evaluation device 400 for fundus fluorescein angiography images, comprising: a data set construction module 410 , a model training module 420 , a model prediction module 430 and a clinical evaluation module 440 .

[0050] The data set construction module 410 is used to obtain multimodal clinical FFA data including FFA sequence images and their corresponding FFA reports and disease diagnosis labels and construct them into a data set after preprocessing.

[0051] The model training module 420 is used to construct an FFA report generation model for generating an FFA report corresponding to the input FFA sequence image, and to guide the training of the FFA report generation model using a dataset and a loss based on the report generation task and a contrastive learning loss based on disease diagnosis supervision.

[0052] The model prediction module 430 is used to input the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report.

[0053] The clinical evaluation module 440 is used to construct a comprehensive clinical evaluation system including an automatic evaluation method based on natural language generation, an automatic clinical efficacy evaluation method centered on key lesions, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale, so as to conduct accurate clinical evaluation of the generated FFA report.

[0054] It should be noted that the report generation and clinical evaluation device for fundus fluorescein angiography images provided in the above-mentioned embodiment and the report generation and clinical evaluation method for fundus fluorescein angiography images belong to the same inventive concept. The specific implementation process is detailed in the embodiment of the report generation and clinical evaluation method for fundus fluorescein angiography images, which will not be repeated here.

[0055] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating reports and clinically evaluating fundus fluorescein angiography images, characterized in that: The following steps are involved: Obtain multimodal clinical FFA data including FFA sequence images and their corresponding real FFA reports and disease diagnosis labels and construct them into a dataset after preprocessing; Construct an FFA report generation model to generate FFA reports corresponding to the input FFA sequence images, and use the dataset and the loss based on the report generation task and the contrastive learning loss based on disease diagnosis supervision to guide the training of the FFA report generation model. Input the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report; A comprehensive clinical evaluation system is constructed, which includes an automatic evaluation method based on natural language generation, an automatic clinical efficacy evaluation method with key lesions as the core, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale, to conduct accurate clinical evaluation of the generated FFA reports.

2. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: The method of obtaining multimodal clinical FFA data including FFA sequence images and their corresponding real FFA reports and disease diagnosis labels and constructing a data set after preprocessing includes: Obtain multimodal clinical FFA data generated by a clinical patient's FFA examination, including FFA sequence images of different phases and a corresponding real FFA report and the patient's eye disease diagnosis label, and construct the multimodal clinical FFA data of several patients into an original multimodal FFA dataset; The original multimodal FFA dataset was organized into a monocular multimodal FFA dataset. The FFA sequence images and real FFA reports in each monocular multimodal FFA dataset were subjected to image preprocessing and text preprocessing respectively, and then constructed into a dataset together with the patient's monocular disease diagnosis label.

3. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: The FFA report generation model includes an input layer, an image feature extractor, an encoder, a decoder, and an output layer connected in sequence. The FFA sequence images are input into the image feature extractor through the input layer to extract image features. The image features are further encoded into high-dimensional feature representations through the encoder to capture the key lesion information and its associations in the image. The high-dimensional feature representations are then decoded by the decoder and converted into a report text in natural language form. Finally, the complete FFA report is output through the output layer.

4. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, wherein: The FFA report generation model is trained using a dataset and a loss based on the report generation task and a contrastive learning loss based on disease diagnosis supervision, including: The FFA sequence images in the dataset are input into the FFA report generation model and the corresponding FFA reports are generated. The loss based on the report generation task is calculated based on the real FFA reports in the dataset and the FFA reports generated by the FFA report generation model. ; According to the disease diagnosis labels in the dataset, positive image pairs and negative image pairs are divided, where the positive sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to the same category of disease diagnosis labels, and the negative sample pairs represent the FFA sequence images and the real FFA reports in the dataset belonging to different categories of disease diagnosis labels, and a contrastive learning loss based on disease diagnosis supervision is constructed. Used to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs; Will and Combined into total loss Used to guide the training of the FFA report generation model.

5. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: Automatic evaluation methods based on natural language generation include: The new FFA reports generated by the FFA report generation model are evaluated using natural language generation indicators, including the BLEU-n indicator, the METEOR indicator, and the ROUGE-L indicator.

6. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: The automatic clinical efficacy evaluation methods focusing on key lesions include: The key lesion identifier is used to automatically identify and standardize the key lesion descriptions pre-defined by experts from the real FFA reports and the FFA reports generated by the model. If the real FFA report and the FFA report generated by the model contain the same key lesion descriptions, it is considered that the FFA report generation model has accurately learned the key lesions presented in the FFA sequence images, and the clinical efficacy indicators including sensitivity, specificity, F1 score and AUC are calculated.

7. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 6, characterized in that: Key lesion descriptions include: hemorrhage, microaneurysm, non-perfused area, neovascularization, vascular occlusion, laser spot, vascular deformation, fluorescence leakage, and no obvious abnormality.

8. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: AI-assisted report writing and diagnostic testing methods include: In the AI-assisted mode, the FFA report generation model automatically generates an FFA report. The doctor can judge the quality of the FFA report generated by the model based on the corresponding FFA sequence images, and then cite, modify or abandon it, and then obtain a report reviewed by the doctor, and finally give a disease diagnosis; In the conventional mode, the FFA report generated by the model is not provided, and the doctor is solely responsible for writing the report and finally giving the disease diagnosis; By recording the time required to write a report for each FFA sequence image and the disease diagnosis results under the two modes, the time efficiency and diagnostic performance improvement effect of the FFA report generation model in assisting doctors in report writing can be determined.

9. The method for generating a report and conducting a clinical evaluation of fundus fluorescein angiography images according to claim 1, characterized in that: Expert rating methods based on Likert scales include: By reading FFA sequence images and real FFA reports, experts used Likert scale to score the FFA reports written by doctors themselves, the FFA reports generated by the FFA report generation model, and the FFA reports reviewed by doctors with the assistance of the FFA report generation model, and evaluated the report generation quality based on the scoring results.

10. A device for generating a report and clinically evaluating fundus fluorescein angiography images, implemented by the method for generating a report and clinically evaluating fundus fluorescein angiography images according to any one of claims 1 to 9, characterized in that: include: Dataset construction module, model training module, model prediction module and clinical evaluation module; The data set construction module is used to obtain multimodal clinical FFA data including FFA sequence images and their corresponding FFA reports and disease diagnosis labels and construct them into a data set after preprocessing; The model training module is used to construct an FFA report generation model for generating an FFA report corresponding to an input FFA sequence image, and uses a data set and a loss based on a report generation task and a contrastive learning loss based on disease diagnosis supervision to guide the FFA report generation model to be trained; The model prediction module is used to input the FFA sequence images to be analyzed into the trained FFA report generation model to obtain a generated FFA report; The clinical evaluation module is used to construct a comprehensive clinical evaluation system including an automatic evaluation method based on natural language generation, an automatic clinical efficacy evaluation method centered on key lesions, an artificial intelligence-assisted report writing and diagnostic testing method, and an expert scoring method based on the Likert scale, so as to perform accurate clinical evaluation on the generated FFA report.

Images