Multimodal ophthalmic base model, training method, and apparatus for cross-scale contrast learning

Abstract

The application provides a multi-modal ophthalmic base model for cross-scale contrast learning, a training method and device, and belongs to the field of artificial intelligence. The method comprises the following steps: obtaining a target fundus image and a target description text corresponding to the target fundus image; inputting the target fundus image and the target description text into an ophthalmic model to obtain a diagnosis report for an eye disease output by the ophthalmic model, wherein the ophthalmic model is trained based on local image features and global image features of reference fundus images in training data and global text features and local text features of diagnosis reports corresponding to the reference fundus images. The accuracy of the generation of the diagnosis report for the eye disease can be improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, specifically relating to a multimodal ophthalmology basic model, training method and device for cross-scale contrastive learning. Background Technology

[0002] With the development of artificial intelligence technology, deep learning has shown broad application prospects in the field of medical image analysis, especially in the auxiliary diagnosis of ophthalmic diseases. Fundus images, as an important non-invasive and low-cost examination method, are commonly used to diagnose various ophthalmic diseases, such as glaucoma, diabetic retinopathy, and macular degeneration. In recent years, large-scale visual baseline models have been continuously introduced into the field of ophthalmology.

[0003] In clinical medical practice, doctors' diagnoses do not rely solely on images, but rather on reasoning and judgment based on multimodal information such as images and medical records. However, most current ophthalmological models are still primarily single-modal, lacking the ability to model semantic relationships between images and text, which limits their understanding of complex clinical scenarios and their ability to adapt to multiple tasks.

[0004] With the development of cross-modal contrastive learning methods such as Contrastive Language–Image Pre-training (CLIP), image-text alignment and joint modeling have become important directions in multimodal learning. Contrastive learning, by constructing positive and negative image-text pairs, can effectively improve the model's cross-modal semantic modeling ability, providing a new approach for building multimodal medical models with open-ended understanding capabilities. However, most existing medical image-text contrastive learning models adopt a coarse-grained approach of globally aligning images with complete text, making it difficult to precisely capture the complex semantic mapping relationships between images and text. Summary of the Invention

[0005] This application provides a multimodal ophthalmology basic model, training method, and apparatus for cross-scale contrastive learning, which can improve the accuracy of the model's output results for eye diseases.

[0006] This application provides a multimodal ophthalmological foundational model for cross-scale contrastive learning, including: Image encoders and text encoders; The image encoder is used to acquire global image features for the entire region of the fundus image and local image features for each sub-region of the fundus image; The text encoder is used to obtain global text features of the diagnostic report corresponding to the fundus image, and local text features of each sentence in the diagnostic report; The multimodal ophthalmology basic model is trained based on the local image features, the local text features, the global text features, and the local text features.

[0007] According to the multimodal ophthalmology basic model of cross-scale contrastive learning provided in this application, the multimodal ophthalmology basic model is used to: acquire a target fundus image and a target description text corresponding to the target fundus image; and output a diagnostic report for an eye disease based on the target fundus image and the target description text, using global text features and local text features trained from the diagnostic report corresponding to the reference fundus image.

[0008] According to the multimodal ophthalmology basic model of cross-scale contrastive learning provided in this application, the multimodal ophthalmology basic model includes an image-text feature alignment module, which is used to: determine a global loss function based on the global image features and the global text features; determine a local loss function based on the local image features and the local text features; and determine a total loss function based on the global loss function and the local loss function, wherein the multimodal ophthalmology basic model is trained based on the total loss function.

[0009] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the global loss function based on the global image features and the global text features, the image-text feature alignment module is used to: obtain the text label of each diagnostic report and the image label of each fundus image; determine the set of diagnostic reports corresponding to each global image feature based on the text label, wherein the diagnostic reports included in the set of diagnostic reports have the same text label; determine the set of fundus images corresponding to each global text feature based on the image label, wherein the fundus images included in the set of fundus images have the same image label; and determine the global loss function based on each global image feature, each global text feature, the set of diagnostic reports, and the set of fundus images.

[0010] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the global loss function based on each of the global image features, each of the global text features, the diagnostic report set, and the fundus image set, the image-text feature alignment module is configured to: for each of the global image features, obtain a first cosine similarity between the global image feature and each of the global text features, and obtain a second cosine similarity between the global text feature corresponding to each of the diagnostic report sets corresponding to the global image feature and the global image feature; for each of the global text features, obtain a third cosine similarity between the global text feature and each of the global image features, and obtain a fourth cosine similarity between the global image feature corresponding to each of the fundus images in the fundus image set corresponding to the global text feature and the global text feature; and determine the global loss function based on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity.

[0011] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in the step of determining a local loss function based on the local image features and the local text features, the image-text feature alignment module is used to: determine the set of sentences corresponding to the local image features, wherein the local text features corresponding to each sentence in the set of sentences belong to the same category; determine the set of image sub-regions corresponding to the local text features, wherein the local image features corresponding to each image sub-region in the set of image sub-regions belong to the same category; and determine a local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions.

[0012] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions, the image-text feature alignment module is used to: for each local image feature, obtain the fifth cosine similarity between the local image feature and each local text feature, and obtain the sixth cosine similarity between the local text feature corresponding to each sentence in the set of sentences corresponding to the local image feature and the local image feature; for each local text feature, obtain the seventh cosine similarity between the local text feature and each local image feature, and obtain the eighth cosine similarity between the local image feature corresponding to each image sub-region in the set of image sub-regions corresponding to the local text feature and the local text feature; and determine the local loss function based on the fifth cosine similarity, the sixth cosine similarity, the seventh cosine similarity, and the eighth cosine similarity.

[0013] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the set of sentences corresponding to the local image features, the image-text feature alignment module is used to: determine the reference sentences corresponding to the local text features; perform clustering processing on the reference sentences to obtain clustering results; and determine the set of sentences corresponding to the local image features based on the reference sentences and the clustering results.

[0014] Based on the multimodal ophthalmology basic model of cross-scale contrastive learning provided in this application, in terms of clustering the reference sentences to obtain clustering results, the image-text feature alignment module is used to: encode each sentence of each diagnostic report to obtain reference features; and cluster each sentence using the K-means clustering algorithm based on the reference features to obtain clustering results.

[0015] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the set of statements corresponding to the local image features, the image-text feature alignment module is used to: obtain multiple fundus structure types included in the diagnostic report, and a description of each fundus structure type in each diagnostic report; determine the reference fundus structure type included in the statement corresponding to the local text feature, and a reference description of the reference fundus structure type; and determine the set of statements corresponding to the local image feature based on the description of each fundus structure type and the reference description of the reference fundus structure type.

[0016] According to the multimodal ophthalmological basic model of cross-scale contrastive learning provided in this application, in determining the set of sentences corresponding to the local image features based on the description of each fundus structure type and the reference description of the reference fundus structure type, the image-text feature alignment module is used to: determine a first vector corresponding to each sentence of each diagnostic report based on the description of each fundus structure type; determine a second vector of the sentences corresponding to the local text features based on the reference description of the reference fundus structure type; and determine the set of sentences corresponding to the local image features based on the first vector and the second vector.

[0017] This application also provides a method for training a multimodal ophthalmic basic model using cross-scale contrastive learning, including: Acquire training data, which includes multiple fundus images and a corresponding diagnostic report for each fundus image; For each fundus image, global image features of the complete region of the reference fundus image and local image features of each sub-region of the fundus image are obtained according to the image encoder. For each diagnostic report, global text features for the diagnostic report and local text features for each sentence in the diagnostic report are obtained according to the text encoder; A multimodal ophthalmology basic model is trained based on the local image features, the local text features, the global text features, and the local text features.

[0018] This application also provides a training device for a multimodal ophthalmic basic model of cross-scale contrastive learning, including: The first acquisition unit is used to acquire training data, which includes multiple fundus images and a diagnostic report corresponding to each fundus image; The second acquisition unit is used to acquire, for each fundus image, global image features of the complete region of the reference fundus image and local image features of each sub-region of the fundus image according to the image encoder; The third acquisition unit is used to acquire, for each diagnostic report, global text features of the diagnostic report and local text features of each sentence in the diagnostic report based on the text encoder. The training unit is used to train a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features.

[0019] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a training method for a multimodal ophthalmic basic model of cross-scale contrastive learning as described above.

[0020] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for a multimodal ophthalmic basic model of cross-scale contrastive learning as described above.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a multimodal ophthalmic basic model of cross-scale contrastive learning as described above.

[0022] This application provides a multimodal ophthalmology foundation model and training method based on cross-scale contrastive learning. The multimodal ophthalmology foundation model includes an image encoder and a text encoder. The image encoder is used to acquire global image features for a complete region of a fundus image and local image features for each sub-region of the fundus image. The text encoder is used to acquire global text features for a diagnostic report corresponding to the fundus image and local text features for each sentence in the diagnostic report. The multimodal ophthalmology foundation model is trained based on the local image features, the local text features, the global text features, and the local text features. By fusing fundus images and medical text, and combining cross-modal contrastive learning with multi-granular semantic training, the ophthalmology model achieves accurate alignment of image and text features and knowledge enhancement, improving the accuracy of output results for eye diseases. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of the multimodal ophthalmological basic model for cross-scale contrastive learning provided in this application.

[0025] Figure 2 This is a schematic diagram of the encoding provided in this application.

[0026] Figure 3 This is a flowchart illustrating the training method for the multimodal ophthalmic basic model based on cross-scale contrastive learning provided in this application.

[0027] Figure 4 This is a block diagram of the functional units of the training device for the multimodal ophthalmic basic model of cross-scale contrastive learning provided in this application.

[0028] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] Most existing medical image-text comparison learning models adopt a coarse-grained approach of globally aligning images with complete text, which makes it difficult to capture the complex semantic mapping relationship between images and text.

[0033] To address the aforementioned issues, this application provides a multimodal ophthalmological basic model, training method, and apparatus for cross-scale contrastive learning. The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0034] Please see Figure 1 This multimodal ophthalmology foundational model for cross-scale contrastive learning includes an image encoder and a text encoder.

[0035] For example Figure 1 As shown, the paired image includes a sub-region of a reference fundus image and the corresponding statement: "Optic disc margin is clear and normal in color, disc rim is slightly narrow, no abnormal neovascularization is seen, RNFL appears slightly thin." During the encoder training phase, i.e., when training the multimodal ophthalmology basic model, global and local text features can be obtained through a text encoder, and global and local image features can be obtained through an image encoder. Then, the multimodal ophthalmology basic model is trained using these global, local, global, and local image features. This enables the multimodal ophthalmology basic model to handle downstream tasks, including but not limited to... Figure 1 The image and text search, report generation, and doctor-patient dialogue functions shown are illustrated.

[0036] The training data used to train the multimodal ophthalmology basic model can be fundus images and corresponding diagnostic reports obtained from hospital case databases or constructed knowledge bases. The global image features are image features obtained from a reference fundus image, while the local image features are image features obtained from each sub-region of the reference fundus image. The global text features are text features obtained from the entire diagnostic report, while the local text features are text features obtained from specific sentences within the diagnostic report. This trained ophthalmology model can be used not only for generating diagnostic reports but also for tasks such as image-text retrieval and doctor-patient dialogue.

[0037] This solution achieves precise alignment of image and text features and knowledge enhancement by fusing fundus images with medical text and combining cross-modal contrastive learning with multi-granular semantic training in an ophthalmic model, thereby improving the accuracy of output results for eye diseases.

[0038] In one possible embodiment, the multimodal ophthalmology basic model is used to: acquire a target fundus image and a target description text corresponding to the target fundus image; and, based on the target fundus image and the target description text, output a diagnostic report for an eye disease, trained using global and local text features of a diagnostic report corresponding to a reference fundus image.

[0039] The target fundus image is the fundus image from which the fundus disease to be diagnosed is determined. The target description text may include medical records or preliminary diagnostic text. The diagnostic report may include possible fundus diseases determined based on the input content, or the probability of each fundus disease, the basis for the judgment, etc. This ophthalmological model is trained based on the local image features and global image features of the reference fundus image in the training data, and the global text features and local text features of the diagnostic report corresponding to the reference fundus image.

[0040] As can be seen, in this embodiment, by fusing fundus images and medical text, and combining an ophthalmological model that integrates cross-modal contrastive learning and multi-granular semantic training, accurate alignment of image and text features and knowledge enhancement are achieved, thereby improving the accuracy of diagnostic reports generated for eye diseases.

[0041] In one possible embodiment, the multimodal ophthalmology basic model includes an image-text feature alignment module, which is used to: determine a global loss function based on the global image features and the global text features; determine a local loss function based on the local image features and the local text features; and determine a total loss function based on the global loss function and the local loss function, wherein the multimodal ophthalmology basic model is trained based on the total loss function.

[0042] Each training dataset includes a data pair consisting of a fundus image used for training and a corresponding diagnostic report. Specifically, at least one sentence in the diagnostic report corresponding to each sub-region of the fundus image can be determined based on image recognition or image comparison. For a given paired image and text, global and local information is extracted from each modality using image and text encoders. The image encoder extracts global and sub-region features of the image, while the text encoder extracts the semantics of the complete report and individual sentences. Training the multimodal ophthalmology base model includes image-text feature alignment, which involves global image-text feature alignment to obtain a global loss and local image-text feature alignment to obtain a local loss.

[0043] For example Figure 3 As shown, the text encoder includes a global text encoder and a local text encoder, and the image encoder includes a local image encoder and a global image encoder. The global text encoder is used to obtain global text features, the local text encoder is used to obtain local text features, the local image encoder is used to obtain local image features, and the global image encoder is used to obtain global image features. Then, a global loss function is obtained based on the global features, and a local loss function is obtained based on the local features. The multimodal ophthalmology basic model is trained together based on the global loss function and the local loss function. Then, local feature alignment is performed based on the obtained local image features and local text features, and global feature alignment is performed based on the global image features and global text features.

[0044] As can be seen, in the embodiments of this application, a contrastive loss function oriented towards equivalent expressions is designed based on global and local features, and a joint report-level (global) and sentence-level (local) alignment strategy is adopted to optimize the model's ability to perceive complex semantics through global and local losses.

[0045] In one possible embodiment, in determining the global loss function based on the global image features and the global text features, the image-text feature alignment module is configured to: obtain a text label for each diagnostic report and an image label for each fundus image; determine a set of diagnostic reports corresponding to each global image feature based on the text labels, wherein the diagnostic reports included in the set of diagnostic reports have the same text labels; determine a set of fundus images corresponding to each global text feature based on the image labels, wherein the fundus images included in the set of fundus images have the same image labels; and determine a global loss function based on each global image feature, each global text feature, the set of diagnostic reports, and the set of fundus images.

[0046] Text tags are used to identify core diagnostic information recorded in the diagnostic report, such as the type and severity of fundus lesions. Image tags are used to identify visual information such as the characteristics and regions of fundus lesions presented in fundus images. Text and image tags can be based on a pre-defined fundus diagnostic classification system, or they can be automatically generated after keyword extraction from the diagnostic report text and preliminary feature recognition of the fundus images. This ensures that each diagnostic report and each fundus image corresponds to unique tag information, and that the text and image tags for the same lesion type remain consistent to facilitate subsequent set matching.

[0047] In one possible embodiment, in determining the global loss function based on each of the global image features, each of the global text features, the set of diagnostic reports, and the set of fundus images, the image-text feature alignment module is configured to: for each of the global image features, obtain a first cosine similarity between the global image feature and each of the global text features, and obtain a second cosine similarity between the global text features corresponding to each of the diagnostic reports in the set of diagnostic reports corresponding to the global image feature and the global image feature; for each of the global text features, obtain a third cosine similarity between the global text feature and each of the global image features, and obtain a fourth cosine similarity between the global image features corresponding to each of the fundus images in the set of fundus images corresponding to the global text feature and the global text feature; and determine the global loss function based on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity.

[0048] In the process of global image-text feature alignment, the fundus image and the diagnostic report can be encoded as a whole, enabling the model to recognize semantically equivalent image-text pairs under various expression methods. The resulting global loss function L global This can be expressed as a mathematical formula: Among them, s ij P(i) is the cosine similarity between global image features and global text features, where P(i) is the set of diagnostic reports with the same text labels as the fundus images matched with the fundus images corresponding to the global image features. Q(j) is the set of fundus images with the same image labels as the fundus images matched with the diagnostic reports corresponding to the global text features. In the middle, the obtained s ij For the second cosine similarity, in In the middle, the obtained s ij The fourth cosine similarity. s ik For global image features I i With each global text feature T jThe cosine similarity, also known as the first cosine similarity. kj For global text features T j With each global image feature I i The cosine similarity, also known as the third cosine similarity. Here, N is the temperature coefficient, N is the number of image samples, and M is the number of text samples. N and M are equal.

[0049] As can be seen, in this embodiment, introducing P(i) and Q(j) can maximize the similarity between the image and its paired text and other semantically similar texts, while maintaining a significant distinction from dissimilar texts. By determining the set of similar image-text pairs based on the text labels of the diagnostic report and the image labels of the reference fundus image, and then calculating the cosine similarity to determine the global loss function, the model's ability to align image-text semantics at the global level can be enhanced, improving the accuracy of recognizing semantically equivalent image-text pairs under various expressions, and providing a more reliable global feature foundation for subsequent tasks such as diagnostic report generation.

[0050] In one possible embodiment, when determining the local loss function based on the local image features and the local text features, the image-text feature alignment module is configured to: determine the set of sentences corresponding to the local image features, wherein the local text features corresponding to each sentence in the set of sentences belong to the same category; determine the set of image sub-regions corresponding to the local text features, wherein the local image features corresponding to each image sub-region in the set of image sub-regions belong to the same category; and determine the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions.

[0051] In this set of statements, the local text features corresponding to each statement belong to the same category. That is, the diagnostic information represented by these statements has the same or similar attributes. For example, they may all be statements describing the lesions in a specific part of the fundus. Their corresponding local text features are all based on the lesion features of that specific part, and therefore belong to the same category of local text features. By grouping such statements into a set of statements, the aggregation of local text features of the same category can be achieved, which is convenient for subsequent targeted alignment with the corresponding local image features.

[0052] In one possible embodiment, regarding the determination of the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions, the image-text feature alignment module is configured to: for each local image feature, obtain a fifth cosine similarity between the local image feature and each local text feature, and obtain a sixth cosine similarity between the local text feature corresponding to each sentence in the set of sentences corresponding to the local image feature and the local image feature; for each local text feature, obtain a seventh cosine similarity between the local text feature and each local image feature, and obtain an eighth cosine similarity between the local image feature corresponding to each image sub-region in the set of image sub-regions corresponding to the local text feature and the local text feature; and determine the local loss function based on the fifth cosine similarity, the sixth cosine similarity, the seventh cosine similarity, and the eighth cosine similarity.

[0053] In the local image-text feature alignment stage, to address the differentiated expressions and potential semantic redundancy within the diagnostic report's sentences, sentence-level segmentation can be performed on the text. The loss function calculation in this stage is similar to that in the global image-text feature alignment stage, also considering whether the image and text belong to the same category. It aims to maximize the similarity between the image and its matching text and its similar semantic text, while minimizing the similarity between the image and non-matching text. This local loss function... This can be expressed as a mathematical formula: Among them, h ij It is the cosine similarity between local image features and global text features, P loacl (i) is the set of statements belonging to the same class as the image sub-regions matched with the local image features. Q(j) is the set of image sub-regions belonging to the same class as the image sub-regions matched with the local text features. P loacl In (i), the obtained h ij For the sixth cosine similarity, in Q loacl In (j), the obtained h ij The eighth cosine similarity. ik h represents the cosine similarity between local image features and each local text feature, i.e., the fourth cosine similarity. kj The cosine similarity between local text features and each local image feature is called the seventh cosine similarity. Here, N represents the temperature coefficient, M represents the number of image samples, and M represents the number of text samples. Since there are multiple sentences in the report, the number of sentences is much greater than the number of images.

[0054] In specific implementations, when determining the set of statements corresponding to local image features and / or the set of image sub-regions corresponding to the local text features, clustering can be used to determine the set of statements and the set of image sub-regions. This clustering method includes structured clustering and unstructured clustering.

[0055] As can be seen, in this embodiment, introducing Ploacl(i) and Qloacl(j) can maximize the similarity between the image and its paired text and other semantically similar texts, while maintaining a significant distinction from dissimilar texts. By performing sentence-level segmentation on the diagnostic report text, determining the local image-text similarity set, and calculating multiple sets of cosine similarities to construct a local loss function, and by maximizing the similarity of similar images and texts and minimizing the similarity of dissimilar images and texts, the model's ability to finely align the semantics of images and texts at the local level can be enhanced, alleviating the alignment bias caused by the differentiated expressions and semantic redundancy of the diagnostic report statements.

[0056] In one possible embodiment, in determining the set of statements corresponding to the local image features, the image-text feature alignment module is configured to: determine the reference statements corresponding to the local text features; perform clustering processing on the reference statements to obtain clustering results; and determine the set of statements corresponding to the local image features based on the reference statements and the clustering results.

[0057] Clustering methods can be divided into two types. Unstructured clustering directly uses sentence vector encoding for unsupervised clustering. Structured clustering, on the other hand, sets semantic centers based on clinical priors (such as "leopard-spot fundus", "no abnormalities seen in the fundus", "hemorrhage", "exudation", etc.) and maps the original text into multi-dimensional... Label vectors enable explicit modeling of fine-grained disease entities. By constructing multi-scale semantic anchors, this strategy mitigates the fluctuations in image-text alignment caused by the diversity of textual expressions, and improves the model's robustness to semantically similar but differently expressed samples.

[0058] In one possible embodiment, regarding the clustering of the reference statements to obtain clustering results, the image-text feature alignment module is used to: encode each statement in each diagnostic report to obtain reference features; and cluster each statement using the K-means clustering algorithm based on the reference features to obtain clustering results.

[0059] When determining the sentence set using unstructured clustering, text semantic clustering can be performed based on Simple Contrastive Learning of Sentence Embeddings (SimCSE) and the K-Means Clustering Algorithm (KMeans). The specific clustering process can be represented as follows: Among them, t n It is an example text in the diagnostic report. It was through The translated text is then encoded using SimCSE, and subsequently clustered using KMeans to obtain multiple cluster centers C. k Then, other statements belonging to the same cluster center as the current statement can be identified as statements in that statement set. Specifically, text translation is performed to improve the understanding of the text by the currently used encoder; if a different encoder with better support for the language used in the current text were used, translation would not be necessary.

[0060] As can be seen, in this embodiment, by determining the reference sentences corresponding to local text features, encoding the sentences of each diagnostic report to obtain reference features, and then using K-means clustering, and combining the reference sentences with the clustering results to determine the set of sentences, the ability to accurately cluster the semantics of diagnostic report sentences can be enhanced, thereby improving the matching accuracy of the set of sentences corresponding to local image features and alleviating the fluctuations in local image-text alignment caused by the diversity of medical text expression.

[0061] In one possible embodiment, the image-text feature alignment module is used to: obtain multiple fundus structure types included in the diagnostic report, and a description of each fundus structure type in each diagnostic report; determine a reference fundus structure type included in the statement corresponding to the local text feature, and a reference description of the reference fundus structure type; and determine the set of statements corresponding to the local image feature based on the description of each fundus structure type and the reference description of the reference fundus structure type.

[0062] The reference fundus structure type can include fundus structures such as the fundus, optic disc, macula, and nerve fiber layer. The reference description of the reference fundus structure type can include descriptions such as "leopard-spot fundus" or "no abnormalities seen in the fundus."

[0063] In one possible embodiment, in determining the set of statements corresponding to the local image features based on the description of each fundus structure type and a reference description of the reference fundus structure type, the image-text feature alignment module is configured to: determine a first vector corresponding to each statement in each diagnostic report based on the description of each fundus structure type; determine a second vector of statements corresponding to the local text features based on the reference description of the reference fundus structure type; and determine the set of statements corresponding to the local image features based on the first vector and the second vector.

[0064] In structured clustering, key feature extraction based on structured rules is primarily employed. Since fundus reports exhibit similar structures, region-specific keyword dictionary matching can be implemented, transforming the data into structured feature vectors. This can be expressed mathematically as: Where F is a vector representation of a report; f i,j The value of is either 0 or 1, depending on whether it contains the corresponding structured text. If it contains the structured text corresponding to a certain structure, the value is 1; otherwise, it is 0. `i` represents the structure of the report, including fundus structures such as the optic disc, macula, and nerve fiber layer, with a total of `n`. `j` is the specific description corresponding to the structure, such as descriptions like "leopard-spot fundus" or "no abnormalities found in the fundus," with a total of `m`. The length of vector F is... .

[0065] When determining the set of image sub-regions, the first vector and the second vector can be determined based on the above method, and then the statement corresponding to the first identical vector that is the same as the second vector is determined as the statement in the set of image sub-regions.

[0066] As can be seen, in this embodiment, the original text in the diagnostic report is mapped to a multi-dimensional format. Label vectors enable explicit modeling of fine-grained disease entities. By constructing multi-scale semantic anchors, the fluctuations in image-text alignment caused by the diversity of textual expressions can be mitigated, and the robustness of the model to semantically similar but differently expressed samples can be improved.

[0067] Please see Figure 3 This application also provides a method for training a multimodal ophthalmic basic model using cross-scale contrastive learning, comprising the following steps: S301, Obtain training data.

[0068] The training data includes multiple fundus images and corresponding diagnostic reports for each image. The training data can be fundus images and corresponding diagnostic reports obtained from a hospital case database or a constructed knowledge base. The global image features are image features obtained from a reference fundus image, while the local image features are image features obtained from each sub-region of the reference fundus image. The global text features are text features obtained from the entire diagnostic report, while the local text features are text features obtained from specific sentences within the diagnostic report. This trained ophthalmology model can be used not only for generating diagnostic reports but also for tasks such as image-text retrieval and doctor-patient dialogue.

[0069] S302, for each fundus image, obtain global image features of the complete region of the reference fundus image and local image features of each image sub-region of the fundus image according to the image encoder.

[0070] S303, for each diagnostic report, obtain global text features for the diagnostic report and local text features for each statement in the diagnostic report according to the text encoder.

[0071] S304, Train a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features.

[0072] In one possible embodiment, training a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features includes: determining a global loss function based on the global image features and the global text features; determining a local loss function based on the local image features and the local text features; determining a total loss function based on the global loss function and the local loss functions, wherein the multimodal ophthalmology basic model is trained based on the total loss function.

[0073] In one possible embodiment, determining the global loss function based on the global image features and the global text features includes: obtaining a text label for each diagnostic report and an image label for each fundus image; determining a set of diagnostic reports corresponding to each global image feature based on the text labels, wherein the diagnostic reports in the set of diagnostic reports have the same text labels; determining a set of fundus images corresponding to each global text feature based on the image labels, wherein the fundus images in the set of fundus images have the same image labels; and determining a global loss function based on each global image feature, each global text feature, the set of diagnostic reports, and the set of fundus images.

[0074] In one possible embodiment, determining the global loss function based on each of the global image features, each of the global text features, the diagnostic report set, and the fundus image set includes: for each of the global image features, obtaining a first cosine similarity between the global image feature and each of the global text features, and obtaining a second cosine similarity between the global text features corresponding to each diagnostic report in the diagnostic report set corresponding to the global image feature and the global image feature; for each of the global text features, obtaining a third cosine similarity between the global text feature and each of the global image features, and obtaining a fourth cosine similarity between the global image features corresponding to each fundus image in the fundus image set corresponding to the global text feature and the global text feature; and determining the global loss function based on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity.

[0075] In one possible instance, determining the local loss function based on the local image features and the local text features includes: determining a set of statements corresponding to the local image features, wherein the local text features corresponding to each statement in the set of statements belong to the same category; determining a set of image sub-regions corresponding to the local text features, wherein the local image features corresponding to each image sub-region in the set of image sub-regions belong to the same category; and determining the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of statements, and the set of image sub-regions.

[0076] In one possible embodiment, determining the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions includes: for each local image feature, obtaining the fifth cosine similarity between the local image feature and each local text feature, and obtaining the sixth cosine similarity between the local text features corresponding to each sentence in the set of sentences corresponding to the local image feature and the local image feature; for each local text feature, obtaining the seventh cosine similarity between the local text feature and each local image feature, and obtaining the eighth cosine similarity between the local image features corresponding to each image sub-region in the set of image sub-regions corresponding to the local text feature and the local text feature; and determining the local loss function based on the fifth cosine similarity, the sixth cosine similarity, the seventh cosine similarity, and the eighth cosine similarity.

[0077] In one possible embodiment, determining the set of statements corresponding to the local image features includes: determining reference statements corresponding to the local text features; performing clustering processing on the reference statements to obtain clustering results; and determining the set of statements corresponding to the local image features based on the reference statements and the clustering results.

[0078] In one possible embodiment, the step of clustering the reference statements to obtain clustering results includes: encoding each statement in each diagnostic report to obtain reference features; and using the K-means clustering algorithm to cluster each statement based on the reference features to obtain clustering results.

[0079] In one possible embodiment, determining the set of statements corresponding to the local image features includes: obtaining multiple fundus structure types included in the diagnostic report, and a description of each fundus structure type in each diagnostic report; determining a reference fundus structure type included in the statements corresponding to the local text features, and a reference description of the reference fundus structure type; and determining the set of statements corresponding to the local image features based on the description of each fundus structure type and the reference description of the reference fundus structure type.

[0080] In one possible embodiment, determining the set of statements corresponding to the local image features based on the description of each fundus structure type and the reference description of the reference fundus structure type includes: determining a first vector corresponding to each statement in each diagnostic report based on the description of each fundus structure type; determining a second vector of statements corresponding to the local text features based on the reference description of the reference fundus structure type; and determining the set of statements corresponding to the local image features based on the first vector and the second vector.

[0081] As can be seen, this scheme extracts global and local image features from fundus images and global and local text features from diagnostic reports by using image encoders and text encoders, respectively. By combining these multi-scale and multi-dimensional features to train a multimodal ophthalmology basic model, the model can fully learn the global correlation and local detail correspondence between fundus images and diagnostic reports, thereby effectively improving the accuracy and adaptability of the multimodal ophthalmology basic model in tasks such as diagnostic report generation, image and text retrieval, and doctor-patient dialogue.

[0082] The following describes a training device for a multimodal ophthalmic basic model based on cross-scale contrastive learning provided in this application. The training device for the multimodal ophthalmic basic model based on cross-scale contrastive learning described below corresponds to the training method for the multimodal ophthalmic basic model based on cross-scale contrastive learning described above.

[0083] Please see Figure 4The training device 400 for a multimodal ophthalmology basic model based on cross-scale contrastive learning includes: a first acquisition unit 401, used to acquire training data, the training data including multiple fundus images and diagnostic reports corresponding to each fundus image; a second acquisition unit 402, used to acquire, for each fundus image, global image features of a complete region of the reference fundus image and local image features of each sub-region of the fundus image according to an image encoder; a third acquisition unit 403, used to acquire, for each diagnostic report, global text features of the diagnostic report and local text features of each sentence in the diagnostic report according to a text encoder; and a training unit 404, used to train the multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features.

[0084] In one possible embodiment, in training a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features, the training unit 404 is specifically configured to: determine a global loss function based on the global image features and the global text features; determine a local loss function based on the local image features and the local text features; and determine a total loss function based on the global loss function and the local loss functions, wherein the multimodal ophthalmology basic model is trained based on the total loss function.

[0085] In one possible embodiment, in determining the global loss function based on the global image features and the global text features, the training unit 404 is specifically configured to: obtain a text label for each diagnostic report and an image label for each fundus image; determine a set of diagnostic reports corresponding to each global image feature based on the text labels, wherein the diagnostic reports included in the set of diagnostic reports have the same text labels; determine a set of fundus images corresponding to each global text feature based on the image labels, wherein the fundus images included in the set of fundus images have the same image labels; and determine a global loss function based on each global image feature, each global text feature, the set of diagnostic reports, and the set of fundus images.

[0086] In one possible embodiment, in determining the global loss function based on each of the global image features, each of the global text features, the set of diagnostic reports, and the set of fundus images, the training unit 404 is specifically configured to: for each of the global image features, obtain a first cosine similarity between the global image feature and each of the global text features, and obtain a second cosine similarity between the global text features corresponding to each of the diagnostic reports in the set of diagnostic reports corresponding to the global image feature and the global image feature; for each of the global text features, obtain a third cosine similarity between the global text feature and each of the global image features, and obtain a fourth cosine similarity between the global image features corresponding to each of the fundus images in the set of fundus images corresponding to the global text feature and the global text feature; and determine the global loss function based on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity.

[0087] In one possible instance, in determining the local loss function based on the local image features and the local text features, the training unit 404 is specifically configured to: determine a set of statements corresponding to the local image features, wherein the local text features corresponding to each statement in the set of statements belong to the same category; determine a set of image sub-regions corresponding to the local text features, wherein the local image features corresponding to each image sub-region in the set of image sub-regions belong to the same category; and determine the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of statements, and the set of image sub-regions.

[0088] In one possible embodiment, in determining the local loss function based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions, the training unit 404 is specifically configured to: for each local image feature, obtain a fifth cosine similarity between the local image feature and each local text feature, and obtain a sixth cosine similarity between the local text features corresponding to each sentence in the set of sentences corresponding to the local image feature and the local image feature; for each local text feature, obtain a seventh cosine similarity between the local text feature and each local image feature, and obtain an eighth cosine similarity between the local image features corresponding to each image sub-region in the set of image sub-regions corresponding to the local text feature and the local text feature; and determine the local loss function based on the fifth cosine similarity, the sixth cosine similarity, the seventh cosine similarity, and the eighth cosine similarity.

[0089] In one possible embodiment, in determining the set of statements corresponding to the local image features, the training unit 404 is specifically configured to: determine the reference statements corresponding to the local text features; perform clustering processing on the reference statements to obtain clustering results; and determine the set of statements corresponding to the local image features based on the reference statements and the clustering results.

[0090] In one possible embodiment, in terms of clustering the reference statements to obtain clustering results, the training unit 404 is specifically used to: encode each statement of each diagnostic report to obtain reference features; and cluster each statement using the K-means clustering algorithm based on the reference features to obtain clustering results.

[0091] In one possible embodiment, in determining the set of statements corresponding to the local image features, the training unit 404 is specifically configured to: acquire multiple fundus structure types included in the diagnostic report, and a description of each fundus structure type in each diagnostic report; determine a reference fundus structure type included in the statement corresponding to the local text features, and a reference description of the reference fundus structure type; and determine the set of statements corresponding to the local image features based on the description of each fundus structure type and the reference description of the reference fundus structure type.

[0092] In one possible embodiment, in determining the set of statements corresponding to the local image features based on the description of each fundus structure type and a reference description of the reference fundus structure type, the training unit 404 is specifically configured to: determine a first vector corresponding to each statement in each diagnostic report based on the description of each fundus structure type; determine a second vector corresponding to the statements of the local text features based on the reference description of the reference fundus structure type; and determine the set of statements corresponding to the local image features based on the first vector and the second vector.

[0093] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. For example... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a training method for a multimodal ophthalmology basic model based on cross-scale contrastive learning. The method includes: acquiring training data, the training data including multiple fundus images and diagnostic reports corresponding to each fundus image; for each fundus image, acquiring global image features for a complete region of the reference fundus image and local image features for each image sub-region of the fundus image according to an image encoder; for each diagnostic report, acquiring global text features for the diagnostic report and local text features for each sentence of the diagnostic report according to a text encoder; and training a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features.

[0094] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a training method for a multimodal ophthalmic basic model of cross-scale contrastive learning provided by the methods described above. The method includes: acquiring training data, the training data including multiple fundus images and diagnostic reports corresponding to each fundus image; for each fundus image, acquiring global image features for a complete region of the reference fundus image and local image features for each image sub-region of the fundus image according to an image encoder; for each diagnostic report, acquiring global text features for the diagnostic report and local text features for each sentence of the diagnostic report according to a text encoder; and training a multimodal ophthalmic basic model based on the local image features, the local text features, the global text features, and the local text features.

[0096] In another aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements a training method for a multimodal ophthalmic basic model of any of the aforementioned cross-scale contrastive learning methods. The method includes: acquiring training data, the training data including multiple fundus images and diagnostic reports corresponding to each fundus image; for each fundus image, acquiring global image features for a complete region of the reference fundus image and local image features for each image sub-region of the fundus image using an image encoder; for each diagnostic report, acquiring global text features for the diagnostic report and local text features for each sentence in the diagnostic report using a text encoder; and training a multimodal ophthalmic basic model based on the local image features, the local text features, the global text features, and the local text features.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal ophthalmological basic model for cross-scale contrastive learning, characterized in that, include: Image encoders and text encoders; The image encoder is used to acquire global image features for the entire region of the fundus image and local image features for each sub-region of the fundus image; The text encoder is used to obtain global text features of the diagnostic report corresponding to the fundus image, and local text features of each sentence in the diagnostic report; The multimodal ophthalmology basic model is trained based on the local image features, the local text features, the global text features, and the local text features.

2. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 1, characterized in that, The multimodal ophthalmic basic model is used for: Obtain the target fundus image and the target description text corresponding to the target fundus image; Based on the target fundus image and the target description text, a diagnostic report for eye diseases is output, which is obtained by training global and local text features of the diagnostic report corresponding to the reference fundus image.

3. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 1, characterized in that, The multimodal ophthalmological basic model includes an image-text feature alignment module, which is used for: A global loss function is determined based on the global image features and the global text features; A local loss function is determined based on the local image features and the local text features; The total loss function is determined based on the global loss function and the local loss function, and the multimodal ophthalmology basic model is trained based on the total loss function.

4. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 3, characterized in that, In determining the global loss function based on the global image features and the global text features, the image-text feature alignment module is used to: Obtain the text label for each of the diagnostic reports and the image label for each of the fundus images; A set of diagnostic reports corresponding to each global image feature is determined based on the text labels, wherein the diagnostic reports included in the set of diagnostic reports have the same text labels; The set of fundus images corresponding to each global text feature is determined based on the image labels, wherein the fundus images included in the set of fundus images have the same image labels; A global loss function is determined based on each of the global image features, each of the global text features, the set of diagnostic reports, and the set of fundus images.

5. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 4, characterized in that, In determining the global loss function based on each of the global image features, each of the global text features, the diagnostic report set, and the fundus image set, the image-text feature alignment module is used to: For each global image feature, the first cosine similarity between the global image feature and each global text feature is obtained, and the second cosine similarity between the global text feature corresponding to each diagnostic report in the diagnostic report set corresponding to the global image feature and the global image feature is obtained; For each global text feature, the third cosine similarity between the global text feature and each global image feature is obtained, and the fourth cosine similarity between the global image feature corresponding to each fundus image in the fundus image set corresponding to the global text feature and the global text feature is obtained. The global loss function is determined based on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity.

6. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 3, characterized in that, In the process of determining a local loss function based on the local image features and the local text features, the image-text feature alignment module is used for: Determine the set of sentences corresponding to the local image features, wherein the local text features corresponding to each sentence in the set of sentences belong to the same category; Determine the set of image sub-regions corresponding to the local text features, wherein the local image features corresponding to each image sub-region in the set of image sub-regions belong to the same category; The local loss function is determined based on the local image features of each fundus image, the local text features of each diagnostic report, the set of sentences, and the set of image sub-regions.

7. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 6, characterized in that, In determining the local loss function based on local image features of each fundus image, local text features of each diagnostic report, the set of sentences, and the set of image sub-regions, the image-text feature alignment module is used to: For each of the local image features, the fifth cosine similarity between the local image feature and each of the local text features is obtained, and the sixth cosine similarity between the local text features corresponding to each statement in the statement set corresponding to the local image feature and the local image feature is obtained. For each of the local text features, the seventh cosine similarity between the local text feature and each of the local image features is obtained, and the eighth cosine similarity between the local image features corresponding to each image sub-region in the set of image sub-regions corresponding to the local text feature and the local text feature is obtained. The local loss function is determined based on the fifth cosine similarity, the sixth cosine similarity, the seventh cosine similarity, and the eighth cosine similarity.

8. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 6, characterized in that, In determining the set of statements corresponding to the local image features, the image-text feature alignment module is used to: Determine the reference statement corresponding to the local text feature; The reference statements are clustered to obtain the clustering results; The set of statements corresponding to the local image features is determined based on the reference statements and the clustering results.

9. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 8, characterized in that, In terms of clustering the reference statements to obtain clustering results, the image-text feature alignment module is used for: Encode each statement in each diagnostic report to obtain reference features; Based on the reference features, the K-means clustering algorithm is used to cluster each statement to obtain the clustering results.

10. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 6, characterized in that, In determining the set of statements corresponding to the local image features, the image-text feature alignment module is used to: Obtain multiple fundus structure types included in the diagnostic report, and a description of each fundus structure type in each diagnostic report; Determine the reference fundus structure type included in the statement corresponding to the local text feature, and the reference description of the reference fundus structure type; Based on the description of each fundus structure type and the reference description of the reference fundus structure type, the set of statements corresponding to the local image features is determined.

11. The multimodal ophthalmological basic model for cross-scale contrastive learning according to claim 10, characterized in that, In determining the set of statements corresponding to the local image features based on the description of each fundus structure type and the reference description of the reference fundus structure type, the image-text feature alignment module is used for: Based on the description of each fundus structure type, determine the first vector corresponding to each statement in each diagnostic report; The second vector of the sentence corresponding to the local text feature is determined based on the reference description of the reference fundus structure type; The set of statements corresponding to the local image features is determined based on the first vector and the second vector.

12. A training method for a multimodal ophthalmic basic model using cross-scale contrastive learning, characterized in that, include: Acquire training data, which includes multiple fundus images and a corresponding diagnostic report for each fundus image; For each fundus image, global image features of the complete region of the reference fundus image and local image features of each sub-region of the fundus image are obtained according to the image encoder. For each diagnostic report, global text features for the diagnostic report and local text features for each sentence in the diagnostic report are obtained according to the text encoder; A multimodal ophthalmology basic model is trained based on the local image features, the local text features, the global text features, and the local text features.

13. A training device for a multimodal ophthalmic basic model based on cross-scale contrastive learning, characterized in that, include: The first acquisition unit is used to acquire training data, which includes multiple fundus images and a diagnostic report corresponding to each fundus image; The second acquisition unit is used to acquire, for each fundus image, global image features of the complete region of the reference fundus image and local image features of each image sub-region of the fundus image according to the image encoder; The third acquisition unit is used to acquire, for each diagnostic report, global text features of the diagnostic report and local text features of each sentence in the diagnostic report based on the text encoder. The training unit is used to train a multimodal ophthalmology basic model based on the local image features, the local text features, the global text features, and the local text features.

Images