Image analysis method, device, computer equipment, storage medium and program product

Abstract

This application relates to an image analysis method, apparatus, computer device, storage medium, and program product. The method includes: training an initial image encoder in an initial eye image analysis model based on partially visible region images from multimodal eye image samples, obtaining an image encoder; determining first feature image samples based on eye image samples of each modality and the image encoder; training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples, obtaining an image analysis sub-model; extracting features from the eye image to be identified based on the image encoder, obtaining a first feature image; and inputting the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified. This method supports multiple modalities of eye images as input to the eye image analysis model, improving the accuracy of the eye image analysis results and overcoming the limitations of traditional eye image analysis models in application.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image analysis method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] Vision disorders not only severely affect patients' visual function but can also trigger a series of complications, reduce quality of life, and even threaten life. Meanwhile, with the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0003] Currently, eye image analysis models are typically used to analyze fixed, single-modality eye images to obtain analysis results. However, the application of such eye image analysis models is limited. Summary of the Invention

[0004] Therefore, it is necessary to provide an image analysis method, apparatus, computer equipment, storage medium, and program product to address the aforementioned technical problems.

[0005] Firstly, this application provides an image analysis method. The method includes:

[0006] Based on the partially visible region images in the multimodal eye image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder;

[0007] Based on the eye image samples of each modality and the image encoder, the first feature image sample is determined;

[0008] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the image analysis sub-model;

[0009] Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0010] The first feature image is input into the image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0011] In one embodiment, the initial image encoder in the initial eye image analysis model is trained based on partially visible region images from multimodal eye image samples to obtain the image encoder, which includes:

[0012] Map a portion of the visible region image from the eye image samples of each modality to the corresponding feature vector;

[0013] Each feature vector is input into the initial image encoder to obtain the corresponding second feature image sample;

[0014] Each of the second feature image samples is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each of the second feature image samples;

[0015] Based on the eye image samples and corresponding reconstructed image samples of each modality, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

[0016] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0017] In one embodiment, if the initial image analysis sub-model includes the initial large language model, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model includes:

[0018] The initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the image analysis sub-model.

[0019] In one embodiment, if the initial image analysis sub-model includes the initial classifier, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model includes:

[0020] The initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0021] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0022] Secondly, this application also provides an image analysis apparatus. The apparatus includes:

[0023] The first training module is used to train the initial image encoder in the initial eye image analysis model based on the partially visible region images in the multimodal eye image samples, so as to obtain the image encoder.

[0024] The first determining module is used to determine the first feature image sample based on the eye image sample of each modality and the image encoder;

[0025] The second training module is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, so as to obtain the image analysis sub-model.

[0026] The feature extraction module is used to extract features from the eye image to be identified based on the image encoder to obtain a first feature image;

[0027] The second determining module is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0028] Thirdly, this application also provides a computer device, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the above methods.

[0029] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0030] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0031] The aforementioned image analysis method, apparatus, computer equipment, storage medium, and program product train an initial image encoder in an initial eye image analysis model based on partially visible region images from multimodal eye image samples, obtaining an image encoder. Based on the eye image samples of each modality and the image encoder, a first feature image sample is determined. An initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample, obtaining an image analysis sub-model. Features are extracted from the eye image to be identified based on the image encoder to obtain a first feature image. The first feature image is input into the image analysis sub-model to obtain the analysis result of the eye image to be identified. In this embodiment, the eye image analysis model can support multiple modalities of eye images as input, improving the accuracy of the eye image analysis results and overcoming the limitation of traditional eye image analysis models. Attached Figure Description

[0032] Figure 1 This is an internal structural diagram of a computer device provided in an embodiment of this application;

[0033] Figure 2 This is a schematic flowchart of an image analysis method provided in an embodiment of this application;

[0034] Figure 3 This is a schematic flowchart of an image encoder training method provided in an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of an image encoder training method provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of an image analysis sub-model training method provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram of another image analysis sub-model training method provided in an embodiment of this application;

[0038] Figure 7 This is a flowchart illustrating another image analysis method provided in an embodiment of this application;

[0039] Figure 8 This is a structural block diagram of an image analysis device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] With the aging population and changing lifestyles, the incidence of vision diseases is on the rise globally. These diseases not only severely affect patients' visual function but can also trigger a series of complications, reduce quality of life, and even threaten lives. Meanwhile, with the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0042] Currently, eye image analysis models are used to analyze fixed single-modality eye images to obtain analysis results. However, the application of this eye image analysis model has limitations.

[0043] The image analysis method provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown. Figure 1 This is an internal structure diagram of a computer device provided in an embodiment of this application. The computer device may be a server, and its internal structure diagram may be as follows: Figure 1 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image analysis method.

[0044] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0045] In one embodiment, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating an image analysis method provided in an embodiment of this application. This method can be applied to... Figure 1 The method, using a computer device, includes the following steps:

[0046] S201, Based on the partially visible region images in the multimodal eye image samples, train the initial image encoder in the initial eye image analysis model to obtain the image encoder.

[0047] Optionally, the multimodal eye image samples may include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0048] In this embodiment, multimodal historical eye images can be acquired, and then quality control is performed on each modality of the historical eye images to exclude images with poor quality, such as artifacts, dark colors, or missing pixels. Then, cubic spline interpolation is used to adjust the image size to 256×256 pixels to obtain multimodal eye image samples. The eye image samples from different modalities can be stored in different folders.

[0049] For example, 85% of the image of the optical coherence tomography (OCT) imaging sample can be covered, and 75% of the images of the fundus imaging sample, fundus angiography imaging sample, ultra-wide-angle fundus imaging sample, and external eye image sample can be covered. The uncovered images are the partially visible areas of the multimodal eye image samples.

[0050] In one embodiment, a portion of the visible region image in the eye image samples of each modality can be mapped to a corresponding feature vector. Each feature vector is input into an initial image encoder to obtain a corresponding second feature image sample. Then, each second feature image sample is input into a corresponding decoder. The decoder reconstructs the image of the occluded region based on each second feature image sample to obtain a corresponding reconstructed image sample. The initial image encoder in the initial eye image analysis model is adjusted based on the eye image samples of each modality and the corresponding reconstructed image samples to obtain an image encoder.

[0051] S202, determine the first feature image sample based on the eye image samples of each modality and the image encoder.

[0052] For example, eye image samples of each modality can be input into an image encoder for feature extraction to obtain the first feature image sample output by the image encoder.

[0053] S203, the initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the image analysis sub-model.

[0054] Optionally, the initial image analysis sub-model may include an initial classifier and / or an initial large language model.

[0055] In one embodiment, if the initial image analysis sub-model includes an initial classifier, the initial classifier can be trained based on the first feature image sample to obtain a target classifier, and the target classifier can be used as the image analysis sub-model.

[0056] Alternatively, if the initial image analysis sub-model includes an initial large language model, the initial large language model can be trained based on the first feature image sample and the diagnostic report sample corresponding to the first feature image sample to obtain the target large language model, and the target large language model can be used as the image analysis sub-model.

[0057] S204, Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image.

[0058] In this embodiment, the eye image to be identified can be input into an image encoder for feature extraction to obtain the first feature image output by the image encoder.

[0059] S205, input the first feature image into the image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0060] Optionally, if the initial image analysis sub-model includes an initial classifier, then the image analysis sub-model includes a target classifier. The first feature image can be input into the target classifier to obtain the analysis result of the eye image to be identified, output by the target classifier. The analysis result of the eye image to be identified, output by the target classifier, includes the probability that the patient corresponding to the eye image to be identified suffers from various diseases.

[0061] Alternatively, if the initial image analysis sub-model includes an initial large language model, then the image analysis sub-model includes a target large language model. The first feature image can be input into the target large language model to obtain the analysis result of the eye image to be identified, output by the target large language model. The analysis result of the eye image to be identified, output by the target large language model, includes the diagnostic report corresponding to the eye image to be identified.

[0062] In this embodiment, an initial image encoder is trained in the initial eye image analysis model based on partially visible region images from multimodal eye image samples, resulting in an image encoder. First feature image samples are determined based on the eye image samples from each modality and the image encoder. An initial image analysis sub-model is then trained based on the first feature image samples, resulting in an image analysis sub-model. Features are extracted from the eye image to be identified based on the image encoder, resulting in a first feature image. This first feature image is then input into the image analysis sub-model to obtain the analysis result of the eye image to be identified. In this embodiment, the eye image analysis model can support multiple modalities of eye images as input, improving the accuracy of the eye image analysis results and overcoming the limitation of traditional eye image analysis models in application.

[0063] Reference Figure 3 , Figure 3 This is a flowchart illustrating an image encoder training method provided in an embodiment of this application. This embodiment relates to a possible implementation of training an initial image encoder in an initial eye image analysis model based on partially visible region images from multimodal eye image samples. Based on the above embodiment, S201 includes the following steps:

[0064] S301, map the visible region images in the eye image samples of each modality to the corresponding feature vectors.

[0065] In this embodiment, the eye image samples of each modality can be divided into image blocks of the same size (16×16 pixels), and then a certain proportion of the image blocks can be randomly selected for masking. For example, 85% of the image blocks of the optical coherence tomography (OCT) imaging sample can be masked, and 75% of the image blocks of the fundus imaging sample, fundus angiography imaging sample, ultra-wide-angle fundus imaging sample, and external eye image sample can be masked.

[0066] For example, refer to Figure 4 , Figure 4 This is a schematic diagram of an image encoder training method provided in an embodiment of this application. Figure 4 As shown, the undisturbed visible regions of each modality can be mapped to corresponding feature vectors through an embedding mapping layer.

[0067] S302, input each feature vector into the initial image encoder to obtain the corresponding second feature image sample.

[0068] In one embodiment, the feature vectors of each modality can be arranged together and used as input to the initial image encoder to obtain the high-dimensional feature representations corresponding to the feature vectors of each modality output by the initial image encoder, i.e., the second feature image samples.

[0069] Optionally, the initial image encoder can be the Vision Transformer Large version, which consists of 24 Transformer blocks.

[0070] S303, input each second feature image sample into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0071] In one embodiment, the decoder can receive the second feature image sample of the corresponding modality output by the initial image encoder and the occluded image block as input, thereby reconstructing the pixel value of the occluded image block and obtaining the reconstructed image sample corresponding to each second feature image sample.

[0072] Optionally, each decoder may contain two Transformer blocks.

[0073] S304. Based on eye image samples of each modality and the corresponding reconstructed image samples, train the initial image encoder in the initial eye image analysis model to obtain the image encoder.

[0074] Optionally, the pixel values ​​of each image block can be normalized based on the eye image samples of each modality and the corresponding reconstructed image samples. Then, the loss between the pixel values ​​of the normalized eye image samples and the pixel values ​​of the corresponding reconstructed image samples can be calculated. The initial image encoder can be adjusted based on the calculated loss to obtain the image encoder.

[0075] In this embodiment, a portion of the visible region image in the eye image samples of each modality is mapped to a corresponding feature vector. Each feature vector is input into the initial image encoder to obtain the corresponding second feature image sample. Each second feature image sample is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample. Based on the eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder. This enables the initial image encoder to be trained based on unlabeled multimodal eye image samples, allowing the image encoder to learn shared feature representations across modalities, thereby better capturing the correlation and specificity between different modalities and improving the accuracy of the eye image analysis results.

[0076] Based on the above embodiments, if the initial image analysis sub-model includes an initial large language model, then S203 above can be implemented in the following way:

[0077] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample, and an image analysis sub-model is obtained.

[0078] Alternatively, the initial large language model can be, for example, an LLaMA large language model.

[0079] For example, refer to Figure 5 , Figure 5 This is a schematic diagram of an image analysis sub-model training method provided in an embodiment of this application. For example... Figure 5 As shown, eye image samples of each modality can be input into the image encoder to obtain the first feature image sample of the output. Then, the diagnostic report sample corresponding to each eye image sample (i.e. the diagnostic report sample corresponding to the first feature image sample) can be obtained. Based on the first feature image sample and the corresponding diagnostic report sample, the initial large language model is pre-trained jointly for vision and language to obtain the target large language model. The target large language model can be used as an image analysis sub-model.

[0080] In one embodiment, during the pre-training phase of the initial large language model, the training period can be set to 5 epochs, and the batch size to 16. The initial learning rate is 2×10⁻⁶. -3 The learning rate is adjusted during training using a cosine decay adjustment strategy. In the instruction fine-tuning phase, training also lasts for 5 epochs with a batch size of 8. The initial learning rate for this phase is 2×10⁻⁶. -5 Similarly, the learning rate is adjusted using a cosine decay strategy.

[0081] In this embodiment, an initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain an image analysis sub-model. This sub-model can learn the complex relationship between the eye image and the diagnostic report text, enabling the trained image analysis sub-model to generate a corresponding diagnostic report or visual question-and-answer result (answering diagnostic-related questions based on the input image) based on the eye image to be identified. This provides doctors with more reliable auxiliary diagnostic suggestions.

[0082] Based on the above embodiments, if the initial image analysis sub-model includes an initial large language model, then S203 above can be implemented in the following way:

[0083] An initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0084] Alternatively, the initial classifier can be, for example, a multilayer perceptron classifier.

[0085] For example, refer to Figure 6 , Figure 6 This is a schematic diagram of another image analysis sub-model training method provided in an embodiment of this application. For example... Figure 6As shown, eye image samples of each modality can be input into the image encoder to obtain the first feature image sample of the output. Then, the first feature image sample is adjusted into an image block of 224×224 pixels and input into the initial classifier to train the initial classifier to obtain the target classifier, and the target classifier is used as the image analysis sub-model.

[0086] In one embodiment, the initial classifier undergoes a total training cycle of 50 rounds (50 epochs) with a batch size of 16. During the first 10 epochs, the learning rate gradually increases from 0 to 5 × 10⁻⁶. -4 Subsequently, a cosine decay strategy was used to reduce the learning rate from 5×10⁻⁶ over the next 40 epochs. -4 Gradually reduce to 1×10 -6 After each epoch of training, the classifier is evaluated on the validation set to obtain the area under the receiver operating characteristic (AUC) curve. After training, the model weights obtained from the epoch with the highest AUC are saved as the final training result to obtain the target classifier, and the result of the target classifier is evaluated on the test set.

[0087] In this embodiment, an initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model. This sub-model can better extract and fuse disease features from different modalities of eye images, thereby improving the accuracy of the analysis results.

[0088] Based on the above embodiments, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0089] For example, fundus imaging (CFP) is a non-contact imaging technique using various clinically common fundus cameras, featuring autofocus and autoexposure, and non-mydriatic 45-degree angle color fundus photographs. Fundus imaging samples are obtained by cropping the fundus image to remove the black background, using cubic spline interpolation to resize the image to 256×256 pixels, and discarding images of poor quality.

[0090] Optical coherence tomography (OCT) is a non-mydriatic OCT examination performed using a clinically common OCT instrument, extracting 128 b-scan sequence images. The OCT sample is obtained by adjusting the size of the 128 b-scan sequence images to 256×256 pixels using cubic spline interpolation.

[0091] Fundus angiography (UWF) is a non-mydriatic, 200-degree high-resolution ultra-wide-angle fundus color image captured by clinical ophthalmologists and professional technicians during routine ophthalmological examinations using a clinically common ultra-wide-angle fundus camera. Fundus angiography samples are obtained by adjusting the size of the ultra-wide-angle fundus color image to 256×256 pixels using cubic spline interpolation.

[0092] Ultra-wide-angle fundus imaging (FFA) is a process where clinical ophthalmologists and technicians collect fundus fluorescence images of patients during routine ophthalmological examinations using a clinically common fluorescein angiography camera. The FFA image is obtained by resizing the acquired fundus fluorescence image to 256×256 pixels using cubic spline interpolation.

[0093] External eye images (EEPs) are photographs of the external eye taken by professional technicians during routine ophthalmological examinations using a standard clinical camera. EEP sample images are obtained by resizing the acquired external eye images to 256×256 pixels using cubic spline interpolation, and by discarding images of poor quality.

[0094] Reference Figure 7 , Figure 7 This is a flowchart illustrating another image analysis method provided in an embodiment of this application. The method includes the following steps:

[0095] S701 maps a portion of the visible region image from the eye image samples of each modality to the corresponding feature vector.

[0096] S702, input each feature vector into the initial image encoder to obtain the corresponding second feature image sample.

[0097] S703, input each second feature image sample into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0098] S704, based on eye image samples of each modality and the corresponding reconstructed image samples, trains the initial image encoder in the initial eye image analysis model to obtain the image encoder.

[0099] S705, determine the first feature image sample based on the eye image samples of each modality and the image encoder.

[0100] S706, train an initial large language model based on the first feature image sample and the corresponding diagnostic report sample to obtain an image analysis sub-model, and / or train an initial classifier based on the first feature image sample to obtain an image analysis sub-model.

[0101] S707 extracts features from the eye image to be recognized based on the image encoder to obtain the first feature image.

[0102] S708, input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0103] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0104] Based on the same inventive concept, this application also provides an image analysis apparatus for implementing the image analysis method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image analysis apparatus embodiments provided below can be found in the limitations of the image analysis method described above, and will not be repeated here.

[0105] In one embodiment, such as Figure 8 As shown, Figure 8 This is a structural block diagram of an image analysis device provided in an embodiment of this application. The device 800 includes:

[0106] The first training module 801 is used to train the initial image encoder in the initial eye image analysis model based on the partially visible region images in the multimodal eye image samples, thereby obtaining the image encoder.

[0107] The first determining module 802 is used to determine the first feature image sample based on the eye image samples of each modality and the image encoder.

[0108] The second training module 803 is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples, and obtain the image analysis sub-model.

[0109] The feature extraction module 804 is used to extract features from the eye image to be identified based on the image encoder to obtain the first feature image.

[0110] The second determining module 805 is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0111] In one embodiment, the first training module 801 includes:

[0112] The mapping unit is used to map a portion of the visible region image in the eye image samples of each modality into a corresponding feature vector.

[0113] The first determining unit is used to input each feature vector into the initial image encoder to obtain the corresponding second feature image sample.

[0114] The second determining unit is used to input each second feature image sample into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample.

[0115] The first training unit is used to train the initial image encoder in the initial eye image analysis model based on eye image samples of each modality and the corresponding reconstructed image samples, thus obtaining the image encoder.

[0116] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0117] In one embodiment, if the initial image analysis sub-model includes an initial large language model, then the second training module 803 includes:

[0118] The second training unit is used to train the initial large language model based on the first feature image sample and the corresponding diagnostic report sample, and obtain the image analysis sub-model.

[0119] In one embodiment, if the initial image analysis sub-model includes an initial classifier, then the second training module 803 includes:

[0120] The third training unit is used to train the initial classifier based on the first feature image samples to obtain the image analysis sub-model.

[0121] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0122] Each module in the aforementioned image analysis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0123] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0124] Based on the partially visible region images in the multimodal eye image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder;

[0125] Based on the eye image samples and image encoder of each modality, the first feature image sample is determined;

[0126] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image samples to obtain the image analysis sub-model;

[0127] Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0128] The first feature image is input into the image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0129] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0130] Map a portion of the visible region image from the eye image samples of each modality to the corresponding feature vector;

[0131] Each feature vector is input into the initial image encoder to obtain the corresponding second feature image sample;

[0132] Each second feature image sample is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample;

[0133] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

[0134] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0135] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0136] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample, and an image analysis sub-model is obtained.

[0137] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0138] An initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0139] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0140] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0141] Based on the partially visible region images in the multimodal eye image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder;

[0142] Based on the eye image samples and image encoder of each modality, the first feature image sample is determined;

[0143] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image samples to obtain the image analysis sub-model;

[0144] Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0145] The first feature image is input into the image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0146] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0147] Map a portion of the visible region image from the eye image samples of each modality to the corresponding feature vector;

[0148] Each feature vector is input into the initial image encoder to obtain the corresponding second feature image sample;

[0149] Each second feature image sample is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample;

[0150] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

[0151] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0152] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0153] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample, and an image analysis sub-model is obtained.

[0154] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0155] An initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0156] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0157] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0158] Based on the partially visible region images in the multimodal eye image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder;

[0159] Based on the eye image samples and image encoder of each modality, the first feature image sample is determined;

[0160] The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image samples to obtain the image analysis sub-model;

[0161] Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain the first feature image;

[0162] The first feature image is input into the image analysis sub-model to obtain the analysis results of the eye image to be identified.

[0163] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0164] Map a portion of the visible region image from the eye image samples of each modality to the corresponding feature vector;

[0165] Each feature vector is input into the initial image encoder to obtain the corresponding second feature image sample;

[0166] Each second feature image sample is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample;

[0167] Based on eye image samples of each modality and corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

[0168] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0169] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0170] An initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample, and an image analysis sub-model is obtained.

[0171] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0172] An initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0173] In one embodiment, the multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image analysis method, characterized in that, The method includes: Based on the partially visible region images in the multimodal eye image samples, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder; The first feature image sample is determined based on the eye image samples of each modality and the image encoder; The initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the image analysis sub-model; Based on the image encoder, feature extraction is performed on the eye image to be identified to obtain a first feature image; The first feature image is input into the image analysis sub-model to obtain the analysis result of the eye image to be identified; The step of training the initial image encoder in the initial eye image analysis model based on partially visible region images from multimodal eye image samples to obtain the image encoder includes: Map a portion of the visible region image from the eye image samples of each modality to a corresponding feature vector; Each of the aforementioned feature vectors is input into the initial image encoder to obtain the corresponding second feature image sample; Each second feature image sample is input into the corresponding decoder to obtain the reconstructed image sample corresponding to each second feature image sample; Based on the eye image samples and corresponding reconstructed image samples of each modality, the initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

2. The method according to claim 1, characterized in that, The initial image analysis sub-model includes an initial classifier and / or an initial large language model.

3. The method according to claim 2, characterized in that, If the initial image analysis sub-model includes the initial large language model, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples to obtain the image analysis sub-model includes: The initial large language model is trained based on the first feature image sample and the corresponding diagnostic report sample to obtain the image analysis sub-model.

4. The method according to claim 2, characterized in that, If the initial image analysis sub-model includes the initial classifier, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image samples to obtain the image analysis sub-model includes: The initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

5. The method according to any one of claims 1-4, characterized in that, The multimodal eye image samples include at least two of the following: fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

6. An image analysis device, characterized in that, The device includes: The first training module is used to train the initial image encoder in the initial eye image analysis model based on the partially visible region images in the multimodal eye image samples, so as to obtain the image encoder. The first determining module is used to determine a first feature image sample based on the eye image samples of each modality and the image encoder; The second training module is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample, so as to obtain the image analysis sub-model. The feature extraction module is used to extract features from the image of the eye to be identified based on the image encoder to obtain a first feature image; The second determining module is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified; The first training module includes: The mapping unit is used to map a portion of the visible region image in the eye image samples of each modality into a corresponding feature vector; The first determining unit is used to input each of the feature vectors into the initial image encoder to obtain the corresponding second feature image sample; The second determining unit is used to input each of the second feature image samples into the corresponding decoder to obtain the reconstructed image samples corresponding to each of the second feature image samples. The first training unit is used to train the initial image encoder in the initial eye image analysis model based on the eye image samples and corresponding reconstructed image samples of each modality, thereby obtaining the image encoder.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images