Image analysis method and device, computer equipment, storage medium and program product

Abstract

The invention relates to an image analysis method and device, computer equipment, a storage medium and a program product. The method comprises the steps of training an initial image encoder in an initial eye image analysis model according to a partial visible region image in a multi-modal eye image sample to obtain an image encoder, determining a first feature image sample according to the eye image sample of each modal and the image encoder, and obtaining a second feature image sample according to the first feature image sample and the second feature image sample; training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model, performing feature extraction on the to-be-recognized eye image based on an image encoder to obtain a first feature image, and inputting the first feature image into the image analysis sub-model to obtain a second feature image; according to the method, the to-be-recognized eye image is analyzed to obtain the analysis result of the to-be-recognized eye image, so that the eye images in various modes can be supported to serve as input of the eye image analysis model, the accuracy of the analysis result of the eye image is improved, and the problem that application of a traditional eye image analysis model is limited is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image analysis method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] Vision diseases not only seriously affect patients' visual function, but may also cause a series of complications, reduce the quality of life, and even threaten life. At the same time, with the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0003] Currently, an eye image analysis model is usually used to analyze a fixed single-modality eye image to obtain an analysis result. However, the application of the eye image analysis model is limited. Summary of the invention

[0004] Based on this, it is necessary to provide an image analysis method, device, computer equipment, storage medium and program product to address the above technical problems.

[0005] In a first aspect, the present application provides an image analysis method. The method comprises:

[0006] According to the partially visible area images in the multimodal eye image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder;

[0007] Determine a first feature image sample according to the eye image samples of each modality and the image encoder;

[0008] Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model;

[0009] Extracting features of the eye image to be identified based on the image encoder to obtain a first feature image;

[0010] The first feature image is input into the image analysis sub-model to obtain an analysis result of the eye image to be identified.

[0011] In one embodiment, the initial image encoder in the initial eye image analysis model is trained based on the partially visible area image in the multimodal eye image sample to obtain the image encoder, including:

[0012] Mapping a portion of the visible area image in each eye image sample of the modality into a corresponding feature vector;

[0013] Input each of the feature vectors into the initial image encoder to obtain a corresponding second feature image sample;

[0014] Input each of the second feature image samples into a corresponding decoder to obtain a reconstructed image sample corresponding to each of the second feature image samples;

[0015] Based on the eye image samples of each modality and the corresponding reconstructed image samples, an initial image encoder in the initial eye image analysis model is trained to obtain the image encoder.

[0016] In one of the embodiments, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0017] In one embodiment, if the initial image analysis sub-model includes the initial large language model, then the initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the image analysis sub-model, including:

[0018] The initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain the image analysis sub-model.

[0019] In one embodiment, if the initial image analysis sub-model includes the initial classifier, then the initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain the image analysis sub-model, including:

[0020] The initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

[0021] In one of the embodiments, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples and external eye image samples.

[0022] In a second aspect, the present application also provides an image analysis device. The device comprises:

[0023] A first training module is used to train an initial image encoder in an initial eye image analysis model according to a partially visible area image in a multimodal eye image sample to obtain an image encoder;

[0024] A first determination module, configured to determine a first feature image sample according to the eye image samples of each modality and the image encoder;

[0025] A second training module is used to train an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model;

[0026] A feature extraction module, used for extracting features of the eye image to be identified based on the image encoder to obtain a first feature image;

[0027] The second determination module is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0028] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0029] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0030] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0031] The above-mentioned image analysis method, apparatus, computer equipment, storage medium and program product train the initial image encoder in the initial eye image analysis model based on the partially visible area image in the multi-modal eye image samples to obtain the image encoder, determine the first feature image sample based on the eye image samples and image encoders of each modality, train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model, extract features of the eye image to be identified based on the image encoder to obtain the first feature image, input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified. In the embodiment of the present application, the eye image analysis model can support eye images of multiple modalities as input, improve the accuracy of the analysis results of the eye image, and overcome the problem of limited application of the traditional eye image analysis model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is an internal structure diagram of a computer device provided in an embodiment of the present application;

[0033] Figure 2 It is a flowchart of an image analysis method provided in an embodiment of the present application;

[0034] Figure 3 It is a flowchart of an image encoder training method provided in an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of an image encoder training method provided in an embodiment of the present application;

[0036] Figure 5 is a schematic diagram of an image analysis sub-model training method provided in an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of another image analysis sub-model training method provided in an embodiment of the present application;

[0038] Figure 7 is a flow chart of another image analysis method provided in an embodiment of the present application;

[0039] Figure 8 It is a structural block diagram of an image analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] With the aging of the population and the change of lifestyle, the incidence of vision diseases is on the rise worldwide. These diseases not only seriously affect the visual function of patients, but also may cause a series of complications, reduce the quality of life, and even threaten life. At the same time, with the advancement of artificial intelligence technology, using eye image analysis models to analyze eye images to assist doctors in diagnosing eye diseases has become an important research direction in this field.

[0042] Currently, an eye image analysis model is used to analyze eye images of a fixed single modality to obtain analysis results. However, the application of the eye image analysis model is limited.

[0043] The image analysis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 is an internal structure diagram of a computer device provided in an embodiment of the present application. The computer device may be a server, and its internal structure diagram may be as follows Figure 1 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image analysis method is implemented.

[0044] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0045] In one embodiment, Figure 2 As shown, Figure 2 is a flow chart of an image analysis method provided in an embodiment of the present application, which can be applied to Figure 1 The computer device comprises the following steps:

[0046] S201, training an initial image encoder in an initial eye image analysis model according to partially visible area images in multimodal eye image samples to obtain an image encoder.

[0047] Optionally, the multimodal eye image samples may include, for example, at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0048] In the embodiment of the present application, multi-modal historical eye images can be obtained, and then the quality control of the historical eye images of each modality is performed to exclude poor quality images, such as artifacts, dark colors, missing pixels, etc. Then, the image size is adjusted to 256×256 pixels using cubic spline interpolation to obtain multi-modal eye image samples. Among them, eye image samples of different modalities can be stored in different folders.

[0049] Exemplarily, 85% of the image of the optical coherence tomography imaging sample can be covered, and 75% of the image of the fundus imaging sample, fundus angiography imaging sample, ultra-wide-angle fundus imaging sample and external eye image sample can be covered. The uncovered image is the image of a partial visible area in the multimodal eye image sample.

[0050] In one embodiment, images of partially visible areas in eye image samples of each modality can be mapped to corresponding feature vectors, and each feature vector can be input into an initial image encoder to obtain a corresponding second feature image sample. Then, each second feature image sample can be input into a corresponding decoder. The decoder reconstructs an image of a covered area based on each second feature image sample to obtain a corresponding reconstructed image sample. Based on the eye image samples of each modality and the corresponding reconstructed image samples, the initial image encoder in the initial eye image analysis model is adjusted to obtain an image encoder.

[0051] S202: Determine a first feature image sample according to the eye image samples of each modality and the image encoder.

[0052] Exemplarily, eye image samples of each modality may be input into an image encoder for feature extraction to obtain first feature image samples output by the image encoder.

[0053] S203: Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model.

[0054] Optionally, the initial image analysis sub-model may include an initial classifier and / or an initial large language model.

[0055] In one embodiment, if the initial image analysis sub-model includes an initial classifier, the initial classifier can be trained based on the first feature image sample to obtain a target classifier, and the target classifier is used as the image analysis sub-model.

[0056] Alternatively, if the initial image analysis sub-model includes an initial large language model, the initial large language model can be trained based on the first feature image sample and the diagnostic report sample corresponding to the first feature image sample to obtain a target large language model, and the target large language model is used as the image analysis sub-model.

[0057] S204: Extract features of the eye image to be recognized based on the image encoder to obtain a first feature image.

[0058] In the embodiment of the present application, the eye image to be identified can be input into an image encoder for feature extraction to obtain a first feature image output by the image encoder.

[0059] S205, inputting the first feature image into the image analysis sub-model to obtain an analysis result of the eye image to be identified.

[0060] Optionally, if the initial image analysis submodel includes an initial classifier, the image analysis submodel includes a target classifier, and the first feature image can be input into the target classifier to obtain the analysis result of the eye image to be identified output by the target classifier. The analysis result of the eye image to be identified output by the target classifier includes the probability of the patient corresponding to the eye image to be identified suffering from various diseases.

[0061] Alternatively, if the initial image analysis sub-model includes an initial large language model, the image analysis sub-model includes a target large language model, and the first feature image can be input into the target large language model to obtain the analysis result of the eye image to be identified output by the target large language model. The analysis result of the eye image to be identified output by the target large language model includes a diagnosis report corresponding to the eye image to be identified.

[0062] In an embodiment of the present application, an initial image encoder in an initial eye image analysis model is trained based on images of partially visible areas in multimodal eye image samples to obtain an image encoder, a first feature image sample is determined based on eye image samples and image encoders of each modality, an initial image analysis sub-model in the initial eye image analysis model is trained based on the first feature image sample to obtain an image analysis sub-model, feature extraction is performed on the eye image to be identified based on the image encoder to obtain a first feature image, the first feature image is input into the image analysis sub-model to obtain an analysis result of the eye image to be identified. In an embodiment of the present application, the eye image analysis model can support eye images of multiple modalities as input, improves the accuracy of the analysis results of the eye image, and overcomes the problem of limited application of traditional eye image analysis models.

[0063] Reference Figure 3 , Figure 3 1 is a flow chart of an image encoder training method provided in an embodiment of the present application. This embodiment involves how to train an initial image encoder in an initial eye image analysis model based on partially visible area images in multimodal eye image samples to obtain a possible implementation of an image encoder. Based on the above embodiment, the above S201 includes the following steps:

[0064] S301, mapping a portion of the visible area image in the eye image samples of each modality into a corresponding feature vector.

[0065] In the embodiment of the present application, the eye image samples of each modality can be divided into image blocks of the same size of 16×16 pixels, and then a certain proportion of the image blocks are randomly selected for masking. For example, the image blocks of the optical coherence tomography imaging samples can be masked by 85%, and the image blocks of the fundus imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples can be masked by 75%.

[0066] For example, refer to Figure 4 , Figure 4 is a schematic diagram of an image encoder training method provided in an embodiment of the present application. Figure 4 As shown, the partially visible area image of each modality that is not covered can be mapped into a corresponding feature vector through an embedding mapping layer.

[0067] S302, input each feature vector into an initial image encoder to obtain a corresponding second feature image sample.

[0068] In one embodiment, the feature vectors of each modality may be arranged together as input to an initial image encoder to obtain a high-dimensional feature representation corresponding to the feature vectors of each modality output by the initial image encoder, that is, a second feature image sample.

[0069] Optionally, the initial image encoder can choose the Vision Transformer Large version, which consists of 24 Transformer blocks.

[0070] S303: Input each second feature image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample.

[0071] In one embodiment, the decoder can receive the second feature image samples of the corresponding modality output by the initial image encoder and the masked image blocks as input, thereby reconstructing the pixel values ​​of the masked image blocks and obtaining reconstructed image samples corresponding to each second feature image sample.

[0072] Optionally, each decoder can contain two Transformer blocks.

[0073] S304: Based on the eye image samples of each modality and the corresponding reconstructed image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder.

[0074] Optionally, the pixel values ​​of each image block can be normalized based on the eye image samples of each modality and the corresponding reconstructed image samples, and then the loss between the pixel values ​​of the normalized eye image samples and the corresponding reconstructed image samples is calculated, and the initial image encoder is adjusted according to the calculated loss to obtain an image encoder.

[0075] In an embodiment of the present application, images of part of the visible area in eye image samples of each modality are mapped into corresponding feature vectors, each feature vector is input into an initial image encoder to obtain a corresponding second feature image sample, each second feature image sample is input into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample, and an initial image encoder in an initial eye image analysis model is trained based on the eye image samples of each modality and the corresponding reconstructed image samples to obtain an image encoder, so that the initial image encoder can be trained based on unlabeled multimodal eye image samples, so that the image encoder can learn shared feature representations across modalities, thereby better capturing the correlation and specificity between different modalities, and thus improving the accuracy of the analysis results of the eye images.

[0076] On the basis of the above embodiment, if the initial image analysis sub-model includes an initial large language model, the above S203 can be implemented in the following manner:

[0077] An initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model.

[0078] Optionally, the initial large language model may be, for example, an LLaMA large language model.

[0079] For example, refer to Figure 5 , Figure 5 Schematic diagram of an image analysis sub-model training method provided in an embodiment of the present application. Figure 5 As shown, eye image samples of each modality can be input into the image encoder to obtain the output first feature image sample, and then the diagnosis report sample corresponding to each eye image sample (i.e., the diagnosis report sample corresponding to the first feature image sample) is obtained, and the initial large language model is pre-trained with joint vision and language based on the first feature image sample and the corresponding diagnosis report sample to obtain the target large language model, and the target large language model can be used as an image analysis sub-model.

[0080] In one embodiment, during the pre-training phase of the initial large language model, the training cycle can be set to 5 epochs and the batch size can be set to 16. The initial learning rate is 2×10 -3 The learning rate is adjusted during the training process through the cosine decay adjustment strategy. In the instruction fine-tuning stage, the training also lasts for 5 epochs with a batch size of 8. The initial learning rate in this stage is 2×10 -5 , the learning rate is also adjusted through the cosine decay strategy.

[0081] In an embodiment of the present application, an initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model, which can learn the complex relationship between the eye image and the diagnosis report text, so that the trained image analysis sub-model can generate a corresponding diagnosis report or visual question and answer result (answer diagnosis-related questions based on the input image) according to the eye image to be identified, thereby providing doctors with more reliable auxiliary diagnosis suggestions.

[0082] On the basis of the above embodiment, if the initial image analysis sub-model includes an initial large language model, the above S203 can be implemented in the following manner:

[0083] An initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model.

[0084] Optionally, the initial classifier may be, for example, a multi-layer perceptron classifier.

[0085] For example, refer to Figure 6 , Figure 6 FIG. 1 is a schematic diagram of another image analysis sub-model training method provided in an embodiment of the present application. Figure 6As shown, eye image samples of each modality can be input into an image encoder to obtain a first feature image sample as an output, and then the first feature image sample can be adjusted to an image block of 224×224 pixels and input into an initial classifier to train the initial classifier to obtain a target classifier, and the target classifier can be used as an image analysis sub-model.

[0086] In one embodiment, the total training cycle of the initial classifier is 50 rounds, that is, 50 epochs, and the batch size is 16. In the first 10 epochs, the learning rate is gradually increased from 0 to 5×10 -4 Then, the learning rate is reduced from 5×10 to 10×10 in the next 40 epochs through the cosine decay strategy. -4 Gradually reduced to 1×10 -6 After each epoch of training, the classifier is evaluated on the validation set to obtain the area under the receiver operating characteristic curve (AUC) of the classifier. After the training is completed, the model weight obtained from the epoch with the highest AUC is saved as the final training result to obtain the target classifier, and the result of the target classifier is evaluated on the test set.

[0087] In an embodiment of the present application, an initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model, which can better extract and fuse disease features of eye images of different modalities based on the image analysis sub-model, thereby improving the accuracy of the analysis results.

[0088] Based on the above embodiments, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples and external eye image samples.

[0089] For example, fundus imaging (CFP) is a non-contact photography technique using a variety of clinically commonly used fundus cameras, with autofocus and auto-exposure for non-mydriatic 45-degree angle fundus color photography. The fundus imaging sample is obtained by cutting out the black background of the fundus image, resizing the image size to 256×256 pixels using cubic spline interpolation, and removing images with poor quality.

[0090] Optical coherence tomography (OCT) is a non-mydriatic OCT examination performed using a commonly used OCT instrument in clinical practice, and 128 b-scan sequence images are extracted. The optical coherence tomography imaging sample is a sample obtained by resizing the 128 b-scan sequence images to a size of 256×256 pixels using cubic spline interpolation.

[0091] Fundus angiography (UWF) is a non-mydriatic 200-degree high-resolution ultra-wide-angle fundus color photograph collected by clinical ophthalmologists and professional technicians during daily ophthalmology diagnosis and treatment using a commonly used ultra-wide-angle fundus camera. Fundus angiography samples are obtained by resizing the ultra-wide-angle fundus color photograph image to 256×256 pixels using cubic spline interpolation.

[0092] Ultra-wide-angle fundus imaging (FFA) is a fluorescent fundus photograph collected by clinical ophthalmologists and professional technicians during daily ophthalmology diagnosis and treatment using a commonly used fluorescent angiography fundus camera. Ultra-wide-angle fundus imaging is a sample obtained by resizing the collected fundus fluorescent photograph to a size of 256×256 pixels using cubic spline interpolation.

[0093] External eye images (EEP) are external eye photos taken by professional technicians during daily ophthalmology diagnosis and treatment using a common camera commonly used in clinical practice. External eye image samples are samples of external eye photos that are resized to 256×256 pixels using cubic spline interpolation and images with poor quality are removed.

[0094] Reference Figure 7 , Figure 7 : is a flow chart of another image analysis method provided in an embodiment of the present application. The method comprises the following steps:

[0095] S701, mapping a portion of the visible area image in the eye image samples of each modality into a corresponding feature vector.

[0096] S702, input each feature vector into an initial image encoder to obtain a corresponding second feature image sample.

[0097] S703: Input each second feature image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample.

[0098] S704, based on the eye image samples of each modality and the corresponding reconstructed image samples, train an initial image encoder in the initial eye image analysis model to obtain an image encoder.

[0099] S705: Determine a first feature image sample according to the eye image samples of each modality and the image encoder.

[0100] S706, training an initial large language model based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model, and / or, training an initial classifier based on the first feature image sample to obtain an image analysis sub-model.

[0101] S707, extracting features of the eye image to be recognized based on the image encoder to obtain a first feature image.

[0102] S708, input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0103] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0104] Based on the same inventive concept, the embodiment of the present application also provides an image analysis device for implementing the above-mentioned image analysis method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image analysis device embodiments provided below can refer to the limitations on the image analysis method above, and will not be repeated here.

[0105] In one embodiment, Figure 8 As shown, Figure 8 800 is a structural block diagram of an image analysis device provided in an embodiment of the present application. The device 800 includes:

[0106] The first training module 801 is used to train an initial image encoder in an initial eye image analysis model according to a partial visible area image in a multimodal eye image sample to obtain an image encoder.

[0107] The first determination module 802 is used to determine the first feature image sample according to the eye image samples of each modality and the image encoder.

[0108] The second training module 803 is used to train the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model.

[0109] The feature extraction module 804 is used to extract features from the eye image to be identified based on the image encoder to obtain a first feature image.

[0110] The second determination module 805 is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0111] In one embodiment, the first training module 801 includes:

[0112] The mapping unit is used to map the partially visible area image in the eye image samples of each modality into a corresponding feature vector.

[0113] The first determining unit is used to input each feature vector into an initial image encoder to obtain a corresponding second feature image sample.

[0114] The second determining unit is used to input each second characteristic image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second characteristic image sample.

[0115] The first training unit is used to train an initial image encoder in an initial eye image analysis model based on eye image samples of each modality and corresponding reconstructed image samples to obtain an image encoder.

[0116] In one of the embodiments, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0117] In one embodiment, if the initial image analysis sub-model includes an initial large language model, the second training module 803 includes:

[0118] The second training unit is used to train an initial large language model based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model.

[0119] In one embodiment, if the initial image analysis sub-model includes an initial classifier, the second training module 803 includes:

[0120] The third training unit is used to train an initial classifier based on the first feature image sample to obtain an image analysis sub-model.

[0121] In one of the embodiments, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0122] Each module in the above-mentioned image analysis device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0123] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0124] According to the partially visible area images in the multimodal eye image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder;

[0125] Determine a first feature image sample according to the eye image samples of each modality and the image encoder;

[0126] Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model;

[0127] Extracting features of the eye image to be identified based on the image encoder to obtain a first feature image;

[0128] The first feature image is input into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0129] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0130] Mapping the partially visible area images in the eye image samples of each modality into corresponding feature vectors;

[0131] Input each feature vector into the initial image encoder to obtain a corresponding second feature image sample;

[0132] Input each second feature image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample;

[0133] Based on the eye image samples of each modality and the corresponding reconstructed image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder.

[0134] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0135] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0136] An initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] An initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model.

[0139] In one embodiment, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0140] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0141] According to the partially visible area images in the multimodal eye image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder;

[0142] Determine a first feature image sample according to the eye image samples of each modality and the image encoder;

[0143] Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model;

[0144] Extracting features of the eye image to be identified based on the image encoder to obtain a first feature image;

[0145] The first feature image is input into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0146] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0147] Mapping the partially visible area images in the eye image samples of each modality into corresponding feature vectors;

[0148] Input each feature vector into the initial image encoder to obtain a corresponding second feature image sample;

[0149] Input each second feature image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample;

[0150] Based on the eye image samples of each modality and the corresponding reconstructed image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder.

[0151] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0152] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0153] An initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0155] An initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model.

[0156] In one embodiment, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0157] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0158] According to the partially visible area images in the multimodal eye image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder;

[0159] Determine a first feature image sample according to the eye image samples of each modality and the image encoder;

[0160] Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model;

[0161] Extracting features of the eye image to be identified based on the image encoder to obtain a first feature image;

[0162] The first feature image is input into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0164] Mapping the partially visible area images in the eye image samples of each modality into corresponding feature vectors;

[0165] Input each feature vector into the initial image encoder to obtain a corresponding second feature image sample;

[0166] Input each second feature image sample into a corresponding decoder to obtain a reconstructed image sample corresponding to each second feature image sample;

[0167] Based on the eye image samples of each modality and the corresponding reconstructed image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder.

[0168] In one embodiment, the initial image analysis sub-model includes an initial classifier and / or an initial large language model.

[0169] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0170] An initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain an image analysis sub-model.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] An initial classifier is trained based on the first feature image sample to obtain an image analysis sub-model.

[0173] In one embodiment, the multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples, and external eye image samples.

[0174] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image analysis method, characterized in that: The method comprises: According to the partially visible area images in the multimodal eye image samples, an initial image encoder in the initial eye image analysis model is trained to obtain an image encoder; Determine a first feature image sample according to the eye image samples of each of the modalities and the image encoder; Training an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model; Extracting features of the eye image to be identified based on the image encoder to obtain a first feature image; The first feature image is input into the image analysis sub-model to obtain an analysis result of the eye image to be identified.

2. The method according to claim 1, characterized in that The method of training an initial image encoder in an initial eye image analysis model according to a partially visible area image in a multimodal eye image sample to obtain an image encoder includes: Mapping a portion of the visible area image in the eye image samples of each modality into a corresponding feature vector; Input each of the feature vectors into the initial image encoder to obtain a corresponding second feature image sample; Inputting each of the second feature image samples into a corresponding decoder to obtain a reconstructed image sample corresponding to each of the second feature image samples; Based on the eye image samples of each of the modalities and the corresponding reconstructed image samples, an initial image encoder in an initial eye image analysis model is trained to obtain the image encoder.

3. The method according to claim 1, characterized in that: The initial image analysis sub-model includes an initial classifier and / or an initial large language model.

4. The method according to claim 3, characterized in that If the initial image analysis sub-model includes the initial large language model, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model includes: The initial large language model is trained based on the first feature image sample and the corresponding diagnosis report sample to obtain the image analysis sub-model.

5. The method according to claim 3, characterized in that: If the initial image analysis sub-model includes the initial classifier, then training the initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain the image analysis sub-model includes: The initial classifier is trained based on the first feature image sample to obtain the image analysis sub-model.

6. The method according to any one of claims 1 to 5, characterized in that: The multimodal eye image samples include at least two of fundus imaging samples, optical coherence tomography imaging samples, fundus angiography imaging samples, ultra-wide-angle fundus imaging samples and external eye image samples.

7. An image analysis device, characterized in that: The device comprises: A first training module is used to train an initial image encoder in an initial eye image analysis model according to a partially visible area image in a multimodal eye image sample to obtain an image encoder; A first determination module, configured to determine a first feature image sample according to the eye image samples of each of the modalities and the image encoder; A second training module is used to train an initial image analysis sub-model in the initial eye image analysis model based on the first feature image sample to obtain an image analysis sub-model; A feature extraction module, configured to extract features of the eye image to be identified based on the image encoder to obtain a first feature image; The second determination module is used to input the first feature image into the image analysis sub-model to obtain the analysis result of the eye image to be identified.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images