Model training method, conversion method, electronic device, medium, and program product

Abstract

The present disclosure relates to the technical field of image processing. Provided are a model training method, a conversion method, an electronic device, a medium, and a program product. The model training method comprises: performing pixel-level alignment processing on a source-modality eye image and a corresponding target-modality eye structure diagnostic video; selecting a key frame from the target-modality eye structure diagnostic video which has been subjected to alignment processing; using, as a first initial domain, the source-modality eye image which has been subjected to alignment processing, using the corresponding target-modality key frame as a first target domain, and performing training to obtain a first model; and using the target-modality key frame as a second initial domain, using, as a second target domain, the target-modality eye structure diagnostic video which has been subjected to alignment processing, and performing training to obtain a second model, so as to obtain an eye dynamic sequence generation model. By means of the technical solution of the present disclosure, a source-modality static ophthalmic image is converted into a target-modality dynamic ophthalmic video; and the present disclosure has the characteristics of non-invasiveness, safety and a low cost.

Description

Model training method, conversion method, electronic device, medium and program product

[0001] This disclosure claims priority to Chinese patent application number 202410360491.4, filed on March 27, 2024, entitled “Model training method, conversion method, electronic device, medium and program product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of image processing technology, and in particular to a method for training an eye dynamic sequence generation model, a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, a device for training an eye dynamic sequence generation model, a device for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0003] Ophthalmic diagnostic technologies used in clinical ophthalmology include fundus fluorescein angiography (FFA), indocyanine green angiography (ICGA), and optical coherence tomography (OCT). FFA requires intravenous injection of contrast agents, which may cause side effects in some patients. ICGA is an invasive diagnostic technology. Although OCT is a non-invasive diagnostic technology that can provide detailed information about the internal structure of the retina, it also has the disadvantages of high equipment and operating costs and the need for professional knowledge.

[0004] Therefore, there is an urgent need for a non-invasive, cost-effective and efficient angiography solution for ophthalmic diagnosis.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0006] The purpose of the present disclosure is to provide a training method for a cross-modal ocular dynamic sequence generation model, a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, a training device, a conversion device, an electronic device, a storage medium and a computer program, so as to at least to some extent overcome the problem that the diagnosis of ocular structure in related technologies is costly or invasive to the human body.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a training method for an eye dynamic sequence generation model is provided, comprising: performing pixel-level alignment processing on a source modality eye image and a corresponding target modality eye structure diagnosis video; selecting a target modality key frame in the target modality eye structure diagnosis video after the alignment processing, wherein the target modality key frame represents the modal characteristics of the target modality eye structure diagnosis video; using the aligned source modality eye image as a first initial domain and the corresponding target modality key frame as a first target domain, and training an image-to-image generation network to obtain a first model; using the target modality key frame as a second initial domain and the aligned target modality eye structure diagnosis video as a second target domain, and training an image-to-video generation network to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain the eye dynamic sequence generation model based on the first model and the second model.

[0009] In one embodiment of the present disclosure, a source modality eye image and a corresponding target modality eye structure diagnostic video are subjected to pixel-level alignment processing, including: for the source module eye image and the target modality eye structure diagnostic video from the same eye in the same examination, a feature segmentation algorithm is used to extract eye inspection features; and pixel-level alignment processing is performed based on the eye inspection features.

[0010] In one embodiment of the present disclosure, pixel-level alignment processing is performed based on the eye inspection features, including: the eye inspection features include retinal blood vessels, pixel-level key points of the retinal blood vessels are detected using a key point detector, and the pixel-level key points are used as an alignment reference to perform feature matching between the source module eye image and the target modality eye structure diagnosis video; matching feature points are estimated based on a random sampling consistency algorithm to estimate the homography matrix between the source module eye image and the retinal blood vessels in the target modality eye structure diagnosis video; abnormal frames in the eye structure diagnosis video are excluded based on the homography matrix to obtain the aligned source module eye image and the target modality eye structure diagnosis video.

[0011] In one embodiment of the present disclosure, the aligned source modality eye image is used as the first initial domain, and the corresponding target modality key frame is used as the first target domain, and the image-to-image generation network is trained to obtain a first model, including: the image-to-image generation network includes a U-shaped structure generator, the lesion supervision loss is used as the loss function, the first initial domain and the first target domain are used to train the U-shaped structure generator, so that the U-shaped structure generator converts the input source modality eye image into a conversion frame similar to the target modality key frame, wherein the U-shaped structure generator includes an encoder and a decoder, the encoder is used to encode the source modality eye image into a representation of a latent space, and the decoder is used to decode the representation of the latent space into the target modality key frame; the conversion frame and the corresponding target modality key frame are used as training data sets, and the perceptual loss and feature matching loss are used as loss functions to continue training the generator to obtain the first model.

[0012] In one embodiment of the present disclosure, the generator includes a global generator network and a local enhancement network, and training the generator also includes: inputting the eye image with a first resolution and the target modality key frame into the local enhancement network for upsampling based on an attention mechanism to output a local enhanced image with a second resolution, wherein the second resolution is N times the first resolution, and N is greater than 1.

[0013] In one embodiment of the present disclosure, the target modality key frame is used as the second initial domain, and the aligned target modality eye structure diagnosis video is used as the second target domain, and the image-to-video generation network is trained to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, including: inputting the target modality key frame into the image-to-video generation network for prediction processing to obtain the first frame of the generated eye dynamic sequence; coupling the first frame and the target modality key frame and inputting the second frame into the image-to-video generation network for prediction processing to obtain the second frame of the generated eye dynamic sequence; predicting the remaining video frames frame by frame based on time sequence until the image-to-video generation network predicts all video frames of the generated eye dynamic sequence; iteratively training the image-to-video generation network so that when the corresponding loss function determines that the difference between the generated eye dynamic sequence and the target modality eye structure diagnosis video as the gold standard is less than an error threshold, the trained image-to-video generation network is determined as the second model.

[0014] In one embodiment of the present disclosure, the target modality key frame is input into the image-to-video generation network for prediction processing to obtain the first frame of the generated eye dynamic sequence, including: calculating the motion vector of the pixels between two adjacent video frames in the target modality eye structure diagnosis video as a feature label; inputting the feature label corresponding to the key frame and the first frame into the image-to-video generation network to perform distortion prediction on the key frame based on the feature label, and outputting the first frame of the eye dynamic sequence.

[0015] In one embodiment of the present disclosure, the method further includes: performing a consistency check on adjacent frames in the eye dynamic sequence based on bidirectional optical flow calculation to align the adjacent frames.

[0016] In one embodiment of the present disclosure, it also includes: calculating the difference between the first frame and the last frame in the target modality eye structure diagnosis video based on the temporal consistency restriction mechanism; performing threshold processing on the difference to obtain the corresponding clinical knowledge supervision mask.

[0017] In one embodiment of the present disclosure, the method further includes: using the clinical knowledge supervision mask to supervise the training of the first model and the training of the second model.

[0018] In one embodiment of the present disclosure, the clinical knowledge supervision mask is used to supervise the training of the first model and the training of the second model, including: in the training of the first model, guiding attention to a first region with significant temporal changes in the generated conversion frame based on the knowledge enhancement attention mechanism; in the training of the second model, guiding attention to a second region with significant temporal changes in the generated conversion frame based on the knowledge enhancement attention mechanism to determine the key region based on the first region and / or the second region; performing temporal temporal consistency and change perception operations on the generated eye dynamic sequence based on the clinical knowledge supervision mask, wherein, in the change perception operation, specific supervision is provided for the key region based on the knowledge-aware discriminator loss; and adjusting the pixel misalignment between the source modality eye image and the key region in the target modality eye structure diagnosis video based on the mask-enhanced local normalized cross entropy loss.

[0019] In one embodiment of the present disclosure, the eye structure diagnostic video of the target modality includes fluorescein fundus angiography or indocyanine green angiography, and the corresponding eye image includes a fundus image; the eye structure diagnostic video of the target modality includes optical coherence tomography video, and the corresponding eye image includes an eye plane scanning image.

[0020] In one embodiment of the present disclosure, the generative network includes at least one of a generative adversarial network, a derivative network of the generative adversarial network, a diffusion model, a derivative model of the diffusion model, a variational autoencoder, and a derivative structure of the variational autoencoder.

[0021] According to another aspect of the present disclosure, a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality is provided, comprising: inputting the static ophthalmic image of the source modality into an eye dynamic sequence generation model, the eye dynamic sequence generation model comprising a first model and a second model, wherein the first model and the second model are generated based on a generative network; performing cross-modal conversion on the eye image based on the first model to generate a target modality key frame; inputting the target modality key frame into the second model to perform time-series-based prediction on multiple video frames to obtain the multiple video frames, so as to generate a dynamic ophthalmic video of the target modality based on the multiple video frames.

[0022] According to another aspect of the present disclosure, a training device for a cross-modal eye dynamic sequence generation model is provided, comprising: a processing module for performing pixel-level alignment processing on a source modality eye image and a corresponding target modality eye structure diagnosis video; a selection module for selecting a target modality key frame in the target modality eye structure diagnosis video after the alignment processing, wherein the target modality key frame represents the modal characteristics of the target modality eye structure diagnosis video; a first training module for training an image-to-image generation network using the aligned source modality eye image as a first initial domain and the corresponding target modality key frame as a first target domain to obtain a first model; a second training module for training an image-to-video generation network using the target modality key frame as a second initial domain and the aligned target modality eye structure diagnosis video as a second target domain to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain the eye dynamic sequence generation model based on the first model and the second model.

[0023] According to another aspect of the present disclosure, there is provided a dynamic ophthalmic video device for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, comprising: an input module for inputting the static ophthalmic image of the source modality into an eye dynamic sequence generation model, the eye dynamic sequence generation model comprising a first model and a second model, wherein the first model and the second model are generated based on a generation network; a conversion module for performing cross-modal conversion on the eye image based on the first model to generate a target modality key frame; a prediction module for inputting the target modality key frame into the second model to perform time-series prediction on a plurality of video frames to obtain the plurality of video frames, so as to generate a dynamic ophthalmic video of the target modality based on the plurality of video frames.

[0024] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the above-mentioned training method for generating a cross-modal eye dynamic sequence model or a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality by executing the executable instructions.

[0025] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned training method for the cross-modal eye dynamic sequence generation model or the method for converting static ophthalmic images of a source modality into dynamic ophthalmic videos of a target modality.

[0026] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned training method for the cross-modal eye dynamic sequence generation model or the method for converting static ophthalmic images of a source modality into dynamic ophthalmic videos of a target modality.

[0027] The embodiments of the present disclosure provide a training scheme for a cross-modal eye dynamic sequence generation model and a scheme for converting static ophthalmic images of a source modality into dynamic ophthalmic videos of a target modality. Through alignment processing, the eye image and the eye structure diagnosis video of the target modality can be aligned at the pixel level to ensure that they have consistent spatial information, so as to accurately capture the pathological features of retinal blood vessels, etc. during the model training process. Furthermore, key frames that can characterize the modal features of the eye structure diagnosis video are selected to maintain consistency with the eye image after alignment processing. Through the training of the generation model, the mapping relationship between the source modality eye image and the target modality eye structure diagnosis video can be learned, thereby realizing the generation of realistic eye dynamic sequences. The generation method of the eye dynamic sequence is not only non-invasive, safe and low-cost, but the generated eye dynamic sequence results can ensure authenticity and accuracy, so as to serve as a reliable judgment basis for clinical diagnosis.

[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0030] FIG1 is a schematic diagram showing a structure of a system for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video image of a target modality according to an embodiment of the present disclosure;

[0031] FIG2 shows a flow chart of a training method for a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0032] FIG3 shows a flow chart of another method for training a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0033] FIG4 is a schematic diagram showing a training method for a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0034] FIG5 is a schematic diagram showing another training method for a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0035] FIG6 shows a flow chart of a training method for yet another cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0036] FIG7 is a schematic diagram showing a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality according to an embodiment of the present disclosure;

[0037] FIG8 shows a flow chart of a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality according to an embodiment of the present disclosure;

[0038] FIG9 is a schematic diagram showing another method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality according to an embodiment of the present disclosure;

[0039] FIG10 is a schematic diagram showing a training device for a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure;

[0040] FIG11 is a schematic diagram of a device for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality according to an embodiment of the present disclosure;

[0041] FIG12 shows a structural block diagram of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0043] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0044] In the field of clinical ophthalmology, fundus fluorescein angiography (FFA) is a key technology for studying retinal circulation dynamics. It is used to diagnose diseases such as diabetic retinopathy, hypertensive retinopathy and macular degeneration. However, this technology requires intravenous injection of contrast agents, which may have related side effects and may not be suitable for some patients. Indocyanine green angiography (ICGA) is an invasive diagnostic technology mainly used for choroidal diseases, macular degeneration, intraocular tumors and retinal vascular diseases. Optical coherence tomography (OCT) provides detailed information about the internal structure of the retina. Although these technologies can obtain detailed information about the internal structure of the eye, they are expensive and require professional knowledge to operate. Some technologies may also cause discomfort to the human body.

[0045] In contrast, color fundus photography is non-invasive and rapid. However, since it can only capture static images, it cannot provide clear contrast between pathological lesions and normal structures, and lacks the ability to detect dynamic processes observed in the above diagnostic methods. Therefore, using machine learning and image-to-video translation technology to generate various dynamic ophthalmic videos from color fundus images has important clinical prospects.

[0046] The solution provided in the present application can align the eye image of the source modality with the eye structure diagnosis video of the target modality at the pixel level through alignment processing, ensuring that they have consistent spatial information, so as to accurately capture the pathological characteristics of retinal blood vessels, etc. during the model training process. Furthermore, key frames that can represent the modal characteristics of the eye structure diagnosis video are selected to maintain consistency with the eye image after alignment processing. By training the generation model, the mapping relationship between the source modality eye image and the target modality eye structure diagnosis video can be learned, thereby realizing the generation of realistic eye dynamic sequences. The generation method of the eye dynamic sequence is not only non-invasive, safe and low-cost, but the generated eye dynamic sequence results can ensure authenticity and accuracy, so as to serve as a reliable judgment basis for clinical diagnosis.

[0047] FIG1 shows a schematic structural diagram of a warning information mass sending decision system according to an embodiment of the present disclosure, which includes a plurality of terminals 120 and a server cluster 140 .

[0048] The terminal 120 may be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the terminal 120 may also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0049] The terminal 120 may be installed with an application for providing group warning information decision-making.

[0050] The terminal 120 is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0051] Server cluster 140 is a single server, or a combination of multiple servers, a virtualization platform, or a cloud computing service center. Server cluster 140 provides backend services for the early warning information mass-distribution decision-making application. Optionally, server cluster 140 performs primary computing tasks, while terminals 120 performs secondary computing tasks; alternatively, server cluster 140 performs secondary computing tasks, while terminals 120 performs primary computing tasks; alternatively, terminals 120 and server cluster 140 utilize a distributed computing architecture for collaborative computing.

[0052] In some optional embodiments, the server cluster 140 is used to store programs for converting static ophthalmic images of a source modality into dynamic ophthalmic videos of a target modality.

[0053] Optionally, the application clients installed in different terminals 120 are the same, or the application clients installed on the two terminals 120 are clients of the same type of application on different control system platforms. Based on the different terminal platforms, the specific form of the application client may also be different. For example, the application client may be a mobile phone client, a PC client, or a World Wide Web (Web) client.

[0054] Those skilled in the art will appreciate that the number of the terminals 120 may be greater or less. For example, there may be only one terminal, or there may be dozens, hundreds, or even more terminals. The embodiments of the present application do not limit the number and device types of the terminals.

[0055] Optionally, the system may further include a management device (not shown in FIG1 ), which is connected to the server cluster 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0056] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0057] Below, the training method of the cross-modal eye dynamic sequence generation model and the method of converting the static ophthalmic image of the source modality into the dynamic ophthalmic video of the target modality in this example embodiment will be described in more detail with reference to the accompanying drawings and embodiments.

[0058] The following describes in more detail the eye dynamic sequence generation model and each step of the generation method in this exemplary embodiment with reference to the accompanying drawings and embodiments.

[0059] As shown in FIG2 , a method for training a cross-modal eye dynamic sequence generation model according to an embodiment of the present disclosure includes:

[0060] Step S202 : performing pixel-level alignment processing on the source modality eye image and the corresponding target modality eye structure diagnosis video.

[0061] In some embodiments, the source modality is a 2D modality and the target modality is a 3D modality.

[0062] In some embodiments, the source modality and the target modality are both 2D modalities or both 3D modalities.

[0063] In some embodiments, the target modality of the eye structure diagnostic video includes fundus fluorescein angiography, indocyanine green angiography, or optical coherence tomography.

[0064] In some embodiments, the target modality eye structure diagnosis video includes fluorescein fundus angiography or indocyanine green angiography, and the corresponding eye image includes a fundus image.

[0065] In some embodiments, the eye structure diagnostic video of the target modality includes an optical coherence tomography video, and the corresponding eye image includes an eye planar scan image.

[0066] Illustratively, pairs of eye images and corresponding sodium fluorescein angiography or static choroidal fluorescein angiography or optical coherence tomography images are aligned at the pixel level, and from each combination, frames representing different stages (early, middle, late, etc.) are sampled to create a balanced simulated video and ensure consistency between different stages.

[0067] Step S204 : selecting target modality key frames in the target modality eye structure diagnosis video after the alignment process, where the target modality key frames represent modality features of the target modality eye structure diagnosis video.

[0068] Among them, some representative frames are selected from the aligned eye structure diagnosis video as key frames. These key frames should be able to represent the modal characteristics of the entire video and maintain consistency with the aligned eye image.

[0069] In some embodiments, the key frame may be the first frame of the eye structure diagnosis video, or may be a video in the eye structure diagnosis video that has an angle mapping relationship with the eye image.

[0070] In step S206 , the aligned source modality eye image is used as the first initial domain, and the corresponding target modality key frame is used as the first target domain, and the image-to-image generative network is trained to obtain a first model.

[0071] The image-to-image generation network may be a generation network from a source modality image to a target modality image.

[0072] The aligned eye image is used as the first initial domain and the corresponding keyframe is used as the first target domain. An image-to-image generative network is used to train these two domains to learn the mapping relationship between the images of the two modalities. The goal is to generate realistic keyframes based on the eye image so that they are visually indistinguishable from the aligned keywords.

[0073] In step S208, the target modality key frame is used as the second initial domain, and the aligned target modality eye structure diagnosis video is used as the second target domain. The image-to-video generation network is trained to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain an eye dynamic sequence generation model based on the first model and the second model.

[0074] Among them, the key frames are used as the second initial domain, and the aligned eye structure diagnosis video is used as the second target domain. The two domains are trained using an image-to-video generative network to learn the conditional probability distribution. The goal is to generate eye structure diagnosis videos with high quality and diversity.

[0075] Furthermore, based on the first model and the second model, a cross-modal eye dynamic sequence generation model is constructed. The model can accept the source modality eye image as input and generate the corresponding eye structure diagnosis video, thereby realizing the generation of eye dynamic sequence.

[0076] In some embodiments, the generative network includes at least one of a generative adversarial network, a derivative network of a generative adversarial network, a diffusion model, a derivative model of a diffusion model, a variational autoencoder, and a derivative structure of a variational autoencoder.

[0077] In this embodiment, through alignment processing, the eye image and the eye structure diagnosis video of the target modality can be aligned at the pixel level to ensure that they have consistent spatial information, so that the pathological characteristics of retinal blood vessels, etc. can be accurately captured during the model training process. Furthermore, key frames that can characterize the modal characteristics of the eye structure diagnosis video are selected to maintain consistency with the eye image after alignment processing. Through training the generation network, the mapping relationship between the source modality eye image and the target modality eye structure diagnosis video can be learned, thereby realizing the generation of realistic eye dynamic sequences. The generation method of the eye dynamic sequence is not only non-invasive, safe and low-cost, but the generated eye dynamic sequence results can ensure authenticity and accuracy, so that they can be used as a reliable judgment basis in clinical diagnosis.

[0078] In one embodiment of the present disclosure, pixel-level alignment of an eye image and a corresponding target modality eye structure diagnosis video includes:

[0079] For the source module eye image and target modality eye structure diagnosis video from the same eye in the same examination, a feature segmentation algorithm is used to extract eye inspection features.

[0080] In some embodiments, ocular inspection features include retinal blood vessels, lens, eyelid structure, iris features, etc.

[0081] In some embodiments, the feature segmentation algorithm is a blood vessel segmentation algorithm, and the eye inspection feature is a retinal blood vessel, that is, the blood vessel segmentation algorithm is used to perform the retinal blood vessel extraction operation.

[0082] Among them, the vascular segmentation algorithm is a computational method used to extract vascular structures from eye images. The vascular segmentation algorithm includes methods based on threshold processing, edge detection, morphological operations, machine learning and deep learning. These algorithms can help doctors diagnose eye diseases and play an important role in ophthalmic image analysis.

[0083] After the extraction operation is completed, pixel-level alignment processing is performed based on eye inspection features.

[0084] In some embodiments, the ocular inspection features are retinal blood vessels, that is, pixel-level alignment processing is performed based on the retinal blood vessels.

[0085] The alignment process can be achieved through image registration technology, such as feature point matching or similarity transformation-based methods. The purpose of this step is to align eye images obtained at different examination time points or with different imaging methods for subsequent comparison and analysis.

[0086] In this example, pixel-level alignment based on retinal blood vessels enables more accurate analysis and comparison of multiple examination results from the same eye. Model training based on these features enables the generated eye dynamic sequence to correctly restore vascular structures with pathological characteristics.

[0087] In one embodiment of the present disclosure, the eye inspection features include retinal blood vessels, and pixel-level alignment processing based on the eye inspection features includes:

[0088] A keypoint detector is used to detect pixel-level keypoints of retinal blood vessels, and the pixel-level keypoints are used as alignment benchmarks for feature matching between the source module eye image and the target modality eye structure diagnosis video.

[0089] In some embodiments, the keypoint detector is an AKAZE keypoint detector.

[0090] Among them, AKAZE (Accelerated-KAZE) is an accelerated KAZE (2D Feature Detection and Descriptor Algorithm) algorithm. The AKAZE algorithm uses a nonlinear scale space to detect key points in retinal vascular images by using Gaussian filters at different image resolutions, and then uses a fast feature detection algorithm to detect image areas with high local symmetry, which are generally considered to be key points in retinal vessels.

[0091] The matching feature points are estimated based on a random sampling consistency algorithm to estimate the homography matrix between the source module eye image and the retinal blood vessels in the target modality eye structure diagnosis video.

[0092] The Random Sampling Consensus Algorithm (RANSAC) is an iterative method for fitting mathematical models and eliminating outliers. It can effectively process data containing noise and outliers. By extracting feature points from the source module eye image and the target modality eye structure diagnosis video frame and using these feature points for matching, the RANSAC algorithm can estimate these matching points to determine the homography transformation relationship between the two images, namely the homography matrix. The homography matrix describes the geometric transformation relationship between two images with different perspectives and can be used to achieve image alignment and registration.

[0093] Abnormal frames in the eye structure diagnosis video are excluded based on the homography matrix to obtain aligned source module eye images and target modality eye structure diagnosis video.

[0094] Abnormal frames are those with poor image quality or that cannot be aligned with the eye image due to factors such as motion blur and noise. By applying the homography matrix, these abnormal frames can be filtered out, resulting in an aligned source module eye image and target modality eye structure diagnosis video.

[0095] In this embodiment, during the alignment process, the angiography video frames in the ocular structure diagnosis video of the same eye are first aligned and registered with each other, and then aligned and registered with the eye image. In some embodiments, a key point detector is used for feature matching, and RANSAC (random sampling consensus) is used to generate a homography matrix and outlier rejection. Furthermore, in order to exclude incorrectly registered pairs, validity restrictions can be added and image pairs with poor registration performance can be filtered out. This can be set empirically based on the data set.

[0096] As shown in FIG3 , in one embodiment of the present disclosure, the aligned source modality eye image is used as the first initial domain, and the corresponding target modality key frame is used as the first target domain. The image-to-image generative network is trained to obtain a first model, including:

[0097] In step S302, the image-to-image generation network includes a U-shaped structure generator, which uses the lesion supervision loss as the loss function, and uses the first initial domain and the first target domain to train the U-shaped structure generator so that the U-shaped structure generator converts the input source modality eye image into a conversion frame similar to the target modality key frame, wherein the U-shaped structure generator includes an encoder and a decoder, the encoder is used to encode the source modality eye image into a representation of the latent space, and the decoder is used to decode the representation of the latent space into output data.

[0098] In step S304, the converted frames and the corresponding target modality key frames are used as training data sets, and the perceptual loss and feature matching loss are used as loss functions to continue training the generator.

[0099] Those skilled in the art will appreciate that all generative networks adapted by the U-shaped structure, including but not limited to generative adversarial networks and diffusion models, are within the scope of protection of this disclosure.

[0100] The aligned first initial domain and the first target domain training are input into a pre-designed generative network. In some embodiments, the generator is constructed with a series of stacked transposed convolutional layers to incrementally enhance the image resolution. The generator is enriched by integrating jump connections and low-level and high-level features to maintain details and contextual information. This strategy enables the generator to gradually generate complex images similar to real eye structure diagnostic videos.

[0101] Step S306, iteratively train the generator until the probability that the model determines the conversion frame as a key frame is greater than the probability threshold, and use the trained generator as the first model.

[0102] In this embodiment, in order to make full use of a large number of angiography videos, a generative network is trained to generate all video frames. The model can use the minimum-maximum game to convert images into different fields during the training process. Furthermore, the generator is iteratively trained using lesion supervision loss, perceptual loss, and feature matching loss as loss functions to obtain a first model. The first model uses the trained generator to generate conversion frames similar to the key frames to ensure the reliability of the subsequent conversion of static ophthalmic images of the source modality into dynamic ophthalmic videos of the target modality.

[0103] In one embodiment of the present disclosure, the generator includes a global generator network and a local enhancement network, and training the generator further includes:

[0104] The eye image with a first resolution and the target modality key frame are input into the local enhancement network and upsampled based on the attention mechanism to output a local enhanced image with a second resolution, where the second resolution is N times the first resolution, and N is greater than 1.

[0105] In some embodiments, G uses a coarse-to-fine structure, including a global generator network G1 and a local enhancement network G2. G1 maintains the resolution of the input and output samples, N=4, that is, the frame output by G2 is 4 times the input, achieving higher resolution.

[0106] An attention mechanism is introduced during the upsampling process of G1 and G2 to transfer information from coarse scale to fine scale and enhance task-specific response areas in shallower network layers.

[0107] In this embodiment, an attention mechanism is incorporated into the network architecture to enhance information transfer, ensure that relevant features participate in deeper updates, and improve the overall quality of the generated conversion frames.

[0108] As shown in Figure 4, as an implementation of the first model, the source modality eye image is a fundus image, and the corresponding target modality eye structure diagnostic video is fluorescein fundus angiography. The fundus image 402 is input into the generator 404 in the first model to obtain a conversion frame 406. Referring to Figure 4, it can be seen that there is a high degree of similarity between the conversion frame 406 and the key frame 408 of the real fluorescein fundus angiography corresponding to the fundus image 402.

[0109] In addition, in some embodiments, the generative network is a generative adversarial network, which includes a generator and may also include a discriminator. In some implementations, three discriminators (D1, D2, and D3) with similar network structures are used to process images of different scales, ensuring the distinction between high-resolution real images and generated FFA images without overfitting. To stabilize the training process, perceptual loss and feature matching loss are introduced to evaluate each feature extraction layer of each discriminator (D1, D2, and D3).

[0110] As shown in Figure 5, the generative adversarial network is used as an image-to-image generative network to train the first model. The eye image is a plane scanning image, and the corresponding target modality eye structure diagnosis video is an optical coherence tomography video. The specific processing process includes: inputting the plane scanning image 502 into the generator 504 to obtain the conversion frame 506, inputting the key frame 508 of the real optical coherence tomography video corresponding to the conversion frame 506 and the plane scanning image 502 into the discriminator 510, and outputting the discrimination result to detect the training result of the first model based on the discrimination result.

[0111] As shown in FIG6 , in one embodiment of the present disclosure, the target modality keyframes are used as the second initial domain, and the aligned target modality eye structure diagnosis video is used as the second target domain. The image-to-video generative network is trained to obtain a second model for frame-by-frame prediction of eye dynamic sequences, including:

[0112] Step S602: Input the target modality key frame into the image and feed it into the video generation network for prediction processing to obtain the first frame of the eye dynamic sequence.

[0113] Among them, the image-to-video generation network is a deep learning model. The generator also includes an encoder and a decoder. The encoder converts the input data into distribution parameters in the latent space. The decoder uses these parameters to sample from the latent space and map it back to the data space. The image-to-video generation network is used to generate a network to predict the eye dynamic sequence and generate the first frame.

[0114] In some embodiments, the image-to-video generation network can be a conditional variational autoencoder (CVAE), which is a variational autoencoder model that combines conditional information. It can better control the output results when generating data and has good generation and learning capabilities.

[0115] Step S604: After coupling the first frame and the target modality key frame, the image is input into the video generation network for prediction processing to obtain the second frame of the generated eye dynamic sequence.

[0116] Among them, by coupling the first frame with the key frame, it can be better used as input data to predict the second frame of the eye dynamic sequence.

[0117] Step S606 , predicting the remaining video frames frame by frame based on the time sequence until the image-to-video generation network predicts all video frames of the eye dynamic sequence.

[0118] Among them, the generative network generates video frames of the entire eye dynamic sequence through iterative prediction.

[0119] Step S608, iteratively train the image-to-video generative network so that when the difference between the generated eye dynamic sequence determined by the corresponding loss function and the target modality eye structure diagnosis video serving as the gold standard is less than an error threshold, the trained image-to-video generative network is determined as the second model.

[0120] Among them, the gold standard refers to the standard or method that is considered to be the most credible and authoritative in the field of medicine and clinical research, and is used to determine the accuracy, effectiveness or reliability of a disease, diagnosis or treatment method.

[0121] In some embodiments, the image-to-video generation network is continuously optimized through iterative training until the difference between the eye dynamic sequence video frames it generates and the real eye structure diagnosis video is less than an error threshold.

[0122] In addition, the loss function used in the training process of the second model is generated based on the reconstruction loss, smoothness constraint, consistency loss and loss in rendering.

[0123] In this embodiment, a second model is obtained by training an image-to-video generative network to generate dynamic videos based on a single static keyframe. This process involves flow prediction and video frame generation. The image-to-video generative network is used to realize the prediction and training of eye dynamic sequences. The model is continuously optimized through iterative training to minimize the difference between the eye dynamic sequence video frames it generates and the real eye structure diagnosis video.

[0124] In one embodiment of the present disclosure, the target modality key frame is input into the image generation network of the video for prediction processing to obtain the first frame of the generated eye dynamic sequence, including:

[0125] The motion vector of pixels between two adjacent video frames in the target modality eye structure diagnosis video is calculated as the feature label.

[0126] Among them, by calculating the pixel motion vectors between two adjacent video frames in the target modality eye structure diagnosis video, the feature labels describing the motion between video frames can be obtained. The motion vectors of these pixels can characterize the motion state between video frames, which is manifested as vascular tissue contrast and dynamic insights, and serve as feature labels for the subsequent input image to video generation network.

[0127] The feature labels corresponding to the key frame and the first frame are input into the image-to-video generation network to perform distortion prediction on the key frame based on the feature labels and output the first frame of the eye dynamic sequence.

[0128] Among them, the image-to-video generation network processes the key frames, that is, performs distortion prediction based on the feature labels to generate the first frame of the eye dynamic sequence. The generation network can use the feature label information to generate the first frame of the eye dynamic sequence that matches the key frames and meets the feature label requirements.

[0129] In some embodiments, the predicted optical flow is used to warp the initial frame based on the feature labels to generate an initial future frame, and the post-processing network refines the frame to solve problems such as occlusion or partial missing.

[0130] In this embodiment, an eye image is used as the first frame input to the generative network, which outputs the first video frame of the eye dynamic sequence. The eye image and the generated first frame are then coupled as input to generate the next frame, and so on. The generative network processes the feature labels of the ocular structure diagnostic video and generates the first frame of the eye dynamic sequence based on these feature labels. This method can generate an eye dynamic sequence that meets the feature label requirements based on the changes in pixel motion vectors between video frames.

[0131] In some embodiments, in order to ensure the smoothness of the generated eye dynamic sequence, multi-frame input and smoothing processing methods can be further introduced. For example, three consecutive frames of the target modality eye structure diagnosis video serving as the gold standard are input into the second model through a sliding window, and three-frame averaging is performed on the frames aggregated in the sliding window to provide a longer temporal context for each generated frame. This processing method helps to make the transition between adjacent video frames smoother, thereby ensuring the continuity of the generated eye dynamic sequence.

[0132] In one embodiment of the present disclosure, a smooth eye dynamic sequence may also be generated based on a temporal consistency restriction mechanism.

[0133] In one embodiment of the present disclosure, it also includes: calculating the difference between the first frame and the last frame in the target modality eye structure diagnosis video based on the temporal consistency restriction mechanism; performing threshold processing on the difference to obtain the corresponding clinical knowledge supervision mask.

[0134] In this embodiment, a clinical knowledge supervision mask is obtained by taking the difference between the first and last frames in the target modality eye structure diagnosis video, while ensuring accuracy without the need for additional manual annotation or model training.

[0135] In one embodiment of the present disclosure, the method further includes: using the clinical knowledge supervision mask to supervise the training of the first model and the training of the second model.

[0136] In one embodiment of the present disclosure, data enhancement may also be applied during model training, and the application process includes randomly selecting generated frames or frames used as gold standards as input during training to enhance its adaptability and robustness in various scenarios.

[0137] In one embodiment of the present disclosure, data enhancement includes but is not limited to random cropping, scaling, and color enhancement.

[0138] In one embodiment of the present disclosure, a clinical knowledge supervision mask is used to supervise the training of a first model and the training of a second model, including: in the training of the first model, guiding attention to a first region with significant temporal changes in the generated transition frame based on a knowledge-enhanced attention mechanism; in the training of the second model, guiding attention to a second region with significant temporal changes in the generated transition frame based on a knowledge-enhanced attention mechanism to determine a key region based on the first region and / or the second region; performing temporal temporal consistency and change perception operations on the generated eye dynamic sequence based on the clinical knowledge supervision mask, wherein, in the change perception operation, specific supervision is provided for the key region based on the knowledge-aware discriminator loss; and adjusting the pixel misalignment between the source modality eye image and the key region in the target modality eye structure diagnosis video based on the mask-enhanced local normalized cross entropy loss.

[0139] In this embodiment, by using clinical knowledge supervision masks to supervise the training of the first model and the training of the second model, the pixel misalignment between the source modality eye image and the target modality eye structure diagnostic video in the key area can be improved, thereby ensuring the accuracy and reliability of the eye dynamic sequence generation operation generated by the trained eye dynamic sequence generation model.

[0140] In some embodiments, during the model training process, gradient-guided loss can also be added to enhance the generation of high-frequency components, including retinal structure and lesions. The input of this process will be eye images, and the output will be a dynamic sequence of the eye at a certain stage. Deep learning algorithms can be developed using PyTorch.

[0141] In one embodiment of the present disclosure, the method further includes: performing a consistency check on adjacent frames in the eye dynamic sequence based on bidirectional optical flow calculation to align the adjacent frames.

[0142] Among them, bidirectional optical flow refers to the method used in computer vision to estimate the direction and speed of optical flow at the pixel level in video sequences.

[0143] In this embodiment, bidirectional optical flow is introduced to perform consistency check to ensure accurate alignment and consistency between generated video frames, thereby ensuring smooth playback of the generated eye dynamic sequence.

[0144] As shown in Figure 7, after the first model and the second model are trained, the first model is used to output key frames, and the second model is used to convert the static ophthalmic image of the source modality, i.e., the key frame, into a dynamic ophthalmic video of the target modality. The source modality eye image is a plane scanning image, and the corresponding target modality eye structure diagnostic video is an optical coherence tomography video. After obtaining the key frame based on the plane scanning image 702, the target modality key frame is input into the second model to obtain a simulated optical coherence tomography video 704.

[0145] As shown in FIG8 , a method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality according to an embodiment of the present disclosure includes:

[0146] Step S802: Input the static ophthalmic image of the source modality into an eye dynamic sequence generation model. The eye dynamic sequence generation model includes a first model and a second model, wherein the first model and the second model are generated based on a generation network.

[0147] Step S804: Perform cross-modal conversion on the eye image based on the first model to generate a target modality keyframe.

[0148] Step S806 : Input the target modality key frame into the second model to predict the multiple video frames based on the time sequence to obtain the multiple video frames, so as to generate a dynamic ophthalmic video of the target modality based on the multiple video frames.

[0149] In this embodiment, by utilizing multiple machine learning models, the features of the eye image are extracted and key frames are generated through the first model, and then the target modality key frames are input into the second model for frame-by-frame prediction and optimization, and finally a complete eye dynamic sequence is obtained.

[0150] In some embodiments, the eye image includes a color fundus image, and the eye dynamic sequence includes a simulated fluorescein fundus angiography. The following is a process of generating a simulated fluorescein fundus angiography (FFA) based on a color fundus image (CFP), and further describes in detail the scheme of matching the retinal blood vessels in the CFP and the real FFA at the pixel level in the present disclosure, and training the generation network to predict high-resolution FFA videos in an autoregressive manner.

[0151] In the first stage, a generative model is pre-trained to predict keyframes from FFA videos. In the second stage, the weights are transferred to CFP-FFA video pairs to predict FFA videos from CFP input. Validation results of this scheme demonstrate realistic generation on xx internal and external test sets subjectively evaluated by three ophthalmologists. Furthermore, when tested on DR (diabetic retinopathy) datasets (including EyePACS, MESSIDOR2, and IDRID), AMD (age-related macular degeneration) datasets, and a multi-disease dataset (a challenging multi-class retinal image library with many rare diseases and severe class imbalance), adding the generated eye dynamic sequences improves the diagnostic accuracy of DR, AMD, and multiple rare diseases, respectively. While further research is needed for clinical application to replace traditional invasive angiography, the validation results demonstrate that this technology can serve as a novel retinal foundation model and can be immediately implemented to improve automated retinal disease screening processes.

[0152] Fluorescein angiography (FFA) is a key method for detecting lesions associated with vascular-retinal barrier breakdown and monitoring DR treatment response. This technology dynamically highlights lesion changes through the injection of dye, which is particularly helpful in highlighting important lesions that are difficult to see clearly on color fundus photographs. However, FFA is an invasive test that requires intravenous injection of dye and may cause serious side effects such as nausea, heart attack, and anaphylactic shock. Therefore, it is not suitable for routine community screening of chorioretinal conditions and can only be performed under close monitoring by professional technicians. Therefore, in areas where the prevalence of DR and AMD is increasing, it is crucial to develop non-invasive, safe, and low-cost alternatives to FFA.

[0153] Preprocessing stage: Fundus images CF and angiography videos from the same eye and the same visit are used for matching. Retinal vessels are extracted from the fundus images CF and angiography videos using a vessel segmentation algorithm to achieve pixel-level image matching. First, angiography video frames of the same eye are registered with each other and then with the fundus images CF. Feature matching is performed using the AKAZE keypoint detector, and RANSAC (random sample consensus) is used to generate homography matrices and outlier rejection. In order to exclude incorrectly registered pairs, validity constraints are added and image pairs with poor registration performance are filtered out, which are set empirically based on the dataset.

[0154] In the first model training phase, to fully utilize the large amount of angiography videos, a generative network is trained to generate all video frames. In some embodiments, the generative network is constructed with a series of stacked transposed convolutional layers to incrementally enhance image resolution. The generator is enriched by integrating skip connections and low-level and high-level features to maintain details and contextual information.

[0155] This strategy enables the generator to gradually produce complex images that resemble real FFA images. Furthermore, FFA videos are generated in an autoregressive manner, where the input is the first frame, followed by the next frame of the venous phase, and the output will be their next frame.

[0156] Second model training stage: The mapping of fundus image CF to angiography uses the weights of the first stage, uses the fundus image CF image as the first frame, and outputs the first FFA video frame, then couples the fundus image CFP with the generated first frame as input to generate the next frame, and so on, resizing the image to 768×768.

[0157] In addition, a gradient-guided loss is added to enhance the generation of high-frequency components, including retinal structures and lesions. The input of this process will be the fundus image CF, and the output will be the FFA video from the venous phase to the late phase.

[0158] Evaluation phase: The generated simulated videos will be evaluated against real videos based on the following metrics:

[0159] Mean Absolute Error (MAE): MAE calculates the average absolute pixel difference between the generated image and the corresponding real image, quantifying the overall difference in pixel values ​​and indicating the level of accurate detail generated.

[0160] Peak Signal-to-Noise Ratio (PSNR): PSNR is an approximate measure of human perception of reconstruction quality. It measures the ratio between the maximum possible power of the signal and the power of the noise interfering with it.

[0161] Structural Similarity Measure (SSIM): SSIM evaluates the structural similarity between images, with a value of 1 indicating perfect similarity and 0 indicating no similarity. SSIM provides insights into the visual similarity and consistency between generated images and real images.

[0162] Multi-Scale Structural Similarity Measure (MS-SSIM): MS-SSIM provides greater flexibility in incorporating variations in different observation conditions and image resolutions.

[0163] Among them, the higher the SSIM, MS-SSIM, and PSNR, the better the quality of the generated image.

[0164] As shown in FIG9 , the eye image is a fundus image, and the corresponding target modality eye structure diagnosis video is fluorescein fundus angiography. After obtaining the key frame based on the fundus image 902 , the target modality key frame is input into the second model to obtain the simulated fluorescein fundus angiography 904 .

[0165] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0166] The following describes a cross-modal eye dynamic sequence generation model training device 1000 according to an embodiment of the present invention with reference to Figure 10. The cross-modal eye dynamic sequence generation model training device 1000 shown in Figure 10 is merely an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0167] The training device 1000 for the cross-modal eye dynamic sequence generation model is expressed in the form of a hardware module. The components of the training device 1000 for a cross-modal eye dynamic sequence generation model may include but are not limited to: a processing module 1002, used to perform pixel-level alignment processing on a source modality eye image and a corresponding target modality eye structure diagnosis video; a selection module 1004, used to select a target modality key frame in the target modality eye structure diagnosis video after the alignment processing, the target modality key frame representing the modal features of the target modality eye structure diagnosis video; a first training module 1006, used to train an image-to-image generation network using the aligned source modality eye image as a first initial domain and the corresponding target modality key frame as a first target domain to obtain a first model; a second training module 1008, used to train an image-to-video generation network using the target modality key frame as a second initial domain and the aligned target modality eye structure diagnosis video as a second target domain to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain an eye dynamic sequence generation model based on the first model and the second model.

[0168] The following describes a dynamic ophthalmic video apparatus 1100 for converting a static ophthalmic image of a source modality into a target modality according to an embodiment of the present invention with reference to FIG11 . The dynamic ophthalmic video apparatus 1100 for converting a static ophthalmic image of a source modality into a target modality shown in FIG11 is merely an example and should not limit the functionality or scope of use of the embodiments of the present invention.

[0169] The apparatus 1100 for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality is implemented as a hardware module. The components of the apparatus 1100 for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality may include, but are not limited to: an input module 1102 for inputting an eye image into an eye dynamic sequence generation model, the eye dynamic sequence generation model comprising a first model and a second model, wherein the first model and the second model are generated based on a generative network; a conversion module 1104 for performing a cross-modal conversion of the eye image based on the first model to generate a target modality keyframe; and a prediction module 1106 for inputting the target modality keyframe into a second model to perform a time-series prediction on multiple video frames to obtain multiple video frames, thereby generating a dynamic ophthalmic video of the target modality based on the multiple video frames.

[0170] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Therefore, various aspects of the present invention may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0171] The electronic device 1200 according to this embodiment of the present invention is described below with reference to FIG12. The electronic device is an eye dynamic sequence generator. The electronic device 1200 shown in FIG12 is only an example and should not limit the functions and scope of use of the embodiment of the present invention.

[0172] As shown in Figure 12, electronic device 1200 is implemented as a general-purpose computing device. Components of electronic device 1200 may include, but are not limited to, the aforementioned at least one processing unit 1210, the aforementioned at least one storage unit 1220, and a bus 1230 connecting various system components (including storage unit 1220 and processing unit 1210).

[0173] The storage unit stores program code, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present invention described in the "Exemplary Method" section above. For example, the processing unit 1210 can perform the scheme described in steps S202 to S208 as shown in Figure 2.

[0174] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 12201 and / or a cache memory unit 12202 , and may further include a read-only memory unit (ROM) 12203 .

[0175] The storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0176] The bus 1230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0177] The electronic device 1200 can also communicate with one or more external devices 1270 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1200, and / or any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 1250. Furthermore, the electronic device 1200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 1200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0178] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the embodiments of the present disclosure.

[0179] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the methods described above. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product comprising program code that, when executed on a terminal device, causes the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section above.

[0180] According to an embodiment of the present invention, a program product for implementing the above-mentioned method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, and the program can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0181] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0182] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0183] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0184] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Python, Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0185] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0186] In addition, although the various steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps, etc. Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the embodiment of the present disclosure. After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A training method for a cross-modal eye dynamic sequence generation model, comprising: Perform pixel-level alignment on the source modality eye image and the corresponding target modality eye structure diagnosis video; Selecting a target modality key frame in the target modality eye structure diagnosis video after the alignment process, wherein the target modality key frame represents a modality feature of the target modality eye structure diagnosis video; Using the aligned source modality eye image as a first initial domain and the corresponding target modality key frame as a first target domain, training an image-to-image generative network to obtain a first model; The target modality key frame is used as the second initial domain, and the aligned target modality eye structure diagnosis video is used as the second target domain. The image-to-video generation network is trained to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain the eye dynamic sequence generation model based on the first model and the second model.

2. The method for training a cross-modal eye dynamic sequence generation model according to claim 1, wherein: Perform pixel-level alignment of the source modality eye image and the corresponding target modality eye structure diagnosis video, including: For the source module eye image and the target modality eye structure diagnosis video from the same eye in the same examination, a feature segmentation algorithm is used to extract eye inspection features; Pixel-level alignment processing is performed based on the eye inspection features.

3. The training method for a cross-modal eye dynamic sequence generation model according to claim 2, wherein: Performing pixel-level alignment processing based on the eye inspection features includes: The eye inspection features include retinal blood vessels, pixel-level key points of the retinal blood vessels are detected using a key point detector, and the pixel-level key points are used as an alignment reference to perform feature matching between the source module eye image and the target modality eye structure diagnosis video; estimating matching feature points based on a random sampling consistency algorithm to estimate a homography matrix between the source module eye image and the retinal blood vessels in the target modality eye structure diagnosis video; Abnormal frames in the eye structure diagnosis video are excluded based on the homography matrix to obtain aligned source module eye images and target modality eye structure diagnosis video.

4. The method for training a cross-modal eye dynamic sequence generation model according to any one of claims 1 to 3, wherein: The aligned source modality eye image is used as a first initial domain, and the corresponding target modality key frame is used as a first target domain. An image-to-image generative network is trained to obtain a first model, including: The image-to-image generation network includes a U-shaped generator, which uses lesion supervision loss as a loss function and uses the first initial domain and the first target domain to train the U-shaped generator, so that the U-shaped generator converts the input source modality eye image into a conversion frame similar to the target modality key frame, wherein the U-shaped generator includes an encoder and a decoder, the encoder is used to encode the source modality eye image into a representation of a latent space, and the decoder is used to decode the representation of the latent space into the target modality key frame; The converted frames and the corresponding target modality key frames are used as training data sets, and the perceptual loss and feature matching loss are used as loss functions to continue training the generator to obtain the first model.

5. The method for training a cross-modal eye dynamic sequence generation model according to claim 4, wherein: The generator includes a global generator network and a local enhancement network, and training the generator further includes: The eye image with a first resolution and the target modality key frame are input into the local enhancement network and upsampled based on the attention mechanism to output a local enhanced image with a second resolution, where the second resolution is N times the first resolution, and N is greater than 1.

6. The method for training a cross-modal eye dynamic sequence generation model according to any one of claims 1 to 5, wherein: The target modality keyframe is used as a second initial domain, and the aligned target modality eye structure diagnosis video is used as a second target domain. The image-to-video generative network is trained to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, including: Inputting the key frame into the image-to-video generation network for prediction processing to obtain the first frame of the generated eye dynamic sequence; coupling the first frame and the key frame and inputting the resultant image into a video generation network for prediction processing to obtain a second frame of the generated eye dynamic sequence; Predicting the remaining video frames frame by frame based on the time sequence until the image-to-video generation network predicts all video frames of the generated eye dynamic sequence; The image-to-video generation network is iteratively trained so that when the corresponding loss function determines that the difference between the generated eye dynamic sequence and the target modality eye structure diagnosis video serving as the gold standard is less than an error threshold, the trained image-to-video generation network is determined as the second model.

7. The method for training a cross-modal eye dynamic sequence generation model according to claim 6, wherein: Inputting the key frame into the image-to-video generation network for prediction processing to obtain the first frame of the generated eye dynamic sequence, including: Calculating a motion vector of pixels between two adjacent video frames in the target modality eye structure diagnosis video as a feature label; Inputting feature labels corresponding to the key frame and the first frame into the image-to-video generation network to perform distortion prediction on the key frame based on the feature labels and outputting the first frame of the eye dynamic sequence.

8. The method for training a cross-modal eye dynamic sequence generation model according to claim 6, wherein: Also includes: Based on bidirectional optical flow calculation, a consistency check is performed on adjacent frames in the eye dynamic sequence to align the adjacent frames.

9. The method for training a cross-modal eye dynamic sequence generation model according to claim 4 or claim 6, wherein: Also includes: Calculating the difference between the first frame and the last frame in the target modality eye structure diagnosis video based on the temporal consistency constraint mechanism; The difference is thresholded to obtain a corresponding clinical knowledge supervision mask.

10. The method for training a cross-modal eye dynamic sequence generation model according to claim 9, wherein: Also includes: The training of the first model and the training of the second model are supervised using the clinical knowledge supervision mask.

11. The method for training a cross-modal eye dynamic sequence generation model according to claim 10, wherein: Supervising the training of the first model and the training of the second model using the clinical knowledge supervision mask includes: In the training of the first model, a knowledge-enhanced attention mechanism is used to guide attention to a first region with significant temporal changes in the generated transformed frame; In the training of the second model, an attention mechanism is enhanced based on the knowledge to guide attention to a second region with significant temporal changes in the generated transition frame, so as to determine a key region based on the first region and / or the second region; performing temporal temporal consistency and change perception operations on the generated eye dynamic sequence based on the clinical knowledge supervision mask, wherein in the change perception operation, specific supervision is provided for the key area based on the knowledge-aware discriminator loss; and A mask-enhanced local normalized cross entropy loss is used to adjust pixel misalignment between the source modality eye image and the key area in the target modality eye structure diagnosis video.

12. The method for training a cross-modal eye dynamic sequence generation model according to any one of claims 1 to 8, wherein: The target modality eye structure diagnosis video includes fluorescein fundus angiography or indocyanine green angiography, and the corresponding eye image includes a fundus image; The eye structure diagnosis video of the target modality includes an optical coherence tomography video, and the corresponding eye image includes an eye plane scanning image.

13. The method for training a cross-modal eye dynamic sequence generation model according to any one of claims 1 to 8, wherein: The generative network includes at least one of a generative adversarial network, a derivative network of the generative adversarial network, a diffusion model, a derivative model of the diffusion model, a variational autoencoder, and a derivative structure of the variational autoencoder.

14. A method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, comprising: Inputting a static ophthalmic image of a source modality into an eye dynamic sequence generation model, wherein the eye dynamic sequence generation model includes a first model and a second model, wherein the first model and the second model are generated based on a generation network; Performing cross-modal conversion on the eye image based on the first model to generate a target modality keyframe; The target modality key frame is input into the second model to predict multiple video frames based on time sequence to obtain the multiple video frames, so as to generate the dynamic ophthalmic video of the target modality based on the multiple video frames.

15. A training device for a cross-modal eye dynamic sequence generation model, comprising: A processing module, configured to perform pixel-level alignment processing on the source modality eye image and the corresponding target modality eye structure diagnosis video; a selection module, configured to select a target modality key frame from the target modality eye structure diagnosis video after the alignment process, wherein the target modality key frame represents a modality feature of the target modality eye structure diagnosis video; A first training module is configured to train an image-to-image generative network using the aligned source modality eye image as a first initial domain and the corresponding target modality key frame as a first target domain to obtain a first model; The second training module is used to use the target modality key frame as the second initial domain and the aligned target modality eye structure diagnosis video as the second target domain to train the image-to-video generation network to obtain a second model for frame-by-frame prediction of the eye dynamic sequence, so as to obtain the eye dynamic sequence generation model based on the first model and the second model.

16. A device for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality, comprising: An input module, configured to input a static ophthalmic image of a source modality into an eye dynamic sequence generation model, wherein the eye dynamic sequence generation model includes a first model and a second model, wherein the first model and the second model are generated based on a generation network; a conversion module, configured to perform cross-modal conversion on the eye image based on the first model to generate a target modality keyframe; A prediction module is used to input the target modality key frame into the second model to predict multiple video frames based on time sequence to obtain the multiple video frames, so as to generate a dynamic ophthalmic video of the target modality based on the multiple video frames.

17. An electronic device comprising: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the method for training a cross-modal eye dynamic sequence generation model as described in any one of claims 1 to 13 or the method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality as described in claim 14 by executing the executable instructions.

18. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for training a cross-modal eye dynamic sequence generation model as described in any one of claims 1 to 13 or the method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality as described in claim 14 is implemented.

19. A computer program product having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the method for training a cross-modal eye dynamic sequence generation model as described in any one of claims 1 to 13 or the method for converting a static ophthalmic image of a source modality into a dynamic ophthalmic video of a target modality as described in claim 14.

Images