Image registration model training method and device, image processing method

Abstract

This disclosure provides an image registration model training method, relating to the field of image processing technology, comprising: processing fundus image samples to be registered using a deformation learning sub-model to obtain deformed image samples; obtaining key point information of the deformed image and key point information of a reference image; obtaining a first loss value based on the key point information of the deformed image and the reference image; adjusting the parameters of the deformation learning sub-model based on the first loss value; adjusting the inverse deformation learning sub-model using a second loss value; iteratively iterating the deformation learning sub-model until the first loss value satisfies a first convergence condition and the second loss value satisfies a second convergence condition, thereby obtaining a trained deformation learning sub-model; and determining the trained deformation learning sub-model as the image registration model. The image registration model obtained through this image registration model training method can perform registration for images of different modalities, achieving the goal of improving the accuracy of multimodal fundus image registration results.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, specifically relating to an image registration model training method and apparatus, and an image processing method. Background Technology

[0002] Fundus images are widely used to record and examine the clinical manifestations of various diseases, and fundus image registration is one of the fundamental methods of fundus image processing and analysis. Fundus image registration enables two or more fundus images to be spatially aligned, thereby assisting doctors in ophthalmic diagnosis and treatment. Therefore, fundus image registration has significant clinical application value.

[0003] Because single-modal fundus images provide limited information, clinical applications typically involve comprehensive analysis of fundus images from different modalities, such as conventional fundus camera images, fundus fluorescein angiography images, and optical coherence tomography angiography (OCTA) images. However, traditional image registration methods are not suitable for registering multimodal fundus images, resulting in low accuracy in the registration results and significantly limiting the development of fundus images in clinical applications. Summary of the Invention

[0004] In view of this, the present disclosure provides an image registration model training method and apparatus, an image processing method, a computer-readable storage medium and an electronic device to solve the problem that traditional fundus image registration methods cannot adapt to multimodal fundus image registration, resulting in low accuracy of multimodal fundus image registration results.

[0005] In a first aspect, one embodiment of this disclosure provides an image registration model training method, comprising: using a deformation learning sub-model, processing the fundus image sample to be registered based on a fundus image sample to be registered and a reference image sample to obtain a deformation image sample, wherein the modalities of the fundus image sample to be registered and the modalities of the reference image sample are different; using a first feature point extraction sub-model, extracting key points of the deformation image sample to obtain deformation image key point information; using the first feature point extraction sub-model, extracting key points of the reference image sample to obtain reference image key point information; based on the deformation image key point information and the reference image key point information, performing loss calculation using a first loss function to obtain a first loss value; adjusting the parameters of the deformation learning sub-model based on the first loss value; and using inverse deformation learning... The sub-model, based on the fundus image sample to be registered and the reference image sample, performs inverse deformation processing on the reference image sample to obtain an inversely deformed reference image sample; based on the inversely deformed reference sample and the fundus sample to be registered, a second loss value is obtained; using the second loss value, the parameters of the inverse deformation learning sub-model are adjusted, and the inverse deformation learning sub-model is iterated repeatedly; the deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thus obtaining a trained deformation learning sub-model, which is then determined as the image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and the reference image to obtain a deformed image, and the deformed image is determined as the registration image corresponding to the fundus image to be registered.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the deformation learning sub-model includes a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transformer layer, and a second transformer layer. The deformation processing includes first deformation processing and second deformation processing. Using the deformation learning sub-model, based on the fundus image sample to be registered and the reference image sample, the fundus image sample to be registered is processed to obtain a deformed image sample, including: using the first convolutional neural network layer, based on the fundus image sample to be registered and the reference image sample, to determine a first deformation field; using the second convolutional neural network layer... A convolutional neural network layer is used to extract local features of the fundus image sample to be registered. A third convolutional neural network layer is used to perform a first deformation processing on the local features of the fundus image sample to be registered based on a first deformation field to determine the deformation result of the local features of the fundus image sample to be registered. A first transformer layer is used to extract global features of the fundus image sample to be registered based on the fundus image sample to be registered. A second transformer layer is used to perform a second deformation processing based on the first deformation field, the deformation result of the local features of the fundus image sample to be registered, and the global features of the fundus image sample to be registered to obtain a deformed image sample.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, a second loss value is determined based on the inverse deformation reference sample and the fundus sample to be registered, including: using a second feature point extraction sub-model to extract key points of the inverse deformation reference image sample and obtain key point information of the inverse deformation reference image; using the second feature point extraction sub-model to extract key points of the fundus image sample to be registered and obtain key point information of the fundus image to be registered; and using a second loss function to calculate the loss based on the key point information of the inverse deformation reference image and the key point information of the fundus image to be registered, and obtain the second loss value.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: utilizing an inverse deformation learning sub-model, based on deformed image samples and fundus image samples to be registered, and performing inverse deformation processing on the deformed image samples to obtain inverse deformation image samples; based on the inverse deformation image samples and fundus image samples to be registered, using a third loss function to calculate loss and obtain a third loss value; adjusting the parameters of the deformation learning sub-model based on the third loss value; iterating the deformation learning sub-model cyclically until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, to obtain a trained deformation learning sub-model, including: iterating the deformation learning sub-model cyclically until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, to obtain a trained deformation learning sub-model.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: using a deformation learning sub-model, deforming the inverse deformation reference image sample based on the inverse deformation reference image sample and the reference image sample to obtain a deformation reference image sample; calculating the loss based on the deformation reference image sample and the reference image sample using a fourth loss function to obtain a fourth loss value; adjusting the parameters of the deformation learning sub-model based on the fourth loss value; iterating the deformation learning sub-model cyclically until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, to obtain a trained deformation learning sub-model, including: iterating the deformation learning sub-model cyclically until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, to obtain a trained deformation learning sub-model.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, before processing the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using the deformation learning sub-model to obtain the deformed image sample, the method further includes: cropping the initial reference image sample based on the initial reference image sample and the fundus image sample to be registered using the region localization sub-model to obtain the reference image sample, wherein the field of view of the initial reference image sample is inconsistent with the field of view of the fundus image sample to be registered, and the fundus image region included in the reference image sample corresponds to the fundus image region included in the fundus image sample to be registered.

[0011] Secondly, one embodiment of this disclosure provides an image processing method, comprising: acquiring multiple fundus images of different modalities; determining one fundus image from the multiple fundus images of different modalities as a reference image, and determining at least one fundus image to be registered from the multiple fundus images of different modalities excluding the reference image; registering the at least one fundus image to be registered using an image registration model based on the reference image and the at least one fundus image to be registered, thereby obtaining a registered image for each of the at least one fundus image to be registered, wherein the image registration model is determined based on the image registration model training method mentioned in the first aspect above.

[0012] Thirdly, one embodiment of this disclosure provides an image registration model training apparatus, comprising: a deformation module, used to process the fundus image sample to be registered based on a fundus image sample to be registered and a reference image sample using a deformation learning sub-model, to obtain a deformed image sample, wherein the modalities of the fundus image sample to be registered and the modalities of the reference image sample are different; a first feature extraction module, used to extract key points of the deformed image sample using a first feature point extraction sub-model, to obtain deformed image key point information; the first feature extraction module is further used to extract key points of the reference image sample using the first feature point extraction sub-model, to obtain reference image key point information; a loss calculation module, used to perform loss calculation based on the deformed image key point information and the reference image key point information using a first loss function, to obtain a first loss value; and an adjustment module, used to adjust the parameters of the deformation learning sub-model based on the first loss value; and an inverse deformation module. The image registration module is used to perform inverse deformation processing on the reference image sample based on the fundus image sample to be registered and the reference image sample using an inverse deformation learning sub-model to obtain an inverse deformation reference image sample. The loss calculation module is also used to obtain a second loss value based on the inverse deformation reference sample and the fundus image sample to be registered. The adjustment module is also used to adjust the parameters of the inverse deformation learning sub-model using the second loss value and iterate the inverse deformation learning sub-model cyclically. The determination module is used to iterate the deformation learning sub-model cyclically until the first loss value meets the first convergence condition and the second loss value meets the second convergence condition to obtain a trained deformation learning sub-model, and determine the trained deformation learning sub-model as the image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and the reference image to obtain a deformed image, and determine the deformed image as the registration image corresponding to the fundus image to be registered.

[0013] Fourthly, one embodiment of this disclosure provides an image processing apparatus, comprising: an acquisition module for acquiring multiple fundus images of different modalities; an image determination module for determining one fundus image from the multiple fundus images of different modalities as a reference image, and determining the fundus images other than the reference image from the multiple fundus images of different modalities as at least one fundus image to be registered; and a registration module for registering the at least one fundus image to be registered based on the reference image and the at least one fundus image to be registered using an image registration model to obtain a registered image for each of the at least one fundus image to be registered, wherein the image registration model is determined based on the image registration model training method mentioned in the first aspect above.

[0014] Fifthly, one embodiment of this disclosure provides an electronic device, the electronic device including: a processor, and a memory for storing a processor-executable computer program, wherein the processor records and executes the computer program for performing the methods mentioned above.

[0015] In a sixth aspect, one embodiment of this disclosure provides a computer storage medium storing a computer program that, when loaded by a processor, is used to execute the methods mentioned above.

[0016] The image registration model training method provided in this disclosure uses a deformation learning sub-model and a feature point extraction sub-model to determine the deformation image samples and key point information of the deformation images. Combined with the key point information of the reference image samples, the deformation learning sub-model is trained, and the trained deformation learning sub-model is then used as the image registration model. Since the modalities of the image samples to be registered and the reference image samples are different during training, the trained image registration model can perform registration for images of different modalities, thereby achieving the purpose of registering multimodal fundus images and improving the accuracy of multimodal fundus image registration results. Furthermore, embodiments of this disclosure adjust the parameters of the inverse deformation learning sub-model and iterate the deformation learning sub-model until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, obtaining a trained deformation learning sub-model. This model can determine the accuracy of the deformation process by inversely deforming the reference image sample to the fundus image sample to be registered. The shorter the iteration process of the inverse deformation learning sub-model, the more accurate the iteration result of the inverse deformation learning sub-model. This results in a shorter iteration time for the deformation learning sub-model and a more accurate image registration result. In addition to shortening the training time, it can further improve the accuracy of the image registration result. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof.

[0018] Figure 1 The diagram shown is an application scenario illustration provided by an embodiment of this disclosure.

[0019] Figure 2 The diagram shown is a flowchart illustrating an image registration model training method provided in an embodiment of this disclosure.

[0020] Figure 3 The diagram shown is a flowchart illustrating how a deformation learning sub-model is used in an embodiment of this disclosure to process a fundus image sample to be registered and a reference image sample to obtain a deformed image sample.

[0021] Figure 4 The diagram shown is a flowchart illustrating the process of determining a second loss value based on an inverse deformation reference sample and a fundus sample to be registered, according to an embodiment of this disclosure.

[0022] Figure 5 The diagram shown is a flowchart illustrating an image registration model training method provided in another embodiment of this disclosure.

[0023] Figure 6 The diagram shown is a flowchart illustrating an image registration model training method provided in another embodiment of this disclosure.

[0024] Figure 7 The diagram shown is a structural schematic of an initial image registration model provided in an embodiment of this disclosure.

[0025] Figure 8 The diagram shown is a flowchart of an image processing method provided in an embodiment of this disclosure.

[0026] Figure 9 The diagram shown is a structural schematic of an image registration model training device provided in an embodiment of this disclosure.

[0027] Figure 10 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this disclosure.

[0028] Figure 11 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] The technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments.

[0030] Fundus images are widely used to record and examine the clinical manifestations of various diseases, such as retinal, optic nerve, glaucoma, macular, and other eye diseases, as well as systemic diseases like diabetes, hypertension, and cardiovascular diseases. Fundus image registration is one of the fundamental methods of fundus image processing and analysis. It enables the spatial alignment of two or more fundus images taken at different times and in different fields of view, thereby assisting doctors in ophthalmic diagnosis and treatment. Furthermore, doctors can analyze fundus images taken at different times to provide more comprehensive information for clinical fundus diseases. Therefore, fundus image registration has significant clinical application value, for example, in assisting doctors in surgical planning, treatment planning, pathological monitoring, comprehensive evaluation of treatment effects, and research on pathogenesis.

[0031] Because single-modal fundus images provide limited information, clinical applications typically involve comprehensive analysis of fundus images from different modalities, such as conventional fundus cameras, fundus fluorescein angiography (GFA), and OCTA, to obtain more complete disease information and facilitate in-depth disease research. This also provides more comprehensive support for physicians in developing treatment plans. However, traditional image registration methods are not suitable for registering multimodal fundus images, resulting in low registration accuracy and significantly limiting the development of fundus image applications in clinical practice.

[0032] The following is combined with Figure 1 A brief introduction will be given to an application scenario of one embodiment of this disclosure.

[0033] Figure 1 The diagram shown illustrates an application scenario provided by an embodiment of this disclosure. Figure 1 As shown, the scenarios applicable to the embodiments of this disclosure include server 110 and image receiving device 120 that is communicatively connected to server 110.

[0034] Specifically, the image receiving device 120 receives a fundus image sample to be registered and a reference image sample. The modalities of the fundus image sample to be registered and the reference image sample are different. The server 110 processes the fundus image sample to be registered using a deformation learning sub-model to obtain a deformation image sample; extracts key points of the deformation image sample using a first feature point extraction sub-model to obtain key point information of the deformation image; extracts key points of the reference image sample using the first feature point extraction sub-model to obtain key point information of the reference image; calculates a loss using a first loss function based on the key point information of the deformation image and the key point information of the reference image to obtain a first loss value; adjusts the parameters of the deformation learning sub-model based on the first loss value; and uses an inverse deformation learning sub-model to... Given a fundus image sample to be registered and a reference image sample, the reference image sample undergoes inverse deformation processing to obtain an inversely deformed reference image sample. Based on the inversely deformed reference sample and the fundus image to be registered, a second loss value is determined. Using the second loss value, the parameters of the inverse deformation learning sub-model are adjusted, and the inverse deformation learning sub-model is iterated repeatedly. The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thus obtaining a trained deformation learning sub-model. This trained deformation learning sub-model is then identified as the image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and the reference image, obtaining a deformed image, which is then identified as the registration image corresponding to the fundus image to be registered. In other words, this scenario implements an image registration model training method.

[0035] Alternatively, the image receiving device 120 is used to acquire multiple fundus images of different modalities, and the server 110 is used to determine one fundus image from the multiple fundus images of different modalities as a reference image, and to determine at least one fundus image to be registered from the multiple fundus images of different modalities excluding the reference image; based on the reference image and at least one fundus image to be registered, an image registration model is used to register the at least one fundus image to be registered, resulting in a registered image for each of the at least one fundus image to be registered. That is, this scenario implements an image processing method. The image registration model mentioned in this scenario can be the image registration model generated in the above scenario, which deforms the fundus image to be registered based on the fundus image to be registered and the reference image, obtains a deformed image, and determines the deformed image as the registered image corresponding to the fundus image to be registered. Since... Figure 1 The above scenario illustrates that the image registration model training method and / or image processing method are implemented using server 110, thereby improving the adaptability of the scenario.

[0036] For example, the fundus image samples to be registered, the reference image samples, and multiple fundus images of different modalities mentioned above include, but are not limited to, images from ordinary fundus cameras, wide-angle fundus cameras, fundus fluorescein angiography images, and OCTA images.

[0037] Figure 2 The diagram shown is a flowchart illustrating an image registration model training method according to an embodiment of this disclosure. Figure 2 As shown, the image registration model training method provided in this embodiment includes the following steps.

[0038] Step S210: Using the deformation learning sub-model, the fundus image sample to be registered is processed based on the fundus image sample to be registered and the reference image sample to obtain the deformation image sample.

[0039] The modalities of the fundus image sample to be registered are different from those of the reference image sample.

[0040] For example, the fundus image sample to be registered is an OCTA image, and the reference image is a regular fundus camera image (i.e., a color fundus image). Using a deformation learning sub-model, the OCTA image is corrected to the deformation field of the regular fundus camera image to obtain a deformed image sample. The deformation field is the displacement field of pixels between the fundus image to be registered and the deformed image, expressed as a function of the change from the fundus image to the reference image. It should be understood that in this embodiment, the fundus image sample to be registered can be an OCTA image or a regular camera image. Correspondingly, the reference image can be a regular fundus camera image or an OCTA image. The fundus image sample to be registered and the reference image sample can have different modalities. The fundus image sample to be registered and the reference image sample include, but are not limited to, images from regular fundus cameras, wide-angle fundus cameras, fundus fluorescein angiography images, and OCTA images. For example, the deformation learning sub-model can be a convolutional neural network model or a deep learning network model.

[0041] Step S220: Using the first feature point extraction sub-model, extract the key points of the deformed image sample to obtain the key point information of the deformed image.

[0042] For example, the deformed image sample obtained above is used to extract feature points from the deformed image sample using a first feature extraction sub-model to obtain key point information of the deformed image. For example, the first feature point extraction sub-model can be a trained feature point extraction model, a deep learning network model, or other network models capable of feature point extraction. This disclosure does not further limit the type of the first feature point extraction sub-model.

[0043] Step S230: Use the first feature point extraction sub-model to extract key points of the reference image sample and obtain key point information of the reference image.

[0044] For example, feature points are extracted from a reference image sample that is used to register the fundus image sample. The reference image sample can be selected as needed.

[0045] Step S240: Based on the key point information of the deformed image and the key point information of the reference image, the first loss function is used to calculate the loss and obtain the first loss value.

[0046] For example, the first loss function can be a loss function between keypoint information, or it can be selected according to requirements. The keypoint information of the deformed image is matched with the keypoint information of the reference image, and the distance between the keypoints in the deformed image and the keypoints in the reference image is determined. This distance is used as the loss function; that is, a distance calculation function is selected as the loss function. It should be understood that any distance calculation function that can achieve the distance between keypoints can be selected as the first loss function, such as the Euclidean distance function, Mahalanobis distance function, cosine similarity function, or Hellinger distance function, etc.

[0047] In some embodiments, the deformation learning sub-model and the first feature point extraction sub-model are embedded in a recurrent adversarial network (RON). The first loss function can be the sum of the GAN Loss loss function of the adversarial network and the loss function of the keypoint information. It should be understood that when the deformation learning sub-model and the first feature point extraction sub-model are embedded in the RAN, the deformation learning sub-model acts as the adversarial network generator, and the first feature point extraction sub-model acts as the adversarial network discriminator. The GAN Loss loss function is obtained based on the generator and the discriminator.

[0048] Step S250: Adjust the parameters of the deformation learning sub-model based on the first loss value.

[0049] Step S260: Using the inverse deformation learning sub-model, based on the fundus image sample to be registered and the reference image sample, the reference image sample is subjected to inverse deformation processing to obtain the inverse deformation reference image sample.

[0050] For example, using an inverse deformation learning sub-model, the reference image sample is deformed according to the deformation field of the fundus image sample to be registered, thereby obtaining an inverse deformation reference image sample. It should be understood that the structure of the inverse deformation learning sub-model in this embodiment can be the same as the deformation learning sub-model described above, and the inverse deformation process of the inverse deformation learning sub-model is the reverse of the deformation process of the deformation learning sub-model.

[0051] Step S270: Determine the second loss value based on the inverse deformation reference sample and the fundus sample to be registered.

[0052] For example, a second loss value is determined based on the distance between the inverse deformation reference sample and the fundus sample to be registered.

[0053] In some embodiments, the specific implementation of step S270 is as follows: Figure 4 As shown, it will not be elaborated further here.

[0054] Step S280: Using the second loss value, adjust the parameters of the inverse deformation learning sub-model and iterate the inverse deformation learning sub-model.

[0055] Step S290: Iterate the deformation learning sub-model until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, obtain the trained deformation learning sub-model, and determine the trained deformation learning sub-model as the image registration model.

[0056] The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and the reference image to obtain the deformed image, and then determine the deformed image as the registration image corresponding to the fundus image to be registered.

[0057] For example, the first convergence condition can be set according to requirements. When the deformation learning sub-model and the first feature point extraction sub-model are embedded into the adversarial network, the first loss value can be achieved by using the gradient descent algorithm to satisfy the first convergence condition. It should be understood that during training, the iteration method and the first convergence condition can be selected according to requirements.

[0058] For example, the second convergence condition can be selected as needed. Iterating the deformation learning sub-model repeatedly until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition allows the inverse deformation process to supervise the forward deformation process. The inverse deformation learning sub-model supervises the deformation learning sub-model during training, thereby further improving the processing accuracy of the trained deformation learner.

[0059] The image registration model training method provided in this disclosure determines deformed image samples and key point information of deformed images through a deformation learning sub-model and a feature point extraction sub-model. It then combines the key point information of a reference image sample to train the deformation learning sub-model, which is then used as the image registration model. Since the modalities of the fundus image sample to be registered and the reference image sample are different, the method achieves the registration of multimodal fundus images, improving the accuracy of the multimodal fundus image registration results. Furthermore, the deformation learning sub-model and the first feature point extraction sub-model in this disclosure can be embedded in a recurrent adversarial network for training, reducing runtime. In application, the deformation result of the trained deformation learning sub-model on the fundus image to be registered is closer to the reference image sample, further improving the accuracy of the multimodal fundus image registration results. Furthermore, this embodiment of the disclosure adjusts the parameters of the inverse deformation learning sub-model and iterates the deformation learning sub-model repeatedly until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thus obtaining a trained deformation learning sub-model. This sub-model can determine the accuracy of the deformation process by inversely deforming a reference image sample to the deformation field of the fundus image sample to be registered. The higher the accuracy of the deformation process, the shorter the iteration process of the inverse deformation learning sub-model, and the more accurate the iteration result of the inverse deformation learning sub-model. This results in a shorter iteration time for the deformation learning sub-model and a more accurate image registration result. While shortening the training time, this further improves the accuracy of the image registration result.

[0060] Figure 3 The diagram illustrates a process of using a deformation learning sub-model, based on a fundus image sample to be registered and a reference image sample, to process the fundus image sample to be registered and obtain a deformed image sample, according to an embodiment of this disclosure. Figure 3 As shown, an embodiment of this disclosure provides a deformation learning sub-model that processes the fundus image sample to be registered based on the fundus image sample to be registered and a reference image sample to obtain a deformed image sample, including the following steps.

[0061] Step S310: Using the first convolutional neural network layer, the first deformation field is determined based on the fundus image sample to be registered and the reference image sample.

[0062] The deformation learning sub-model includes a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transformer layer, and a second transformer layer.

[0063] For example, the first convolutional neural network layer is used to determine the first deformation field of the fundus image sample to be registered according to the deformation field of the reference image sample.

[0064] Step S320: Use the second convolutional neural network layer to extract local features of the fundus image sample to be registered.

[0065] For example, by using a second convolutional neural network layer, local features in the image are captured through convolutional pooling, and local features of the fundus image sample to be registered are extracted.

[0066] Step S330: Using the third convolutional neural network layer, perform first deformation processing on the local features of the fundus image sample to be registered based on the first deformation field to determine the local feature deformation result of the fundus image sample to be registered.

[0067] The deformation process includes a first deformation process and a second deformation process.

[0068] Step S340: Using the first transformer layer, extract the global features of the fundus image samples to be registered based on the fundus image samples to be registered.

[0069] For example, the first transformer layer is able to acquire long-range dependencies and local features in the fundus image sample to be registered.

[0070] Step S350: Using the second transformer layer, based on the first deformation field, the local feature deformation result of the fundus image sample to be registered, and the global features of the fundus image sample to be registered, a second deformation processing is performed to obtain a deformed image sample.

[0071] For example, the second transformer layer can perform a second deformation process by combining local and global features to obtain deformed image samples.

[0072] In some embodiments, the deformation learning sub-model employs a three-layer downsampling and three-layer upsampling structure. Long-range dependencies and global features in the fundus image samples (and / or reference image samples) to be registered are captured through convolutional kernel pooling, and long-range dependencies and global features are captured through Transformer layers. The deformation field from the fundus image to the reference image is learned by combining global and local features.

[0073] The model training method provided in this embodiment of the present disclosure, since the deformation learning sub-model includes convolutional layers (i.e., the first convolutional neural network layer, the second convolutional neural network layer and the third convolutional neural network layer) and transformer layers (i.e. the first transformer layer and the second transformer layer), can obtain deformation image samples by combining global features and local features, which improves the processing of local small deformations and further improves the accuracy of image registration results obtained by using the image registration model.

[0074] Figure 4 The diagram illustrates a process for obtaining a second loss value based on an inverse deformation reference sample and a fundus sample to be registered, according to an embodiment of this disclosure. Figure 4 As shown in the embodiments of this disclosure, the determination of the second loss value based on the inverse deformation reference sample and the fundus sample to be registered includes the following steps.

[0075] Step S410: Using the second feature point extraction sub-model, extract the key points of the inverse deformation reference image sample to obtain the key point information of the inverse deformation reference image.

[0076] For example, the second feature point extraction sub-model can be the same as the first feature point extraction sub-model described above, or it can be another feature point extraction model, as long as it can achieve the purpose of extracting key points of the inverse deformation reference image sample. This embodiment of the disclosure does not further limit the structure of the second feature point extraction sub-model.

[0077] Step S420: Using the second feature point extraction sub-model, extract the key points of the fundus image sample to be registered, and obtain the key point information of the fundus image to be registered.

[0078] For example, the inverse deformation reference image sample and the fundus image sample to be registered can be simultaneously input into the second feature point extraction sub-model. Correspondingly, steps S420 and S430 can be performed simultaneously or in an adjusted order.

[0079] Step S430: Based on the key point information of the inverse deformation reference image and the key point information of the fundus image to be registered, the second loss function is used to calculate the loss and obtain the second loss value.

[0080] For example, the second loss function is the same as the first loss function, both being distance functions that calculate the distance between keypoints. The second loss function can be the same as the first loss function, or it can be other distance functions selected according to requirements, such as Euclidean distance function, Mahalanobis distance function, cosine similarity function, or Hellinger distance function, etc.

[0081] This embodiment of the disclosure obtains key point information of the inverse deformation reference image and key point information of the fundus image to be registered through a feature point extraction model. Based on these key point information, a second loss function is used to calculate the loss and obtain a second loss value. The second loss value has higher accuracy, which helps to make the iterative results of the inverse deformation learning sub-model more accurate during training, thereby improving the accuracy of the deformation process and further enhancing the accuracy of the image registration results.

[0082] Figure 5 The diagram shown is a flowchart illustrating an image registration model training method according to another embodiment of this disclosure. Figure 4 This disclosure extends from the embodiments shown. Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 4 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0083] like Figure 5 As shown, another embodiment of the image registration model training method provided in this disclosure further includes the following steps.

[0084] Step S510: Using the inverse deformation learning sub-model, based on the deformed image samples and the fundus image samples to be registered, the deformed image samples are subjected to inverse deformation processing to obtain inverse deformation image samples.

[0085] For example, using the inverse deformation learning sub-model, the deformed image sample is subjected to inverse deformation processing according to the deformation field of the fundus image sample to be registered, so as to obtain the inverse deformation image sample.

[0086] Step S520: Based on the inverse deformation image samples and the fundus image samples to be registered, the third loss function is used to calculate the loss and obtain the third loss value.

[0087] For example, the third loss function can be the same as the first and second loss functions, or it can be another function that calculates the distance between feature points.

[0088] Step S530: Adjust the parameters of the deformation learning sub-model based on the third loss value.

[0089] The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition, and the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, thus obtaining a trained deformation learning sub-model.

[0090] For example, the third convergence condition can be selected as needed, and the deformation learning sub-model is iterated cyclically until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition. The resulting trained deformation learning sub-model and the adjusted inverse deformation learning sub-model can perform more accurate deformation on the deformed image samples and the fundus image samples to be registered, and can better supervise the deformation learning sub-model, thereby further improving the accuracy of the image registration model.

[0091] In some embodiments, the deformation learning sub-model, the inverse deformation learning sub-model, the first feature point extraction sub-model, and the second feature point extraction sub-model are embedded in the recurrent adversarial network, and the convergence condition can be determined by combining the loss function GAN Loss of the generator and the discriminator, the first loss function, the second function, and the third loss function.

[0092] This embodiment of the disclosure uses an inverse deformation learning sub-model to perform inverse deformation processing on deformed image samples. Since the deformed image samples are obtained by deforming the fundus image to be registered based on the deformation field of the reference image, the inverse deformation processing of the deformed image samples and the loss values ​​of the inversely deformed image samples and the fundus image samples to be registered are determined. The parameters of the deformation learning sub-model are then adjusted based on the loss values. The trained deformation learning sub-model can more accurately determine the deformation field of the fundus image to be registered and the reference image. Therefore, the image registration model training method provided by this embodiment of the disclosure can further improve the accuracy of the image registration model.

[0093] Figure 6 The diagram shown is a flowchart illustrating an image registration model training method according to another embodiment of this disclosure. Figure 5 This disclosure extends from the embodiments shown. Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 5 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0094] like Figure 6 As shown, another embodiment of the image registration model training method provided in this disclosure further includes the following steps.

[0095] Step S610: Using the deformation learning sub-model, based on the inverse deformation reference image sample and the reference image sample, the inverse deformation reference image sample is deformed to obtain the deformation reference image sample.

[0096] For example, by using a deformation learning sub-model, the inverse deformation reference image sample is deformed according to the reference image sample to obtain a deformation reference image sample.

[0097] Step S620: Based on the deformed reference image sample and the reference image sample, the fourth loss function is used to calculate the loss and obtain the fourth loss value.

[0098] For example, the fourth loss function can be the same as the first, second, and third loss functions, or it can be other functions that calculate the distance between feature points.

[0099] Step S630: Adjust the parameters of the deformation learning sub-model based on the fourth loss value.

[0100] The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, thereby obtaining a trained deformation learning sub-model. This includes iterating repeatedly until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, thereby obtaining a trained deformation learning sub-model.

[0101] For example, the fourth convergence condition can be selected as needed, and the deformation learning sub-model is iterated cyclically until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, thereby obtaining a trained deformation learning sub-model. Therefore, the trained deformation learning sub-model can improve the accuracy of the inverse deformation reference image sample according to the deformation of the reference image sample, thereby further improving the accuracy of the image registration model.

[0102] This disclosure improves the accuracy of the image registration model by enhancing the accuracy of the inverse deformation reference image sample deformation according to the reference image sample.

[0103] In some embodiments, before processing the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using the deformation learning sub-model to obtain the deformed image sample, the image registration model training method further includes: cropping the initial reference image sample based on the initial reference image sample and the fundus image sample to be registered using a region localization sub-model to obtain a reference image sample, wherein the field of view of the initial reference image sample is inconsistent with the field of view of the fundus image sample to be registered, and the fundus image region included in the reference image sample corresponds to the fundus image region included in the fundus image sample to be registered. For example, when the field of view of the initial reference image sample is larger than that of the fundus image sample to be registered, the initial reference sample is cropped according to the field of view of the fundus image sample to be registered using the region localization sub-model to obtain the reference sample. It should be understood that the purpose of processing using the region localization sub-model is to ensure that the field of view of the reference image sample and the fundus image sample to be registered are consistent. Therefore, when the field of view of the initial reference image sample is smaller than that of the fundus image to be registered, the initial reference image is determined as the reference image. A region localization sub-model is used to crop the fundus image sample to be registered, resulting in a processed fundus image to be registered with the same field of view as the reference image. This embodiment of the present disclosure uses a region localization sub-model to ensure that the field of view of the reference image sample is the same as that of the fundus image sample to be registered, reducing the computational load during model training and improving the model's processing speed.

[0104] Figure 7The diagram shown is a structural schematic of an initial image registration model provided in an embodiment of this disclosure. Figure 7 As shown, an embodiment of this disclosure provides an initial image registration model structure including: a region localization submodel 710, a deformation learning submodel 720, a first feature point extraction submodel 730, an inverse deformation learning submodel 740, and a second feature point extraction submodel 750. The initial image registration model mentioned in this embodiment utilizes the aforementioned image registration model training method to train the initial image registration model, thereby obtaining the final image registration model. Exemplarily, during training, the fundus image sample to be registered and the selected initial reference image sample enter the region localization submodel 710 to obtain a reference image sample. The reference image sample and the fundus image sample to be registered undergo deformation processing in the deformation learning submodel 720 to obtain a deformed image sample. Using the first feature extraction submodel 730, key point information of the deformed image and key point information of the reference image are obtained, and then a first loss value is calculated. The inverse deformation learning submodel 740 is used to process the reference image sample and the fundus image sample to be registered to obtain an inversely deformed reference image sample and calculate a second loss value. The deformation learning sub-model 720 and the inverse deformation learning sub-model 740 are adjusted based on the first and second loss values ​​until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition. The trained deformation learning sub-model 720 is then determined as the image registration model. It should be understood that during actual training, the deformation learning sub-model and the inverse deformation learning sub-model can also be used to obtain deformation reference image samples and inverse deformation image samples, and to calculate the third and fourth loss values. The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, ultimately obtaining the trained deformation learning sub-model.

[0105] Figure 8 The diagram shown is a schematic flowchart of an image processing method provided in an embodiment of this disclosure. Figure 8 As shown, the image processing method provided in this embodiment includes the following steps.

[0106] Step S810: Acquire multiple fundus images of different modalities.

[0107] For example, fundus images of different modalities can be OCTA images and fundus fluorescence angiography fundus images, OCTA images and ordinary fundus images, etc.

[0108] Step S820: One fundus image from multiple fundus images of different modalities is determined as a reference image, and the fundus images other than the reference image from multiple fundus images of different modalities are determined as at least one fundus image to be registered.

[0109] For example, one fundus image selected from OCTA images, fundus fluorescein angiography images, and ordinary fundus images is determined as the reference image. At least one fundus image to be registered is selected from multiple fundus images of different modalities, excluding the reference image.

[0110] Step S830: Based on the reference image and at least one fundus image to be registered, the at least one fundus image to be registered is registered using an image registration model to obtain the registration images of each of the at least one fundus image to be registered.

[0111] The image registration model is determined based on the image registration model training method mentioned above.

[0112] For example, by using an image registration model, at least one fundus image to be registered is registered to obtain a registration image for each of the fundus images to be registered. The registration image is obtained by deforming the fundus image to be registered according to the reference image.

[0113] The image processing method of this disclosure, based on multiple fundus images of different modalities, uses an image registration model to obtain a registration image for each of at least one fundus image to be registered. Since the image registration model is determined using the aforementioned image registration model training method, the obtained registration image results are more accurate, providing more comprehensive assistance to doctors in formulating treatment plans and advancing related research.

[0114] Figure 9 The diagram shown is a structural schematic of an image registration model training device provided in an embodiment of this disclosure. Figure 9As shown, the image registration model training device 900 provided in the embodiments of this disclosure includes: a deformation module 910, a first feature extraction module 920, a loss calculation module 930, an adjustment module 940, an inverse deformation processing module 950, and a determination module 960. Specifically, the deformation module 910 is used to process the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using a deformation learning sub-model to obtain a deformed image sample, wherein the modality of the fundus image sample to be registered is different from that of the reference image sample; the first feature extraction module 920 is used to extract key points of the deformed image sample using a first feature point extraction sub-model to obtain key point information of the deformed image; the first feature extraction module 920 is also used to extract key points of the reference image sample using the first feature point extraction sub-model to obtain key point information of the reference image; the loss calculation module 930 is used to perform loss calculation based on the key point information of the deformed image and the key point information of the reference image using a first loss function to obtain a first loss value; the adjustment module 940 is used to adjust the parameters of the deformation learning sub-model based on the first loss value; the inverse deformation processing module 950 is used to... Using an inverse deformation learning sub-model, the reference image sample is inversely deformed based on the fundus image sample to be registered and the reference image sample to obtain an inverse deformation reference image sample. The loss calculation module 930 is also used to obtain a second loss value based on the inverse deformation reference sample and the fundus image sample to be registered. The adjustment module 940 is also used to adjust the parameters of the inverse deformation learning sub-model using the second loss value and iterate the inverse deformation learning sub-model cyclically. The determination module 960 is used to iterate the deformation learning sub-model cyclically until the first loss value meets the first convergence condition and the second loss value meets the second convergence condition to obtain the trained deformation learning sub-model. The trained deformation learning sub-model is determined as the image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and the reference image to obtain a deformed image. The deformed image is determined as the registration image corresponding to the fundus image to be registered.

[0115] In some embodiments, the deformation learning sub-model includes a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transformer layer, and a second transformer layer. The deformation module 910 is further configured to use the deformation learning sub-model to process the fundus image samples to be registered based on the fundus image samples to be registered and the reference image samples to obtain deformed image samples, including: using the first convolutional neural network layer to determine a first deformation field based on the fundus image samples to be registered and the reference image samples; and using the second convolutional neural network layer to extract the deformation field to be registered. The local features of the fundus image sample to be registered are obtained. Using the third convolutional neural network layer, the local features of the fundus image sample to be registered are subjected to the first deformation processing based on the first deformation field to determine the deformation result of the local features of the fundus image sample to be registered. Using the first transformer layer, the global features of the fundus image sample to be registered are extracted based on the fundus image sample to be registered. Using the second transformer layer, the second deformation processing is performed based on the first deformation field, the deformation result of the local features of the fundus image sample to be registered, and the global features of the fundus image sample to be registered to obtain the deformed image sample.

[0116] In some embodiments, the image registration model training device further includes a second feature point extraction module, and the loss calculation module 930 is further configured to: extract key points of the inverse deformation reference image sample using the second feature point extraction sub-model to obtain key point information of the inverse deformation reference image; extract key points of the fundus image sample to be registered using the second feature point extraction sub-model to obtain key point information of the fundus image to be registered; and perform loss calculation using a second loss function based on the key point information of the inverse deformation reference image and the key point information of the fundus image to be registered to obtain a second loss value.

[0117] In some embodiments, the inverse deformation processing module 950 is further configured to, using the inverse deformation learning sub-model, perform inverse deformation processing on the deformed image samples and the fundus image samples to be registered, to obtain inverse deformation image samples; the loss calculation module 930 is further configured to, based on the inverse deformation image samples and the fundus image samples to be registered, perform loss calculation using a third loss function to obtain a third loss value; the adjustment module 940 is further configured to, based on the third loss value, adjust the parameters of the deformation learning sub-model; and the determination module 960 is further configured to, iteratively iterate the deformation learning sub-model until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, to obtain a trained deformation learning sub-model.

[0118] In some embodiments, the deformation module 910 is further configured to, using the deformation learning sub-model, perform deformation processing on the inverse deformation reference image sample based on the inverse deformation reference image sample and the reference image sample to obtain a deformation reference image sample. The loss calculation module 930 is further configured to, based on the deformation reference image sample and the reference image sample, perform loss calculation using a fourth loss function to obtain a fourth loss value; the adjustment module 940 is further configured to, based on the fourth loss value, adjust the parameters of the deformation learning sub-model; the determination module 960 is further configured to, iteratively iterate the deformation learning sub-model until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, thereby obtaining a trained deformation learning sub-model.

[0119] In some embodiments, the image registration model training apparatus further includes a cropping module. Before processing the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using the deformation learning sub-model to obtain the deformed image sample, the cropping module is used to crop the initial reference image sample based on the initial reference image sample and the fundus image sample to be registered using the region localization sub-model to obtain the reference image sample. The field of view of the initial reference image sample is inconsistent with the field of view of the fundus image sample to be registered, and the fundus image region included in the reference image sample corresponds to the fundus image region included in the fundus image sample to be registered.

[0120] Figure 10 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this disclosure. Figure 10 As shown, an embodiment of the image processing apparatus 1000 provided by this disclosure includes an acquisition module 1010, an image determination module 1020, and a registration module 1030. Specifically, the acquisition module 1010 is used to acquire multiple fundus images of different modalities; the image determination module 1020 is used to determine one fundus image from the multiple fundus images of different modalities as a reference image, and to determine the fundus images other than the reference image from the multiple fundus images of different modalities as at least one fundus image to be registered; the registration module 1030 is used to register the at least one fundus image to be registered based on the reference image and the at least one fundus image to be registered using an image registration model, to obtain a registered image for each of the at least one fundus image to be registered, wherein the image registration model is determined based on the image registration model training method mentioned above.

[0121] Below, for reference Figure 11 To describe an electronic device according to embodiments of the present disclosure Figure 11 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure.

[0122] Figure 11The electronic device 1100 shown (which may specifically be a computer device) includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, processor 1102, and communication interface 1103 are interconnected via the bus 1104.

[0123] The memory 1101 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1101 may store a program, and when the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps in the image registration model training method or image processing method of the embodiments of this disclosure.

[0124] The processor 1102 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required to be performed by each unit in the image registration model training device or image processing device of the present disclosure embodiments.

[0125] The processor 1102 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image registration model training method and image processing method of this disclosure can be completed by the integrated logic circuits in the hardware of the processor 1102 or by instructions in software form. The processor 1102 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1101. The processor 1102 reads the information in the memory 1101 and, in conjunction with its hardware, performs the functions required by the units included in the image registration model training device or image processing device of the present disclosure, or executes the image registration model training method and image processing method of the present disclosure.

[0126] The communication interface 1103 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the electronic device 1100 and other devices or communication networks. For example, the communication interface 1103 can be used to acquire a sample of fundus image to be registered or a fundus image to be registered.

[0127] Bus 1104 may include a pathway for transmitting information between various components of electronic device 1100 (e.g., memory 1101, processor 1102, communication interface 1103).

[0128] It should be noted that, although Figure 11 The illustrated electronic device 1100 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 1100 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 1100 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device 1100 may only include the devices necessary for implementing the embodiments of this disclosure, and may not necessarily include... Figure 11 All the devices shown.

[0129] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0130] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0131] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0134] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. The computer-readable storage medium can be any combination of one or more readable media.

[0135] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for training an image registration model, characterized in that, include: Using a deformation learning sub-model, the fundus image sample to be registered is processed based on the fundus image sample to be registered and the reference image sample to obtain a deformed image sample. The modality of the fundus image sample to be registered is different from that of the reference image sample. Using the first feature point extraction sub-model, the key points of the deformed image sample are extracted to obtain the key point information of the deformed image; Using the first feature point extraction sub-model, key points of the reference image sample are extracted to obtain key point information of the reference image. Based on the key point information of the deformed image and the key point information of the reference image, a first loss function is used to calculate the loss and obtain a first loss value. Based on the first loss value, adjust the parameters of the deformation learning sub-model; Using the inverse deformation learning sub-model, based on the fundus image sample to be registered and the reference image sample, the reference image sample is subjected to inverse deformation processing to obtain the inverse deformation reference image sample; Based on the inverse deformation reference image sample and the fundus image sample to be registered, a second loss value is obtained; Using the second loss value, adjust the parameters of the inverse deformation learning sub-model and iterate the inverse deformation learning sub-model. The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thereby obtaining a trained deformation learning sub-model. The trained deformation learning sub-model is then determined as an image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and a reference image to obtain a deformed image. The deformed image is then determined as the registration image corresponding to the fundus image to be registered.

2. The method according to claim 1, characterized in that, The deformation learning sub-model includes a first convolutional neural network layer, a second convolutional neural network layer, a third convolutional neural network layer, a first transformer layer, and a second transformer layer. The deformation processing includes first deformation processing and second deformation processing. The step of using the deformation learning sub-model to process the fundus image sample to be registered, based on the sample and a reference image, to obtain a deformed image sample includes: Using the first convolutional neural network layer, a first deformation field is determined based on the fundus image sample to be registered and the reference image sample; Using the second convolutional neural network layer, local features of the fundus image sample to be registered are extracted; Using the third convolutional neural network layer, the local features of the fundus image sample to be registered are subjected to the first deformation processing based on the first deformation field to determine the local feature deformation result of the fundus image sample to be registered; Using the first transformer layer, global features of the fundus image sample to be registered are extracted based on the fundus image sample to be registered; Using the second transformer layer, based on the first deformation field, the local feature deformation result of the fundus image sample to be registered, and the global features of the fundus image sample to be registered, the second deformation processing is performed to obtain the deformed image sample.

3. The method according to claim 1, characterized in that, The step of obtaining a second loss value based on the inverse deformation reference image sample and the fundus image sample to be registered includes: Using the second feature point extraction sub-model, the key points of the inverse deformation reference image sample are extracted to obtain the key point information of the inverse deformation reference image. Using the second feature point extraction sub-model, key points of the fundus image sample to be registered are extracted to obtain key point information of the fundus image to be registered. Based on the key point information of the inverse deformation reference image and the key point information of the fundus image to be registered, a second loss function is used to calculate the loss and obtain the second loss value.

4. The method according to claim 3, characterized in that, Also includes: Using the inverse deformation learning sub-model, based on the deformed image sample and the fundus image sample to be registered, the deformed image sample is subjected to inverse deformation processing to obtain the inverse deformation image sample; Based on the inverse deformation image sample and the fundus image sample to be registered, a third loss function is used to calculate the loss and obtain the third loss value. Based on the third loss value, the parameters of the deformation learning sub-model are adjusted; The process of iterating the deformation learning sub-model until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thereby obtaining a trained deformation learning sub-model, includes: The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, thereby obtaining the trained deformation learning sub-model.

5. The method according to claim 4, characterized in that, Also includes: Using the deformation learning sub-model, the inverse deformation reference image sample is deformed based on the inverse deformation reference image sample and the reference image sample to obtain the deformation reference image sample; Based on the deformed reference image sample and the reference image sample, a fourth loss function is used to calculate the loss and obtain the fourth loss value. Based on the fourth loss value, the parameters of the deformation learning sub-model are adjusted; The process of iterating the deformation learning sub-model until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, and the third loss value satisfies the third convergence condition, to obtain the trained deformation learning sub-model, includes: The deformation learning sub-model is iterated repeatedly until the first loss value satisfies the first convergence condition, the second loss value satisfies the second convergence condition, the third loss value satisfies the third convergence condition, and the fourth loss value satisfies the fourth convergence condition, thereby obtaining the trained deformation learning sub-model.

6. The method according to any one of claims 1 to 5, characterized in that, Before processing the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using the deformation learning sub-model to obtain the deformed image sample, the method further includes: Based on the initial reference image sample and the fundus image sample to be registered, the initial reference image sample is cropped using a region localization sub-model to obtain the reference image sample. The field of view of the initial reference image sample is larger than that of the fundus image sample to be registered, and the field of view of the fundus image region included in the reference image sample is consistent with that of the fundus image region included in the fundus image sample to be registered.

7. An image processing method, characterized in that, include: Acquire multiple fundus images of different modalities; One fundus image from the multiple fundus images of different modalities is determined as a reference image, and the fundus images other than the reference image from the multiple fundus images of different modalities are determined as at least one fundus image to be registered; Based on the reference image and the at least one fundus image to be registered, the at least one fundus image to be registered is registered using an image registration model to obtain a registered image for each of the at least one fundus image to be registered, wherein the image registration model is determined based on the image registration model training method described in any one of 1 to 6.

8. An image registration model training device, characterized in that, include: The deformation module is used to process the fundus image sample to be registered based on the fundus image sample to be registered and the reference image sample using a deformation learning sub-model to obtain a deformed image sample, wherein the modality of the fundus image sample to be registered is different from that of the reference image sample. The first feature extraction module is used to extract key points of the deformed image sample by using the first feature point extraction sub-model, and obtain key point information of the deformed image; The first feature extraction module is also used to extract key points of the reference image sample using the first feature point extraction sub-model, and obtain reference image key point information. The loss calculation module is used to perform loss calculation based on the key point information of the deformed image and the key point information of the reference image, using a first loss function, to obtain a first loss value; An adjustment module is used to adjust the parameters of the deformation learning sub-model based on the first loss value; The inverse deformation processing module is used to perform inverse deformation processing on the reference image sample based on the fundus image sample to be registered and the reference image sample using the inverse deformation learning sub-model, so as to obtain the inverse deformation reference image sample. The loss calculation module is also used to obtain a second loss value based on the inverse deformation reference image sample and the fundus image sample to be registered; The adjustment module is further configured to adjust the parameters of the inverse deformation learning sub-model using the second loss value, and to iterate the inverse deformation learning sub-model cyclically. The determination module is used to iterate the deformation learning sub-model until the first loss value satisfies the first convergence condition and the second loss value satisfies the second convergence condition, thereby obtaining a trained deformation learning sub-model. The trained deformation learning sub-model is then determined as an image registration model. The image registration model is used to deform the fundus image to be registered based on the fundus image to be registered and a reference image to obtain a deformed image. The deformed image is then determined as the registration image corresponding to the fundus image to be registered.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1 to 7.

Images