Image registration method and apparatus, image processing method

Abstract

This disclosure discloses an image registration method and apparatus, as well as an image processing method, relating to the field of image processing technology. The image registration method includes: determining matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model comprising deformable convolutional layers and fully connected layers, based on the fundus image to be corrected and a reference fundus image; and fusing the fundus image to be corrected and the reference fundus image based on the matching relationship data to obtain a registered image corresponding to the fundus image to be corrected. Because deformable convolution can capture minute deformations between the fundus image to be corrected and the reference fundus image through their respective feature points, it reduces interference from differences in resolution, size, and geometric distortion between the fundus image to be corrected and the reference fundus image, as well as interference from occlusion and blurring in the image to be corrected, thereby improving the accuracy of image registration.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image registration method and apparatus, and an image processing method. Background Technology

[0002] Image registration enables multiple fundus images to achieve a complete geometric correspondence in the spatial domain. By registering fundus images of a patient at different times, doctors can determine changes in the patient's fundus and understand the progression of the disease. Traditional image registration methods include grayscale-based registration methods, edge-based registration methods, and contour-based registration methods.

[0003] However, traditional registration methods require linear illumination changes and rigid body transformations, or contrast within a certain range, in fundus images to achieve accurate registration. But because lesions in fundus images often have unique characteristics, fundus images may suffer from interference such as occlusion, geometric distortion, blurring, and severe nonlinear illumination changes, leading to reduced accuracy of traditional registration methods. Therefore, there is an urgent need for an image registration method to address the low accuracy of traditional methods when registering fundus images. Summary of the Invention

[0004] In view of this, the present disclosure provides an image registration method and apparatus, and an image processing method, to solve the problem of low accuracy of traditional registration methods when registering fundus images.

[0005] In a first aspect, an embodiment of this disclosure provides an image registration method, comprising: determining matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers, based on a fundus image to be corrected and a reference fundus image, wherein the deformable convolutional layer is used to extract feature points of the fundus image to be corrected and the reference fundus image respectively, and the fully connected layer is used to generate matching relationship data between the fundus image to be corrected and the reference fundus image based on the feature points of the fundus image to be corrected and the reference fundus image respectively; and fusing the fundus image to be corrected and the reference fundus image based on the matching relationship data to obtain a registered image corresponding to the fundus image to be corrected.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, based on the fundus image to be rectified and the reference fundus image, a deep learning model including deformable convolutional layers and fully connected layers is used to determine the matching relationship data between the fundus image to be rectified and the reference fundus image. This includes: using deformable convolutional layers to extract features from the fundus image to be rectified and the reference fundus image respectively, and determining the feature points of the fundus image to be rectified and the reference fundus image respectively; using fully connected layers to perform homography mapping transformation on the feature points of the fundus image to be rectified and the reference fundus image respectively, determining the homography matrix, and determining the homography matrix as the matching relationship data between the fundus image to be rectified and the reference fundus image.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, before using the deformable convolutional layer to extract features from the fundus image to be corrected and the reference fundus image respectively, and determining the feature points of each of the fundus image to be corrected and the reference fundus image, the image registration method further includes: acquiring the original fundus image to be corrected and the original reference fundus image; performing vascular extraction processing on the original fundus image to be corrected and the original reference fundus image respectively, and determining the fundus image to be corrected and the reference fundus image, wherein both the fundus image to be corrected and the reference fundus image include vascular image regions; wherein using the deformable convolutional layer to extract features from the fundus image to be corrected and the reference fundus image respectively, and determining the feature points of each of the fundus image to be corrected and the reference fundus image, includes: using the deformable convolutional layer to extract features from the vascular image regions in the fundus image to be corrected and the vascular image regions in the reference fundus image respectively, and determining the feature points of each of the fundus image to be corrected and the reference fundus image.

[0008] In conjunction with the first aspect, in certain implementations of the first aspect, blood vessel extraction processing is performed on the original fundus image to be corrected and the original reference fundus image to determine the fundus image to be corrected and the reference fundus image, including: performing blood vessel extraction on the original fundus image to be corrected and the original reference fundus image to obtain the original blood vessel image to be corrected and the original reference blood vessel image; performing binarization processing on the original blood vessel image to be corrected and the original reference blood vessel image to obtain the original binary image of the fundus image to be corrected and the original binary image of the reference fundus image; and determining the fundus image to be corrected and the reference fundus image based on the original binary image of the fundus image to be corrected and the original binary image of the reference fundus image.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, the step of determining the fundus image to be corrected and the reference fundus image based on the original binary image to be corrected and the original reference binary image, the original fundus image to be corrected and the original reference fundus image, includes: performing erosion processing on the original binary image to be corrected and the original reference binary image respectively to determine the vessel centerline of the original binary image to be corrected and the vessel centerline of the original vessel region of the original binary image; performing dilation processing on the vessel centerline of the original binary image to be corrected and the vessel centerline of the original binary image respectively to determine the vessel dilation region of the original binary image to be corrected and the vessel dilation region of the original binary image; extracting the region corresponding to the vessel dilation region of the original binary image to be corrected in the original fundus image to be corrected to obtain the fundus image to be corrected; and extracting the region corresponding to the vessel dilation region of the original reference fundus image in the original reference fundus image to obtain the reference fundus image.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, before determining the matching relationship data between the fundus image to be rectified and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers based on the fundus image to be rectified and the reference fundus image, the method further includes: determining fundus image samples and reference fundus image samples corresponding to the fundus image samples; training an initial network model including initial deformable convolutional layers and initial fully connected layers based on the fundus image samples and the reference fundus image samples corresponding to the fundus image samples to obtain a deep learning model.

[0011] Secondly, an embodiment of this disclosure provides an image processing method, comprising: acquiring multiple fundus images arranged according to shooting time; determining one fundus image from the multiple fundus images as a reference fundus image, and determining the fundus images other than the reference fundus image from the multiple fundus images as multiple fundus images to be corrected; registering the multiple fundus images to be corrected based on the reference fundus image and the multiple fundus images to be corrected using the image registration method mentioned in the first aspect, to obtain a registered image for each of the multiple fundus images to be corrected; and comparing the registered images of each of the multiple fundus images to be corrected to determine the image differences of the multiple fundus images to be corrected.

[0012] Thirdly, an embodiment of this disclosure provides an image registration apparatus, comprising: a determining module, configured to determine matching relationship data between the fundus image to be corrected and the reference fundus image based on a fundus image to be corrected and a reference fundus image, using a deep learning model including deformable convolutional layers and fully connected layers, wherein the deformable convolutional layer is used to extract feature points of the fundus image to be corrected and the reference fundus image respectively, and the fully connected layer is used to generate matching relationship data between the fundus image to be corrected and the reference fundus image based on the feature points of the fundus image to be corrected and the reference fundus image respectively; and a registration module, configured to fuse the fundus image to be corrected and the reference fundus image based on the matching relationship data to obtain a registered image corresponding to the fundus image to be corrected.

[0013] Fourthly, an embodiment of this disclosure provides an image processing apparatus, comprising: an acquisition module for acquiring multiple fundus images arranged according to shooting time; a reference image determination module for determining one fundus image from the multiple fundus images as a reference fundus image, and determining the fundus images other than the reference fundus image from the multiple fundus images as multiple fundus images to be corrected; a registration determination module for registering the multiple fundus images to be corrected based on the reference fundus image and the multiple fundus images to be corrected using the image registration method mentioned in the first aspect, to obtain a registration image for each of the multiple fundus images to be corrected; and a comparison module for comparing the registration images of each of the multiple fundus images to be corrected to determine the image differences of the multiple fundus images to be corrected.

[0014] Fifthly, an embodiment of this disclosure provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to perform the method mentioned in the first aspect above.

[0015] In a sixth aspect, one embodiment of this disclosure provides a computer-readable storage medium storing a computer program for performing the methods mentioned in the first aspect above.

[0016] The image registration method provided in this disclosure uses a deep learning model including deformable convolution and fully connected layers to determine the matching relationship data between the fundus image to be corrected and the reference fundus image. Based on this matching relationship data, the registered image for the image to be corrected is determined. Because deformable convolution can capture minute deformations between the fundus image to be corrected and the reference fundus image by extracting feature points from each image, it reduces interference from differences in resolution, size, and geometric distortion between the two images, as well as interference from occlusion and blurring in the image to be corrected, thereby improving the accuracy of image registration. Attached Figure Description

[0017] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to offer a further understanding of the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof.

[0018] Figure 1 The diagram shown illustrates an application scenario of the image registration method provided in one embodiment of this disclosure.

[0019] Figure 2 The diagram shown is a flowchart of an image registration method provided in an embodiment of this disclosure.

[0020] Figure 3 The diagram shows a flowchart of an embodiment of the present disclosure, which uses a deep learning model including deformable convolutional layers and fully connected layers to determine the matching relationship data between the fundus image to be corrected and the reference fundus image.

[0021] Figure 4 The diagram shown is a flowchart of an image registration method provided in another embodiment of this disclosure.

[0022] Figure 5 The diagram shown is a flowchart illustrating the process of performing vascular extraction processing on the original fundus image to be corrected and the original reference fundus image to determine the fundus image to be corrected and the reference fundus image, according to an embodiment of this disclosure.

[0023] Figure 6 The diagram shown is a flowchart of an image registration method provided in another embodiment of this disclosure.

[0024] Figure 7 The diagram shown is a flowchart of an image processing method provided in an embodiment of this disclosure.

[0025] Figure 8 The diagram shown is a structural schematic of an image registration device provided in an embodiment of this disclosure.

[0026] Figure 9 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this disclosure.

[0027] Figure 10 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0028] The technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments.

[0029] Image registration enables multiple fundus images to achieve a complete geometric correspondence in the spatial domain. Its purpose is to spatially align two or more images from different viewpoints and at different times through spatial transformations between the images to be registered, thereby assisting ophthalmologists in conducting comprehensive clinical assessments. By registering fundus images of patients at different times, doctors can determine changes in the patient's fundus and understand the progression of the disease. Traditional image registration methods include grayscale-based registration methods, edge-based registration methods, and contour-based registration methods.

[0030] However, traditional registration methods require linear illumination changes and rigid body transformations, or contrast within a certain range, in fundus images to achieve accurate registration. For example, grayscale-based registration methods can only achieve accurate registration under linear illumination changes and rigid body transformations during fundus image capture; while edge- or contour-based registration methods, where target recognition is only applicable to very small ranges of illumination changes, require a certain range to achieve accurate registration. However, lesions in fundus images often have unique characteristics. For example, fundus images from cataract patients have lower contrast than other images. Furthermore, fundus images may contain interference such as occlusion, geometric distortion, blurring, and severe nonlinear illumination changes. Therefore, the accuracy of traditional registration methods is reduced. Thus, there is an urgent need for an image registration method to address the low accuracy of traditional methods when registering fundus images.

[0031] The following is combined with Figure 1 A brief introduction to the application scenarios of the embodiments of this disclosure is provided.

[0032] Figure 1 The diagram illustrates an application scenario of an image registration method provided in an embodiment of this disclosure. Figure 1 As shown, this scenario is the registration of fundus image A to be corrected. Specifically, the application scenario of the image registration method includes server 110 and user terminal 120. There is a communication connection between server 110 and user terminal 120. Server 110 is used to execute the image registration method mentioned in the embodiments of this disclosure.

[0033] For example, in practical applications, a user sends a registration instruction for a fundus image A to be corrected to a server 110 via a user terminal 120. Upon receiving the instruction, the server 110 determines the image A to be corrected and a reference fundus image B. Based on the image A and the reference fundus image B, the server uses a deep learning model including deformable convolutional layers and fully connected layers to determine the matching relationship data between the two images. The deformable convolutional layers extract feature points from both the image A and the reference fundus image B, and the fully connected layers generate the matching relationship data based on these feature points. Based on the matching relationship data, the server fuses the image A with the reference fundus image B to obtain the registered image corresponding to the image A. The server sends the registration image corresponding to the fundus image A to be corrected to the user terminal 120 so that the user can view the registration image of the fundus image A to be corrected through the user terminal 120.

[0034] For example, the user terminal 120 mentioned above includes, but is not limited to, computer terminals such as desktop computers and laptops, and mobile terminals such as tablet computers and mobile phones.

[0035] The following is combined with Figures 2 to 6 The image registration method disclosed herein is introduced.

[0036] Figure 2 The diagram shown is a flowchart illustrating an image registration method provided in an embodiment of this disclosure. Figure 2 As shown, the image registration method provided in this embodiment includes the following steps.

[0037] Step S210: Based on the fundus image to be rectified and the reference fundus image, a deep learning model including deformable convolutional layers and fully connected layers is used to determine the matching relationship data between the fundus image to be rectified and the reference fundus image.

[0038] Deformable convolutional layers are used to extract feature points from the fundus image to be rectified and the reference fundus image, while fully connected layers are used to generate matching relationship data between the fundus image to be rectified and the reference fundus image based on the feature points of the fundus image to be rectified and the reference fundus image.

[0039] For example, the deep learning model can be a Homography Transform Network (HTN) model. The Homography Transform Network model can determine the matching relationship data between the fundus image to be corrected and the reference fundus image using feature points from both the image to be corrected and the reference image. Variable convolutional layers can be used to extract feature points from both the image to be corrected and the reference fundus image, determining subtle deformations between them. Fully connected layers are used to determine the global relationship between the image to be corrected and the reference image based on their respective feature points, ultimately generating the matching relationship data between the two fundus images.

[0040] Step S220: Based on the matching relationship data between the fundus image to be corrected and the reference fundus image, the fundus image to be corrected and the reference fundus image are fused to obtain the registration image corresponding to the fundus image to be corrected.

[0041] For example, based on the matching relationship data between the fundus image to be corrected and the reference fundus image, the mapping relationship between the fundus image to be corrected and the reference fundus image is determined. Using the mapping relationship between the fundus image to be corrected and the reference fundus image, the fundus image to be corrected and the reference fundus image are fused to obtain the registration image corresponding to the fundus image to be corrected.

[0042] This embodiment of the disclosure uses a deep learning model including deformable convolution and fully connected layers for registration. Since deformable convolution can capture the small deformations between the fundus image to be corrected and the reference fundus image by extracting the feature points of each of the fundus image to be corrected and the reference fundus image, it reduces the interference of different resolutions, different sizes, geometric distortions, etc. between the reference fundus image and the fundus image to be corrected, as well as occlusion and blurring in the image to be corrected, on image registration, making the registration result more accurate, thereby solving the traditional problems.

[0043] Figure 3 The diagram illustrates a flowchart of an embodiment of this disclosure, illustrating how a deep learning model, including deformable convolutional layers and fully connected layers, determines the matching relationship data between a fundus image to be corrected and a reference fundus image. Figure 3 As shown in the embodiments of this disclosure, the method for determining the matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers includes the following steps.

[0044] Step S310: Using deformable convolutional layers, feature extraction is performed on the fundus image to be corrected and the reference fundus image respectively to determine the feature points of the fundus image to be corrected and the reference fundus image.

[0045] For example, by using deformable convolutional layers to extract feature points from the fundus image to be corrected and the reference image, it is possible to determine the minute deformations of the fundus image to be corrected and the reference fundus image.

[0046] In step S320, a fully connected layer is used to perform homography mapping transformation on the feature points of the fundus image to be corrected and the reference fundus image respectively, to determine the homography matrix, and the homography matrix is ​​determined as the matching relationship data between the fundus image to be corrected and the reference fundus image.

[0047] For example, homography mapping transformation is performed on feature points of both the fundus image to be corrected and the reference fundus image using at least four feature points to determine an eight-parameter homography matrix. The parameters of the homography matrix are then calculated. This homography matrix with known parameters is used as the matching data between the fundus image to be corrected and the reference fundus image.

[0048] In this embodiment, the homography matrix is ​​determined by performing homography mapping transformation on the feature points of the fundus image to be corrected and the reference fundus image. The homography matrix is ​​then used as the matching relationship data between the fundus image to be corrected and the reference image, which simplifies the image registration calculation process and improves the efficiency of image registration.

[0049] Figure 4 The diagram shown is a flowchart illustrating an image registration method provided in another embodiment of this disclosure. Figure 3 Extending from the illustrated embodiment Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0050] like Figure 4 As shown, before using the deformable convolutional layer to extract features from the fundus image to be corrected and the reference fundus image respectively, and determining the feature points of the fundus image to be corrected and the reference fundus image respectively, the image registration method further includes the following steps.

[0051] Step S410: Obtain the original fundus image to be corrected and the original reference fundus image.

[0052] For example, the display area of ​​the original fundus image to be corrected may be smaller than the display area of ​​the original reference image.

[0053] Step S420: Perform blood vessel extraction processing on the original fundus image to be corrected and the original reference fundus image respectively to determine the fundus image to be corrected and the reference fundus image.

[0054] Both the fundus images to be corrected and the reference fundus images include vascular image regions.

[0055] Using deformable convolutional layers, feature extraction is performed on the fundus image to be calibrated and the reference fundus image to determine the feature points of each image. This includes: using deformable convolutional layers to extract features from the vascular image region in the fundus image to be calibrated and the vascular image region in the reference fundus image to determine the feature points of each image.

[0056] For example, blood vessel extraction can be performed on the fundus image to be corrected and the reference fundus image separately using a deep learning network, or feature extraction can be performed on the fundus image to be corrected and the reference fundus image separately using computer vision technology. Other blood vessel segmentation methods can also be used to extract features from the fundus image to be corrected and the reference fundus image separately. This disclosure does not limit the specific steps of the blood vessel extraction method.

[0057] This embodiment of the disclosure determines the fundus image to be corrected and the reference image by performing blood vessel extraction processing on the original image to be corrected and the original reference image. Using deformable convolutional layers, feature extraction is performed on both the fundus image to be corrected and the reference fundus image to determine the feature points of each image. The obtained feature points are located in the vascular image region. Since blood vessels, as a fundus feature, are not easily altered, the registration result is more accurate compared to other fundus features.

[0058] Figure 5 The diagram illustrates a process according to an embodiment of this disclosure, involving vascular extraction processing of an original fundus image to be corrected and an original reference fundus image to determine the fundus image to be corrected and the reference fundus image. Figure 5 As shown in the embodiments of this disclosure, the steps of performing blood vessel extraction processing on the original fundus image to be corrected and the original reference fundus image to determine the fundus image to be corrected and the reference fundus image include the following steps.

[0059] Step S510: Extract blood vessels from the original fundus image to be corrected and the original reference fundus image to obtain the original blood vessel image to be corrected and the original reference blood vessel image.

[0060] For example, blood vessels are extracted from the original fundus image to be corrected and the original reference fundus image, respectively, to determine the blood vessels in the original fundus image to be corrected and the original reference fundus image, thereby obtaining the original blood vessel image to be corrected and the original blood vessel image.

[0061] Step S520: The original vascular image to be corrected and the original reference vascular image are binarized to obtain the original fundus binarized image to be corrected and the original reference binarized image.

[0062] For example, the original vascular image to be corrected and the original reference vascular image are binarized based on a preset threshold. It should be understood that the preset threshold can be set according to needs, and the embodiments of this disclosure do not specifically limit the preset threshold.

[0063] Step S530: Based on the original binarized image to be corrected and the original reference binarized image, the original fundus image to be corrected and the original reference fundus image, determine the fundus image to be corrected and the reference fundus image.

[0064] For example, based on the original binarized image to be corrected and the original reference binarized image, the original fundus image to be corrected and the original reference fundus image, the fundus image to be corrected and the reference fundus image are determined by morphological processing.

[0065] In some embodiments, step S530 specifically includes: performing erosion processing on the original binary image to be corrected and the original reference binary image respectively to determine the vessel centerline of the original binary image to be corrected and the vessel centerline of the original vessel region of the original binary image; performing dilation processing on the vessel centerline of the original binary image to be corrected and the vessel centerline of the original binary image respectively to determine the vessel dilation region of the original binary image to be corrected and the vessel dilation region of the original binary image; extracting the region corresponding to the vessel dilation region of the original binary image to be corrected from the original fundus image to be corrected to obtain the fundus image to be corrected; extracting the region corresponding to the vessel dilation region of the original reference binary image from the original fundus image to obtain the reference fundus image. For example, the vessel centerline of the original binary image to be corrected and the vessel centerline of the original binary image are dilated, with the centerline as the reference, and the dilation is performed by 30 pixels. It should be understood that the size of the pixels for dilation processing can be selected as needed. In the original fundus image to be corrected, the region corresponding to the vascular dilation region in the original binary image to be corrected is extracted to obtain the fundus image to be corrected.

[0066] This embodiment extracts blood vessels from the original fundus image to be corrected and the original reference image to obtain the original fundus image to be corrected and the original reference fundus image. The original fundus image to be corrected and the original reference fundus image are then binarized to ensure that their range includes the vascular region. This allows the feature points extracted by the deep learning model to be based on the vascular region. Since the vascular region is less prone to deformation, the registration result of the fundus image to be corrected is more accurate. Furthermore, this embodiment erodes the original binarized image to be corrected and the original binarized image to be referenced, and then dilates the vessel centerlines of the eroded original binarized image to be corrected and the original binarized image to be referenced. By extracting the fundus image to be corrected and the reference image, the error in feature point extraction by the deep learning model and the disappearance of feature points can be reduced, further improving the accuracy of the registration result of the fundus image to be registered.

[0067] Figure 6 The diagram shown is a flowchart illustrating an image registration method provided in another embodiment of this disclosure. Figure 2 Extending from the illustrated embodiment Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0068] Figure 6 As shown, before determining the matching relationship data between the fundus image to be calibrated and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers, the image registration method further includes the following steps.

[0069] Step S610: Determine the fundus image sample and the reference fundus image sample corresponding to the fundus image sample.

[0070] For example, a reference fundus image sample is determined based on the fundus image sample. The reference fundus image sample corresponding to the fundus image sample can be pre-set, or it can be determined based on the fundus image sample according to the actual registration requirements.

[0071] Step S620: Train an initial network model including an initial deformable convolutional layer and an initial fully connected layer based on fundus image samples and corresponding reference fundus image samples to obtain a deep learning model.

[0072] For example, an initial network model including an initial deformable convolutional layer and an initial fully connected layer is trained based on fundus image samples and corresponding reference fundus image samples. The loss value is obtained through a loss function, and the initial network model is adjusted until the loss value reaches a preset threshold to obtain a deep learning model.

[0073] The embodiments of this disclosure train an initial network model including an initial deformable convolutional layer and an initial fully connected layer using fundus image samples and corresponding reference fundus image samples. The initial network model is adjusted by the loss value until the loss value reaches a preset threshold, thereby improving the robustness and accuracy of the deep learning model.

[0074] The following is combined with Figure 7 The image processing method disclosed herein is described.

[0075] Figure 7 The diagram shown is a schematic flowchart of an image processing method provided in an embodiment of this disclosure. Figure 7 As shown, the image processing method provided in this embodiment includes the following steps.

[0076] Step S710: Obtain multiple fundus images arranged according to the shooting time.

[0077] For example, multiple fundus images of a patient are arranged according to the time of the images being captured, such that the multiple fundus images are arranged in chronological order of the capture time.

[0078] Step S720: One fundus image from the multiple fundus images is identified as the reference fundus image, and the fundus images other than the reference fundus image from the multiple fundus images are identified as multiple fundus images to be corrected.

[0079] For example, depending on the requirements, a fundus image from a certain time period can be selected as the reference fundus image, or the earliest image captured can be directly selected as the reference image. This disclosure does not specifically limit the criteria for selecting the reference image.

[0080] Step S730: Based on the reference fundus image and multiple fundus images to be corrected, the multiple fundus images to be corrected are registered using an image registration method to obtain the registered images of each of the multiple fundus images to be corrected.

[0081] For example, multiple fundus images to be corrected are registered using the image registration method mentioned above to obtain registration maps for each of the multiple fundus images to be corrected.

[0082] Step S740: Compare the registration images of the multiple fundus images to be corrected to determine the image differences among the multiple fundus images to be corrected.

[0083] This embodiment of the present disclosure uses the image registration method mentioned above to obtain registration maps for each of the multiple fundus images to be corrected, and compares the registration maps of the multiple fundus images to be corrected to determine the image differences between the multiple fundus images to be corrected. This can help doctors determine the progression of the disease based on the patient's fundus images.

[0084] Figure 8 The diagram shown is a structural schematic of an image registration device provided in an embodiment of this disclosure. Figure 8 The image registration apparatus 800 provided in this embodiment includes a determining module 810 and a registration module 820. Specifically, the determining module 810 is used to determine the matching relationship data between the fundus image to be corrected and the reference fundus image based on the fundus image to be corrected and the reference fundus image, using a deep learning model including deformable convolutional layers and fully connected layers. The deformable convolutional layers are used to extract feature points from the fundus image to be corrected and the reference fundus image respectively, and the fully connected layers are used to generate the matching relationship data between the fundus image to be corrected and the reference fundus image based on the feature points from the fundus image to be corrected and the reference fundus image respectively. The registration module 820 is used to fuse the fundus image to be corrected and the reference fundus image based on the matching relationship data to obtain a registered image corresponding to the fundus image to be corrected.

[0085] In some embodiments, the determining module 810 is further configured to: use deformable convolutional layers to extract features from the fundus image to be corrected and the reference fundus image respectively, and determine the feature points of the fundus image to be corrected and the reference fundus image respectively; use fully connected layers to perform homography mapping transformation on the feature points of the fundus image to be corrected and the reference fundus image respectively, determine the homography matrix, and determine the homography matrix as the matching relationship data between the fundus image to be corrected and the reference fundus image.

[0086] In some embodiments, the determining module 810 is further configured to: acquire an original fundus image to be corrected and an original reference fundus image; perform vascular extraction processing on the original fundus image to be corrected and the original reference fundus image respectively to determine the fundus image to be corrected and the reference fundus image, wherein both the fundus image to be corrected and the reference fundus image include vascular image regions; wherein, using deformable convolutional layers, feature extraction is performed on the fundus image to be corrected and the reference fundus image respectively to determine the feature points of the fundus image to be corrected and the reference fundus image respectively, including: using deformable convolutional layers, feature extraction is performed on the vascular image regions in the fundus image to be corrected and the vascular image regions in the reference fundus image respectively to determine the feature points of the fundus image to be corrected and the reference fundus image respectively.

[0087] In some embodiments, the determining module 810 is further configured to: extract blood vessels from the original fundus image to be corrected and the original reference fundus image respectively to obtain the original vascular image to be corrected and the original reference vascular image; perform binarization processing on the original vascular image to be corrected and the original reference vascular image respectively to obtain the original binary image of the fundus to be corrected and the original binary image of the reference; and determine the fundus image to be corrected and the reference fundus image based on the original binary image to be corrected and the original binary image of the reference, the original fundus image to be corrected and the original reference fundus image.

[0088] In some embodiments, the determining module 810 is further configured to: perform erosion processing on the original binary image to be corrected and the original reference binary image respectively to determine the vessel centerline of the original binary image to be corrected and the vessel centerline of the original vessel region of the original binary image; perform dilation processing on the vessel centerline of the original binary image to be corrected and the vessel centerline of the original binary image respectively to determine the vessel dilation region of the original binary image to be corrected and the vessel dilation region of the original binary image; extract the region corresponding to the vessel dilation region of the original binary image to be corrected in the original fundus image to be corrected to obtain the fundus image to be corrected; and extract the region corresponding to the vessel dilation region of the original reference binary image in the original fundus image to obtain the reference fundus image.

[0089] In some embodiments, the determining module 810 is further configured to determine the fundus image sample and the reference fundus image sample corresponding to the fundus image sample; and to train an initial network model including an initial deformable convolutional layer and an initial fully connected layer based on the fundus image sample and the reference fundus image sample corresponding to the fundus image sample to obtain a deep learning model.

[0090] Figure 9 The diagram shown is a structural schematic of an image processing apparatus provided in an embodiment of this disclosure. Figure 9 The image processing apparatus 900 provided in this embodiment of the present disclosure includes: an acquisition module 910, a reference image determination module 920, a registration determination module 930, and a comparison module 940. Specifically, the acquisition module 910 is used to acquire multiple fundus images arranged according to the shooting time; the reference image determination module 920 is used to determine one fundus image from the multiple fundus images as a reference fundus image, and to determine the fundus images other than the reference fundus image from the multiple fundus images as multiple fundus images to be corrected; the registration determination module 930 is used to register the multiple fundus images to be corrected based on the reference fundus image and the multiple fundus images to be corrected using the image registration method mentioned in the first aspect, to obtain a registration image for each of the multiple fundus images to be corrected; the comparison module 940 is used to compare the registration images of each of the multiple fundus images to be corrected to determine the image differences of the multiple fundus images to be corrected.

[0091] Figure 10 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Figure 10 The electronic device 1000 shown (which may specifically be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, processor 1002, and communication interface 1003 are interconnected via the bus 1004.

[0092] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 may store a program, and when the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps in the image registration method or image processing method of the embodiments of this disclosure.

[0093] The processor 1002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by each unit of the image registration apparatus of this disclosure embodiment.

[0094] The processor 1002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image registration method of this disclosure can be completed by the integrated logic circuitry in the hardware of the processor 1002 or by instructions in software form. The processor 1002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1001. The processor 1002 reads the information in the memory 1001 and, in conjunction with its hardware, performs the functions required by the units included in the image registration apparatus of this disclosure, or executes the image registration method or image processing method of this disclosure.

[0095] The communication interface 1003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the electronic device 1000 and other devices or communication networks. For example, the communication interface 1003 can be used to determine the fundus image to be corrected and a reference fundus image.

[0096] Bus 1004 may include a pathway for transmitting information between various components of electronic device 1000 (e.g., memory 1001, processor 1002, communication interface 1003).

[0097] It should be noted that, although Figure 10 The illustrated electronic device 1000 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 1000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 1000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device 1000 may only include the devices necessary for implementing the embodiments of this disclosure, and may not necessarily include... Figure 10 All the devices shown.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0099] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0103] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0104] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An image registration method, characterized in that, include: Based on a fundus image to be corrected and a reference fundus image, a deep learning model including deformable convolutional layers and fully connected layers is used to determine the matching relationship data between the fundus image to be corrected and the reference fundus image. The deformable convolutional layers are used to extract feature points from both the fundus image to be corrected and the reference fundus image to determine minute deformations in the two images. The fully connected layers are used to determine the global relationship between the fundus image to be corrected and the reference fundus image based on their respective feature points, generating the matching relationship data. Both the fundus image to be corrected and the reference fundus image include vascular image regions. Based on the matching relationship data between the fundus image to be corrected and the reference fundus image, the fundus image to be corrected and the reference fundus image are fused to obtain the registration image corresponding to the fundus image to be corrected; The process of determining the matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers includes: Using the deformable convolutional layer, feature extraction is performed on the vascular image region in the fundus image to be corrected and the vascular image region in the reference fundus image, respectively, to determine the feature points of the fundus image to be corrected and the reference fundus image. Using the fully connected layer, homography mapping transformation is performed on the feature points of the fundus image to be corrected and the reference fundus image respectively to determine the homography matrix, and the homography matrix is ​​determined as the matching relationship data between the fundus image to be corrected and the reference fundus image; Before using the deformable convolutional layer to extract features from the fundus image to be corrected and the reference fundus image, and determining the feature points of each of the fundus image to be corrected and the reference fundus image, the method further includes: Acquire the original fundus image to be corrected and the original reference fundus image; the display area of ​​the original fundus image to be corrected is smaller than the display area of ​​the original reference fundus image; Vascular extraction is performed on the original fundus image to be corrected and the original reference fundus image to obtain the original vascular image to be corrected and the original reference vascular image, respectively. The original vascular image to be corrected and the original reference vascular image are binarized respectively to obtain the original fundus binarized image to be corrected and the original reference binarized image; Erosion processing is performed on the original binarized image to be corrected and the original reference binarized image respectively to determine the blood vessel centerline of the original binarized image to be corrected and the blood vessel centerline of the original blood vessel region of the original reference binarized image. The blood vessel centerline of the original binary image to be corrected and the blood vessel centerline of the original reference binary image are respectively subjected to dilation processing to determine the blood vessel dilation region of the original binary image to be corrected and the blood vessel dilation region of the original reference binary image. In the original fundus image to be corrected, the region corresponding to the vascular dilation region of the original binary image to be corrected is extracted to obtain the fundus image to be corrected. In the original reference fundus image, the region corresponding to the vascular dilation region of the original reference binarized image is extracted to obtain the reference fundus image.

2. The method according to claim 1, characterized in that, Before determining the matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers, the method further includes: Determine the fundus image sample and the corresponding reference fundus image sample; The deep learning model is obtained by training an initial network model, including an initial deformable convolutional layer and an initial fully connected layer, based on the fundus image samples and the corresponding reference fundus image samples.

3. An image processing method, characterized in that, include: Acquire multiple fundus images arranged chronologically by shooting time; One fundus image from the plurality of fundus images is designated as a reference fundus image, and the fundus images other than the reference fundus image from the plurality of fundus images are designated as multiple fundus images to be corrected. Based on the reference fundus image and the multiple fundus images to be corrected, the multiple fundus images to be corrected are registered using the image registration method described in claim 1 or 2, to obtain the registered images of each of the multiple fundus images to be corrected. The registered images of the multiple fundus images to be corrected are compared to determine the image differences between the multiple fundus images to be corrected.

4. An image registration device, characterized in that, include: A determination module is used to determine the matching relationship data between the fundus image to be corrected and the reference fundus image based on the fundus image to be corrected and the reference fundus image, using a deep learning model including deformable convolutional layers and fully connected layers. The deformable convolutional layer is used to extract feature points of the fundus image to be corrected and the reference fundus image respectively, and determine the minute deformations of the fundus image to be corrected and the reference fundus image. The fully connected layer is used to determine the global relationship between the fundus image to be corrected and the reference fundus image based on the feature points of the fundus image to be corrected and the reference fundus image respectively, and generate the matching relationship data between the fundus image to be corrected and the reference fundus image. The registration module is used to fuse the fundus image to be corrected and the reference fundus image based on the matching relationship data between the fundus image to be corrected and the reference fundus image to obtain a registration image corresponding to the fundus image to be corrected. The process of determining the matching relationship data between the fundus image to be corrected and the reference fundus image using a deep learning model including deformable convolutional layers and fully connected layers includes: Using the deformable convolutional layer, feature extraction is performed on the vascular image region in the fundus image to be corrected and the vascular image region in the reference fundus image, respectively, to determine the feature points of the fundus image to be corrected and the reference fundus image. Using the fully connected layer, homography mapping transformation is performed on the feature points of the fundus image to be corrected and the reference fundus image respectively to determine the homography matrix, and the homography matrix is ​​determined as the matching relationship data between the fundus image to be corrected and the reference fundus image; Before using the deformable convolutional layer to extract features from the fundus image to be corrected and the reference fundus image, and determining the feature points of each of the fundus image to be corrected and the reference fundus image, the method further includes: Acquire the original fundus image to be corrected and the original reference fundus image; the display area of ​​the original fundus image to be corrected is smaller than the display area of ​​the original reference fundus image; Vascular extraction is performed on the original fundus image to be corrected and the original reference fundus image to obtain the original vascular image to be corrected and the original reference vascular image, respectively. The original vascular image to be corrected and the original reference vascular image are binarized respectively to obtain the original fundus binarized image to be corrected and the original reference binarized image; Erosion processing is performed on the original binarized image to be corrected and the original reference binarized image respectively to determine the blood vessel centerline of the original binarized image to be corrected and the blood vessel centerline of the original blood vessel region of the original reference binarized image. The blood vessel centerline of the original binary image to be corrected and the blood vessel centerline of the original reference binary image are respectively subjected to dilation processing to determine the blood vessel dilation region of the original binary image to be corrected and the blood vessel dilation region of the original reference binary image. In the original fundus image to be corrected, the region corresponding to the vascular dilation region of the original binary image to be corrected is extracted to obtain the fundus image to be corrected. In the original reference fundus image, the region corresponding to the vascular dilation region of the original reference binarized image is extracted to obtain the reference fundus image.

5. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for performing the method described in any one of claims 1 to 3.

Images