Fundus image registration method based on key point matching network
By using rigid and affine registration networks with key point matching in fundus image registration, combined with spatial transformation layer and mask area training, the problem of insufficient fundus image registration efficiency and accuracy in the prior art is solved, and rapid , high-precision image alignment.
Patent Information
- Application Number
- CN202110167328.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-02-05
AI Technical Summary
Existing fundus image registration methods are inefficient, especially the unsupervised methods based on deep learning are not effective on fundus images, resulting in insufficient registration speed and accuracy.
Using rigid registration networks and affine registration networks based on key point matching, after the rigid registration network is initially registered, the affine registration network is finely registered, and the network is trained using the spatial transformation layer and mask area to improve matching. Accurate accuracy.
The speed and accuracy of fundus image registration are significantly improved, and the problem of poor performance of unsupervised registration schemes is solved, achieving fast and accurate image alignment.
Smart Images

Figure CN112819867B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a fundus image registration method based on a key point matching network, and belongs to the technical field of image processing. Background Art
[0002] Fundus color photography is a common medical image used for the diagnosis of fundus diseases. For fundus color photography of the same patient taken at different time points, doctors can determine changes in the patient's condition by changes in imaging. These images are an important basis for longitudinal analysis of fundus images. However, the shooting angles and shooting equipment of these images may be different, resulting in misalignment between the images. Therefore, fundus image registration is an essential part of longitudinal tracking of fundus images.
[0003] At present, many studies have focused on the registration problem of fundus images. Traditional fundus image registration methods can be mainly divided into two categories, one is feature-based registration method, and the other is grayscale similarity measurement-based registration method. Traditional registration methods usually extract features first, such as key points, blood vessels, etc., then match these features, and finally iteratively optimize the transformation parameters based on these matched feature information. This method iteratively optimizes the model parameters for each pair of images to be registered, so the registration speed is very slow. It may even take more than ten minutes to match a pair of images.
[0004] With the rise of deep learning, some researchers have proposed medical image registration methods based on deep learning. These methods have already trained the registration model in the training phase, and no longer require iterative optimization in the testing phase. The registered image can be obtained through forward propagation by directly inputting the image, which greatly improves the registration speed. However, for fundus image registration, there are currently few registration methods based on deep learning. Common methods used for other modalities are not applicable to fundus images. For example, unsupervised registration methods often rely on grayscale similarity measurement, and methods based on grayscale similarity measurement do not work well on fundus images. Summary of the invention
[0005] The purpose of the present invention is to provide a fundus image registration method based on a key point matching network, so as to solve the problems existing in the prior art.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] According to a first aspect, an embodiment of the present invention provides a fundus image registration method based on a key point matching network, the method comprising:
[0008] Acquire a sample fundus image, wherein the sample fundus image includes n image pairs, each image pair includes a fixed image and a floating image, and n is an integer greater than 2;
[0009] Inputting the sample fundus image into a rigid registration network to obtain a rigid transformation matrix;
[0010] Performing a rigid transformation on the key point coordinates in the fixed image according to the rigid transformation matrix to obtain transformed key point coordinates;
[0011] Generate a rigidly registered fundus image after the floating image is registered according to the key point coordinates obtained by transformation using a spatial transformation layer;
[0012] Training the rigid registration network according to the sample fundus image and the rigid registration fundus image to obtain a trained rigid registration network, wherein the trained rigid registration network is used to perform preliminary registration on the fundus image;
[0013] The affine registration network is trained according to the rigidly registered fundus image and the sample fundus image to obtain a trained affine registration network, and the trained affine registration network is used to perform fine registration on the fundus image.
[0014] Optionally, inputting the sample fundus image into a rigid registration network to obtain a rigid transformation matrix includes:
[0015] The sample fundus image is input into the rigid registration network to obtain the rigid transformation matrix including transformation parameters, wherein the transformation parameters include the angle θ of rotation around the center (x0, y0), the horizontal axis translation t x and the longitudinal translation t y At least two of the .
[0016] Optionally, performing a rigid transformation on the key point coordinates in the fixed image according to the rigid transformation matrix to obtain the transformed key point coordinates includes:
[0017] The transformed coordinates of the key points are:
[0018] in, is the jth key point coordinate in the fixed image, is the coordinate of the jth key point obtained by transformation, the maximum value of j is the number of key points, T r is the rigid transformation matrix.
[0019] Optionally, the loss function of the rigid registration network is:
[0020]
[0021] in, is a floating image i m The coordinates of the jth key point on is the coordinate of the jth key point in the registered image, and the maximum value of j is the number of key points.
[0022] Optionally, the training the affine registration network according to the rigidly registered fundus image and the sample fundus image to obtain the trained affine registration network includes:
[0023] For each image pair, obtain the mask M of the fixed image f And the mask M of the registered image W ;
[0024] According to the mask M of the fixed image f and the rigid registration image to determine the registration image overlap portion I of the fixed image and the rigid registration image m ;
[0025] According to the mask M of the rigid registration image W and the fixed image to determine the fixed image overlap portion I of the fixed image and the rigid registration image f ;
[0026] According to the I m and the I f The affine registration network is trained.
[0027] Optionally, the I m and the I f Train an affine registration network, including:
[0028] The I m and the I f Input into the affine registration network to obtain an affine transformation matrix;
[0029] According to the affine transformation matrix, the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates;
[0030] The spatial transformation layer is used to generate the I according to the key point coordinates obtained by transformation. f Affine registered images after registration;
[0031] According to the I f And the affine registration image trains the affine registration network to obtain a trained affine registration network.
[0032] Optionally, the mask M according to the fixed image f and the rigid registration image to determine the registration image overlap portion I of the fixed image and the rigid registration image m ,include:
[0033] Said Among them, i w is the registered image;
[0034] The mask M according to the rigid registration image W and the fixed image to determine the fixed image overlap portion I of the fixed image and the rigid registration image f ,include:
[0035] Among them, i f is the fixed image.
[0036] In a second aspect, a fundus image registration method based on a key point matching network is provided, the method comprising:
[0037] Acquire a target fixed fundus image and a target floating fundus image;
[0038] Inputting the target fixed fundus image and the target floating fundus image into a trained rigid registration network to obtain a rigid transformation matrix, wherein the trained rigid registration network is obtained by training using the training method described in the first aspect;
[0039] Performing a rigid transformation on the key point coordinates in the target fixed fundus image according to the rigid transformation matrix to obtain transformed key point coordinates;
[0040] Generate a target rigidly registered fundus image after the target floating fundus image is registered according to the key point coordinates obtained by transformation using a spatial transformation layer;
[0041] According to the target rigidly registered fundus image, the target floating fundus image and the trained affine registration network, a registered target affine registered fundus image is obtained, and the trained affine registration network is obtained by training using the training method described in the first aspect.
[0042] Optionally, obtaining the registered target affine-registered fundus image according to the target rigid-registered fundus image, the target floating fundus image and the trained affine registration network includes:
[0043] Obtain the mask M of the target fixed fundus image f And the mask M of the target rigidly registered fundus image W ;
[0044] The mask M of the fundus image is fixed according to the target f and the target rigidly registered fundus image to determine the registered image overlap portion I of the target fixed fundus image and the target rigidly registered fundus image m ;
[0045] The mask M of the fundus image is rigidly registered according to the target W and the target fixed fundus image to determine a fixed image overlapping portion I of the target fixed fundus image and the target rigidly registered fundus image f ;
[0046] The I m and the I f Input into the trained affine registration network to obtain the affine transformation matrix;
[0047] According to the affine transformation matrix, the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates;
[0048] The spatial transformation layer is used to generate the I according to the key point coordinates obtained by transformation. f Registered images after registration.
[0049] By acquiring a sample fundus image, the sample fundus image includes n image pairs, each image pair includes a fixed image and a floating image, and n is an integer greater than 2; the sample fundus image is input into a rigid registration network to obtain a rigid transformation matrix; the key point coordinates in the fixed image are rigidly transformed according to the rigid transformation matrix to obtain the transformed key point coordinates; a spatial transformation layer is used to generate a rigidly registered fundus image after the floating image is registered according to the transformed key point coordinates; the rigid registration network is trained according to the floating image and the rigidly registered fundus image to obtain a trained rigid registration network, the trained rigid registration network is used to perform preliminary registration on the fundus image, and thereafter an affine registration network is trained, the trained affine registration network is used to perform fine registration on the fundus image. That is, a rigid registration network and an affine registration network based on key point matching are proposed, and after preliminary registration through the rigid registration network, fine registration through the affine registration network is performed, which effectively solves the problem of poor registration effect of the unsupervised registration scheme in the existing scheme, and achieves the effect of improving the registration accuracy.
[0050] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A method flow chart of a fundus image registration method based on a key point matching network provided by an embodiment of the present invention;
[0052] Figure 2 A schematic diagram of key points marked in a fundus image provided by one embodiment of the present invention;
[0053] Figure 3 A schematic diagram of a possible network structure of a rigid registration network provided by an embodiment of the present invention;
[0054] Figure 4 A possible training diagram of a rigid registration network provided by one embodiment of the present invention;
[0055] Figure 5 A schematic diagram of training a rigid registration network and an affine registration network provided in one embodiment of the present invention;
[0056] Figure 6 A schematic diagram of a fundus image collected according to an embodiment of the present invention;
[0057] Figure 7 A method flow chart of a fundus image registration method based on a key point matching network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0058] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0060] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0061] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0062] Please refer to Figure 1 , which shows a method flow chart of a fundus image registration method based on a key point matching network provided by an embodiment of the present application, such as Figure 1 As shown, the method includes:
[0063] Step 101, acquiring a sample fundus image, wherein the sample fundus image includes n image pairs, each image pair includes a fixed image and a floating image, and n is an integer greater than 2;
[0064] The sample fundus images used in this application include color fundus photos of four types of patients, and the four diseases are diabetic retinopathy, central artery occlusion, branch artery occlusion, and age-related macular degeneration. The dataset contains a total of 492 pairs of images from 165 patients, and the original image size is 3046*2572. Optionally, each sample fundus image is pre-annotated with m key points, and the m key points are evenly distributed in each image, and m is an integer greater than 2. This application takes m=3 as an example, for example, please refer to Figure 2 , which shows a possible key point distribution diagram. The annotated image is characterized by the coordinates of the annotated key points.
[0065] Optionally, after obtaining the sample fundus image, the sample fundus image can be preprocessed, and the preprocessing steps include: extracting the green channel, scaling it to a size of 512*512, and enhancing the grayscale image through contrast-limited adaptive histogram equalization. Optionally, the albumentations data enhancement library can be used to amplify the image data, specifically by random translation and random rotation.
[0066] Step 102, inputting the sample fundus image into a rigid registration network to obtain a rigid transformation matrix;
[0067] Please refer to Figure 3 , which shows a possible network structure diagram of the rigid registration network used in the embodiment of the present application. The rigid registration network can be a regression network with a ResNet18 structure.
[0068] The rigid transformation matrix includes transformation parameters, which include the angle θ of rotation around the center (x0, y0), the horizontal axis translation t x and the longitudinal translation t y In addition, unless otherwise specified, the following examples are given with the transformation parameters including the above three at the same time.
[0069] Fixed image to i f , the floating image is i m , please refer to Figure 4 In this step, the preprocessed fixed image if and floating image i m The channels are combined to form the input of the rigid registration network. After that, the input passes through four residual blocks to obtain a 512-dimensional feature map. The feature map is then converted into a 512-dimensional vector by the global average pooling layer, and then passes through two fully connected layers to obtain 1024-dimensional and 3-dimensional vectors respectively. The 3-dimensional vector is the 3 transformation parameters mentioned above, and the 3 transformation parameters constitute the rigid transformation matrix T r .
[0070] In one possible embodiment, T r It can be expressed as:
[0071]
[0072] Step 103, performing a rigid transformation on the key point coordinates in the fixed image according to the rigid transformation matrix to obtain transformed key point coordinates;
[0073] The transformed coordinates of the key points are:
[0074] in, is the jth key point coordinate in the fixed image, is the coordinate of the jth key point obtained by transformation, the maximum value of j is the number of key points, T r is the rigid transformation matrix. j=1,2,3.
[0075] Step 104, using a spatial transformation layer to generate a rigidly registered fundus image after the floating image is registered according to the key point coordinates obtained by transformation;
[0076] The spatial transformation layer is used to generate the registered image, that is, the floating image i m Distorted image i after rigid transformation w ,i w The pixel value at each position is m If the spatial transformation layer adopts the bilinear interpolation algorithm, then i w superior The pixel value is given by i m superior The four adjacent pixels are bilinearly interpolated. That is to say, for the same image pair, the distorted coordinates generated after coordinate transformation are These are the sampling points that will be applied to the floating image in the spatial transformation layer. w Upper key point coordinates The pixel value on i is just m Zhongyu Coordinates of matched key points The pixel interpolation at the position is obtained, so i w Talent and i f Alignment.
[0077] Step 105 , training the rigid registration network according to the floating image and the rigidly registered fundus image to obtain a trained rigid registration network, and the trained rigid registration network is used to perform preliminary registration on the fundus image.
[0078] The loss function of the rigid registration network uses the registered image i w Key Points With floating image i m Key Points The root mean square error between them can be expressed as:
[0079]
[0080] in, is a floating image i m The coordinates of the jth key point on is the coordinate of the jth key point in the registered image, and the maximum value of j is the number of key points.
[0081] Optional, please refer to Figure 5 In order to further improve the registration accuracy, after training the rigid registration network, an affine registration network can be trained based on the rigidly registered fundus image and the sample fundus image, and then the rigidly registered fundus image registered by the rigid registration network can be further registered by the affine registration network. The training method of the affine registration network is as follows:
[0082] First, for each image pair, obtain the mask M of the fixed image f And the mask M of the rigidly registered fundus image W ;
[0083] Please refer to Figure 6 , the two images to be registered cannot completely overlap after registration due to differences in shooting angles, eye movement, etc. The non-overlapping areas on the edges may interfere with the learning of the neural network, so these areas need to be removed. This step generates a mask of the original image to be registered by thresholding to extract the image of the fundus color photo part. The mask of the fixed image is denoted as M f , the mask of the image generated by rigid registration is denoted as M W .
[0084] Second, according to the mask M of the fixed image f and the rigidly registered fundus image to determine a registered image overlap portion I of the fixed image and the rigidly registered fundus image m ;
[0085] The mask M of the fixed image f And rigid registration generates image i w Do the multiplication, write Thus, the overlapped part I of the rigid registration generated image of the fixed image and the rigid registration generated image is obtained. m .
[0086] Third, according to the mask M of the registered image W and the fixed image to determine a fixed image overlap portion I of the fixed image and the rigidly registered fundus image f ;
[0087] The mask M of the image generated by rigid registration W With fixed image i f Do the multiplication, write Thus, the overlapped part I of the fixed image and the image generated by rigid registration is obtained. f。
[0088] Fourth, according to the I m and the I f An affine registration network is trained, and the trained affine registration network is used to perform affine registration on the image registered by the rigid registration network.
[0089] The affine registration network is also Figure 3 The regression network on the left is a regression network with a ResNet18 structure.
[0090] Optionally, this step may include:
[0091] (1) The I m and the I f Input into the affine registration network to obtain an affine transformation matrix;
[0092] Similar to the training steps of the rigid registration network mentioned above, this step can be used to train I m ,I f The channels are combined to form the input of the affine registration network, and then the input passes through four residual blocks to obtain a 512-dimensional feature map. The feature map is then converted into a 512-dimensional vector by the global average pooling layer, and then passes through two fully connected layers to obtain 1024-dimensional and 6-dimensional vectors respectively. The 6-dimensional vector is the affine transformation matrix. The affine transformation matrix T a The 6 parameters can be expressed as:
[0093]
[0094] (2) According to the affine transformation matrix, the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates;
[0095] Through the affine transformation matrix to fix the image I f The coordinates of the three key points Perform affine transformation to obtain the coordinates of the three key points after transformation The transformed key point coordinates are:
[0096]
[0097] Among them, j = 1, 2, 3 represents the coordinates of the jth key point, and are the coordinates generated by the affine transformation layer, and isI f Coordinates of the top keypoint.
[0098] (3) Using the spatial transformation layer to generate the I f The registered image after registration;
[0099] This step is similar to the above step 104. Specifically, the spatial transformation layer is used to generate the registered image, that is, the floating image I m Distorted image after affine transformation I w ,I w The pixel value at each position is in I m According to the definition of the spatial transformation layer, if the bilinear interpolation algorithm is used, I w superior The pixel value is given by I m superior The four adjacent pixels are bilinearly interpolated, and is calculated by formula (6). In other words, for the same pair of images to be registered, their distorted coordinates generated by the coordinate transformation layer are These are the sampling points that will be applied to the floating image in the spatial transformation layer. w Upper key point coordinates The pixel value on I m Zhongyu Coordinates of matched key points The pixel interpolation at the position is obtained, so I w Talent and I f Alignment.
[0100] (4) According to the I f And the rigidly registered fundus image is used to train the affine registration network to obtain a trained affine registration network.
[0101] The loss function of the affine registration network can be used as Iw Key Points with I m Key Points The root mean square error between them, the loss function loss can be defined as:
[0102]
[0103] in isI m The j-th pair of key point coordinates on According to formula (6), I f The jth pair of keypoint coordinates Perform affine transformation T a get.
[0104] After the above steps, the trained rigid registration network and the trained affine registration network can be obtained.
[0105] In summary, by acquiring a sample fundus image, the sample fundus image includes n image pairs, each image pair includes a fixed image and a floating image, and n is an integer greater than 2; the sample fundus image is input into a rigid registration network to obtain a rigid transformation matrix; the key point coordinates in the fixed image are rigidly transformed according to the rigid transformation matrix to obtain the transformed key point coordinates; a rigid registration fundus image after the floating image registration is generated according to the transformed key point coordinates using a spatial transformation layer; the rigid registration network is trained according to the floating image and the rigid registration fundus image to obtain a trained rigid registration network, the trained rigid registration network is used to perform preliminary registration on the fundus image, and thereafter an affine registration network is trained, the trained affine registration network is used to perform fine registration on the fundus image. That is, a rigid registration network and an affine registration network based on key point matching are proposed, and after preliminary registration by the rigid registration network, fine registration by the affine registration network is performed, which effectively solves the problem of poor registration effect of the unsupervised registration scheme in the existing scheme, and achieves the effect of improving the registration accuracy.
[0106] Please refer to Figure 7 , which shows a method flow chart provided by an embodiment of the present application, such as Figure 7 As shown, the method includes:
[0107] Step 701, acquiring a target fixed fundus image and a target floating fundus image;
[0108] After obtaining the target fixed fundus image and the target floating fundus image to be registered, the target fixed fundus image and the target floating fundus image may be preprocessed. The preprocessing steps are similar to those in the above embodiment and will not be described in detail.
[0109] Step 702, inputting the target fixed fundus image and the target floating fundus image into a trained rigid registration network to obtain a rigid transformation matrix;
[0110] This step is similar to step 102 in the above embodiment and will not be described again.
[0111] Step 703, performing a rigid transformation on the key point coordinates in the target fixed fundus image according to the rigid transformation matrix to obtain transformed key point coordinates;
[0112] This step is similar to step 103 in the above embodiment and will not be described again.
[0113] Step 704 : Generate a target rigidly registered fundus image after the target floating fundus image is registered, using a spatial transformation layer according to the key point coordinates obtained by transformation.
[0114] This step is similar to step 104 in the above embodiment and will not be described in detail here.
[0115] It should be noted that, according to the target rigidly registered fundus image, the target floating fundus image and the trained affine registration network, the registered target affine registered fundus image is obtained, and the trained affine registration network is the network trained by the above embodiment. The steps of affine registration may include:
[0116] First, obtain the mask M of the target fixed fundus image. f And the mask M of the target rigidly registered fundus image W ;
[0117] Second, the mask M of the fundus image is fixed according to the target f and the target rigidly registered fundus image to determine the registered image overlap portion I of the target fixed fundus image and the target rigidly registered fundus image m ;
[0118] Third, the mask M of the fundus image is rigidly registered according to the target W and the target fixed fundus image to determine a fixed image overlapping portion I of the target fixed fundus image and the target rigidly registered fundus image f ;
[0119] Fourth, the I m and the I f Input into the trained affine registration network to obtain the affine transformation matrix;
[0120] Fifth, transform the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates;
[0121] Sixth, a spatial transformation layer is used to generate the I f Affine-registered fundus images after registration.
[0122] The above steps and Figure 1 The specific implementation in the illustrated embodiment is similar, except that the above embodiment uses a sample fixed fundus image and a sample floating fundus image, while this step uses a target fixed fundus image and a target floating fundus image.
[0123] In this embodiment, the network architecture used is Figure 5 The network architecture after removing the background color part. Figure 1 In the embodiment shown, the trained rigid registration network and the trained affine registration network are connected in series, and the automatic extraction algorithm of the overlapping area is connected to the connection part, so that an end-to-end registration network can be realized, and the image to be registered, that is, the target fixed fundus image i f and target floating fundus image i m By inputting the network, we can directly get the registered image. This process can be expressed as:
[0124] Among them, Φ r is a rigid transformation, Φ a is an affine transformation, Represents space transformation.
[0125] In order to quantitatively evaluate the performance of the method proposed in the present invention, the registration results are evaluated according to the following two indicators: RMSE (Root mean squared error) and AUC (Area under curve). RMSE calculates the root mean square error between the 10 pairs of key points in the test set, and AUC calculates the area of the curve drawn with RMSE as the threshold for successful registration and the registration success rate of the entire test set as the ordinate.
[0126] Table 1 shows the results of the ablation experiment and the comparative experiment of the present invention. The first four rows compare the two existing unsupervised registration methods based on deep learning. It can be seen that the objective function proposed by us can greatly improve the registration accuracy compared with the existing unsupervised learning method; the last three rows are ablation experiments, which show the effectiveness of the two-step registration scheme.
[0127] Table 1
[0128]
[0129] In summary, by acquiring a target fixed image and a target floating image; inputting the target fixed image and the target floating image into a trained rigid registration network to obtain a rigid transformation matrix, wherein the trained rigid registration network is obtained by the training method described in the above embodiment; performing a rigid transformation on the key point coordinates in the target fixed image according to the rigid transformation matrix to obtain the transformed key point coordinates; using a spatial transformation layer to generate a target rigid registration image after registration of the target floating image according to the transformed key point coordinates, and then obtaining a registered target affine registered fundus image according to the target rigid registration fundus image, the target floating fundus image and the trained affine registration network. That is, a rigid registration network and an affine registration network based on key point matching are proposed, and after preliminary registration by the rigid registration network, fine registration by the affine registration network is performed, which effectively solves the problem of poor registration effect of the unsupervised registration scheme in the existing scheme, and achieves the effect of improving the registration accuracy.
[0130] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0131] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A fundus image registration method based on key point matching network, characterized in that: The method comprises: Acquire a sample fundus image, wherein the sample fundus image includes n image pairs, each image pair includes a fixed image and a floating image, and n is an integer greater than 2; Inputting the sample fundus image into a rigid registration network to obtain a rigid transformation matrix; Performing a rigid transformation on the key point coordinates in the fixed image according to the rigid transformation matrix to obtain transformed key point coordinates; Generate a rigidly registered fundus image after the floating image is registered according to the key point coordinates obtained by transformation using a spatial transformation layer; Training the rigid registration network according to the sample fundus image and the rigid registration fundus image to obtain a trained rigid registration network, wherein the trained rigid registration network is used to perform preliminary registration on the fundus image; Training an affine registration network according to the rigidly registered fundus image and the sample fundus image to obtain the trained affine registration network, wherein the trained affine registration network is used to perform fine registration on the fundus image; The step of inputting the sample fundus image into a rigid registration network to obtain a rigid transformation matrix comprises: inputting the sample fundus image into a rigid registration network to obtain the rigid transformation matrix including transformation parameters, wherein the transformation parameters include an angle θ of rotation around a center (x0, y0), a horizontal axis translation amount t x and the longitudinal translation t y At least two of the following; The training of the affine registration network according to the rigidly registered fundus image and the sample fundus image to obtain the trained affine registration network includes: for each image pair, obtaining a mask M of the fixed image f And the mask M of the rigidly registered fundus image W ; According to the mask M of the fixed image f and the rigidly registered fundus image to determine a registered image overlap portion I of the fixed image and the rigidly registered fundus image m ; According to the mask M of the rigid registration fundus image W and the fixed image to determine a fixed image overlap portion I of the fixed image and the rigidly registered fundus image f According to the I m and the I f The affine registration network is trained.
2. The method according to claim 1, characterized in that The step of performing a rigid transformation on the key point coordinates in the fixed image according to the rigid transformation matrix to obtain the transformed key point coordinates includes: The transformed coordinates of the key points are: in, is the jth key point coordinate in the fixed image, is the coordinate of the jth key point obtained by transformation, the maximum value of j is the number of key points, T r is the rigid transformation matrix.
3. The method according to claim 1, characterized in that The loss function of the rigid registration network is: in, is a floating image i m The coordinates of the jth key point on is the coordinate of the jth key point obtained by transformation, and the maximum value of j is the number of key points.
4. The method according to claim 1, characterized in that: According to the I m and the I f Train an affine registration network, including: The I m and the I f Input into the affine registration network to obtain an affine transformation matrix; According to the affine transformation matrix, the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates; The spatial transformation layer is used to generate the I according to the key point coordinates obtained by transformation. f Affine registered images after registration; According to the I f And the affine registration image trains the affine registration network to obtain a trained affine registration network.
5. The method according to claim 1, characterized in that The mask M according to the fixed image f and the rigidly registered fundus image to determine a registered image overlap portion I of the fixed image and the rigidly registered fundus image m ,include: Said Among them, i w is the registered image; The mask M of the fundus image according to the rigid registration W and the fixed image to determine a fixed image overlap portion I of the fixed image and the rigidly registered fundus image f ,include: Among them, i f is the fixed image.
6. A fundus image registration method based on key point matching network, characterized in that: The method comprises: Acquire a target fixed fundus image and a target floating fundus image; Inputting the target fixed fundus image and the target floating fundus image into a trained rigid registration network to obtain a rigid transformation matrix, wherein the trained rigid registration network is trained by the method according to any one of claims 1 to 5; Performing a rigid transformation on the key point coordinates in the target fixed fundus image according to the rigid transformation matrix to obtain transformed key point coordinates; Generate a target rigidly registered fundus image after the target floating fundus image is registered according to the key point coordinates obtained by transformation using a spatial transformation layer; According to the target rigidly registered fundus image, the target floating fundus image and the trained affine registration network, a registered target affine registered fundus image is obtained, and the trained affine registration network is trained by any one of the methods described in claims 1 to 5.
7. The method according to claim 6, characterized in that The step of obtaining a registered target affine-registered fundus image according to the target rigid-registered fundus image, the target floating fundus image and the trained affine registration network comprises: Obtain the mask M of the target fixed fundus image f And the mask M of the target rigidly registered fundus image W ; The mask M of the fundus image is fixed according to the target f and the target rigidly registered fundus image to determine the registered image overlap portion I of the target fixed fundus image and the target rigidly registered fundus image m ; The mask M of the fundus image is rigidly registered according to the target W and the target fixed fundus image to determine a fixed image overlapping portion I of the target fixed fundus image and the target rigidly registered fundus image f ; The I m and the I f Input into the trained affine registration network to obtain the affine transformation matrix; According to the affine transformation matrix, the I f Perform affine transformation on the key point coordinates in to obtain the transformed key point coordinates; The spatial transformation layer is used to generate the I according to the key point coordinates obtained by transformation. f Registered images after registration.
