Multi-modal eye fundus image registration method based on key point pair intelligent matching

Abstract

The invention discloses a multi-modal eye fundus image registration method based on key point pair intelligent matching, and particularly relates to the technical field of computer image processing, a multi-modal eye fundus image data set is divided and enhanced and then sent to a YOLOv8 neural network for training, an optimal weight obtained through training is selected for prediction, and a prediction result is obtained; the prediction result of the YOLOv8 neural network comprises two parts, one part is a test set picture marking key point pairs, bounding box categories and confidence; the other part is a text format file containing key point pair categories, bounding box coordinates and key point pair coordinates; and selecting three key point pairs according to a prediction result, calculating an affine transformation matrix by the three key point pairs, and performing affine transformation on the image to obtain a registration result.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a multimodal fundus image registration method based on intelligent matching of key point pairs. Background Art

[0002] Medical image registration is a basic step in computer-aided diagnosis and image-guided surgical treatment. In fundus image registration, intensity-based registration methods were first used. Intensity-based registration methods regard the problem as an iterative optimization problem and are mainly focused on developing various similarity functions, including (normalized) cross-correlation (CC), (normalized) mutual information (MI), etc. However, intensity-based registration methods are very sensitive to changes in image illumination and contrast. Feature-based registration methods are more effective than intensity-based registration methods. The idea is to extract key features from the input image, calculate a descriptor for each feature, match the closest features in the two images, and then establish a transformation relationship. Scale-invariant feature transform (SIFT) is groundbreaking in feature-based matching methods. Compared with SIFT, speeded-up robust features (SURF) and ORB (Oriented FAST and Rotated BRIEF) are more computationally efficient. However, when the image pairs to be registered have different morphologies due to different cameras, modalities, or background changes caused by retinal lesions, the registration accuracy of these traditional methods that appeared before deep learning technology is unsatisfactory; at the same time, the high time cost caused by iterative optimization is another problem faced by traditional methods.

[0003] The deep learning-based fundus image registration method uses a three-stage method (segmentation, detection and description, and exclusion of outliers). The advantage of this method is that it bypasses the intensity gap between different modalities and achieves significant registration accuracy, but the disadvantage is that it is relatively complex. In other medical image registrations, most nonlinear transformations are used. When it comes to fundus image registration, linear transformation is the most commonly used method. Fundus image registration usually uses CF images and FFA images, which rely entirely on visible light illumination for imaging. This imaging principle, coupled with the natural movement of the subject's eyeball, may result in large differences in image brightness, contrast, and viewing angle when taking multiple photos. Nonlinear transformations are difficult to obtain good registration results under such large differences.

[0004] The differences in imaging principles of fundus image acquisition devices of different modalities lead to large differences in brightness and contrast in multimodal fundus images. The intensity-based registration method is very sensitive to the brightness difference of the image and is difficult to be directly applied to multimodal fundus image registration. At the same time, the main directions of the feature descriptors of the same feature on fundus images of different modalities are quite different, resulting in a low matching success rate of the features, making it difficult for the feature-based registration method to obtain stable registration results. In short, the traditional fundus image registration method cannot be directly applied to multimodal fundus image registration. In addition, the time cost of the traditional registration method is high, and it cannot be compared with the deep learning method in terms of registration efficiency. Although deep learning technology has shown great application potential in fundus image registration tasks, the existing deep learning-based registration method research is still in its initial stage, and most of them rely on specific network structures or the registration model is too complex, so there is still a broad space for exploration. Summary of the invention

[0005] To this end, the present invention provides a multimodal fundus image registration method based on intelligent matching of key point pairs to solve the problems raised in the background technology.

[0006] In order to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multimodal fundus image registration method based on intelligent matching of key point pairs, wherein a multimodal fundus image data set is divided and enhanced and then sent to a YOLOv8 neural network for training, and the optimal weight obtained by training is selected for prediction to obtain a prediction result; the prediction result of the YOLOv8 neural network comprises two parts, one part is a test set image with marked key point pairs, bounding box categories and confidence levels; the other part is a text format file containing key point pair categories, bounding box coordinates and key point pair coordinates; three key point pairs are selected according to the prediction results, an affine transformation matrix is ​​calculated from the three key point pairs, and an affine transformation is performed on the image to obtain a registration result; Selection of key point pairs: When calculating the linear transformation matrix, two key point pairs can be selected to calculate the rigid matrix, at least three key point pairs can be selected to calculate the affine matrix, or at least four key point pairs can be selected to calculate the homography matrix. To simplify the operation, at least three key point pairs are usually selected to calculate the affine matrix. To obtain a better registration effect, the distribution of key point pairs should be as dispersed as possible. Select three key point pairs from the text format file. The steps are as follows: S1-1. Set the threshold Thre and set the key points P in the text format file i ∈P for classification, P represents the key point set, each key point P i Contains attributes {Class i , box_x i , box_y i ,box_widthi , box_height i , x i ,y i}, where Class i is the category of the key point pair, box_x i is the horizontal coordinate of the upper left corner of the bounding box, box_y i is the ordinate of the upper left corner of the bounding box, box_width i is the bounding box width, box_height i is the bounding box height, x i is the horizontal coordinate of the key point, y i is the ordinate of the key point, i=0,1,2,…,N, representing N key points; If the horizontal coordinate x of the key point i If the coordinate is less than the threshold Thre, the corresponding key point P i It is classified into the set CFPoints; otherwise, it is classified into the set FFAPoints, and the set CFPoints and FFAPoints form a key point pair; The pseudo code is as follows: / / Traverse all key points in the text format file For i = 0...N do / / If the key point P i x i The coordinate value is less than the threshold Thre If x i <Thre then / / Put the key point information into CFPoints, "←" means put P i Add to the collection CFPoints or collection FFAPoints CFPoints ← P i Else FFAPoints ← P i End End S1-2. Select a key point CFkp from the set CFPoints f As the reference point, calculate the key point CFkp f With other key points CFkp i Euclidean distance dis_CFkp fi , and find the distance to the key point CFkp f The key point with the longest Euclidean distance CFkp m , connect CFkpf With CFkp m , and we get the straight line CFkpl; Then calculate other key points CFkp i The vertical distance to the straight line CFkpl, select the key point CFkp with the largest vertical distance e ; In the set FFAPoints select the key point CFkp f , CFkp m and CFkp e Class with the same keypoint pair i Three key points as FFAkp f , FFAkp m and FFAkp e ; The three key points (CFkp f ,FFAkp f ), (CFkp m , FFAkp m ), (CFkp e , FFAkp e ) will be used to calculate the affine matrix; The pseudo code is as follows: / / Select a reference point CFkpf from CFPoints, assuming that the first point P is selected 0 As a reference point, P 0 Contains attributes {Class 0 , box_x 0 , box_y 0 ,box_width 0 , box_height 0 , x 0 ,y 0} CFkpf = P 0 / / Initialize the maximum distance max_distance = 0 / / Calculate other key points P in CFPoints i to CFkp f Distance, find the farthest point CFkp m For i = 1...N do / / Calculate the Euclidean distance dis_CFkpf between CFkpf and other key points dis_CFkpfi =

[0007] / / If the current dis_CFkp fi If it is greater than max_distance, update max_distance and the maximum distance point CFkp m If dis_CFkp fi >max_distance then max_distance = dis_CFkp fi CFkpm = P i End End / / Connect to CFkp f and CFkp m , and we get the straight line CFkpl CFkpl: ,in, ,

[0008] / / Initialize the maximum distance max_verticaldis = 0 / / Calculate other key points P i The distance to the straight line CFkpl, where (i=1,2,…N, i≠m), find the farthest point CFkpe For i = 1...N do / / Calculate the vertical distance from other key points in CFPoints to CFkpl dis_CFkp_line i dis_CFkp_line i =

[0009] / / If the current dis_CFkp_line i If it is greater than max_verticaldis, update max_verticaldis and the maximum vertical distance point CFkp e If dis_CFkp_line i >max_verticaldis then max_verticaldis = dis_CFkp_line i FqV e =P i End End / / Select the point with CFkp from FFAPoints f , CFkp m , CFkp e With the same Class i The three key points of (i = f, m, e) are as follows: f , CFkp m , CFkp e The corresponding category is Class f , Class m , Class e / / Traverse all key points P in FFAPoints j For j = 0…N do / / If the key point P j The categories and key points CFkp f Category matching If Class j = Class f then FFAkpf = P j Else if Class j = Class m then FFA m =P j Else Classj = Class e then FFA e =P j End End / / (CFkp f , FFAkp f ), (CFkp m , FFAkp m ), (CFkp e , FFAkp e ) are three key point pairs for calculating the affine matrix.

[0010] Image registration; based on the selected key point pair (CFkp f , FFAkp f ), (CFkp m , FFAkp m ), (CFkpe , FFAkp e ), transform the affine matrix to achieve image registration.

[0011] Preferably, the steps for making the data set are as follows: The directly obtained multimodal fundus image dataset cannot be directly sent to the neural network for training. It needs to be made into a dataset that conforms to the neural network input format in this task; the multimodal fundus images (such as CF images and FFA images) are scaled to the same size and spliced ​​to obtain the original dataset; the vascular bifurcation points are selected as key points, and several key point pairs are selected on the original dataset. They are framed with bounding boxes of appropriate sizes, and each pair of key point pairs and the corresponding bounding boxes are marked as a category to obtain a picture with key point pairs, bounding boxes and category information, namely, labels; the original dataset and the labels together constitute the dataset.

[0012] Preferably, the steps of data set division and enhancement are as follows: The prepared dataset is divided into training set, validation set and test set in a ratio of 8:1:1, and data enhancement is performed on the training set and validation set, including but not limited to rotation, mirroring and filtering operations, to improve the accuracy of the model prediction results and the generalization ability of the model.

[0013] Preferably, the data-enhanced training set and validation set are sent to the YOLOv8 neural network for training to obtain the optimal weights. The steps are as follows: S4-1. Use CSPDarkNet as the Backbone, including the SPPF module and the C2f module, and send the training set to the Backbone to extract multi-level features; S4-2. The multi-level features from Backbone are further integrated using NAS-FPN (feature pyramid network generated by neural network structure search method) to detect and locate key point pairs at different scales (mainly focusing on coarse-grained, medium-grained, and fine-grained information on the image at three scales: large, medium, and small); S4-3. Output the classification results and position coordinates of the key point pairs on the verification set, and save the optimal weights of the YOLOv8 neural network during the verification process to obtain the optimal prediction results.

[0014] Preferably, the formula for affine matrix transformation is as follows: ; in, is the new coordinate of the moving image after affine transformation, is the coordinate of the moving image before affine transformation, is the affine transformation matrix calculated based on the selected key point pairs, a, b, c, d are the parameters of the affine transformation matrix, which represent the parameters after the rotation, scaling and shearing transformation are combined, respectively. , is the translation amount.

[0015] The present invention has the following advantages: 1. The present invention provides a new solution for multimodal fundus image registration under conditions of large differences in image brightness, contrast, and viewing angle, and is fully applicable to single-modality fundus image registration.

[0016] 2. The present invention can not only achieve high-precision and high-efficiency registration of multimodal fundus images, but also has a simple method and does not require a specific or complex network structure, wherein the neural network only needs to have the ability to detect key points.

[0017] The method proposed in the present invention provides a novel and simple method for realizing multimodal fundus image registration under conditions of large differences in image brightness, contrast, and viewing angle. It realizes fast and accurate registration of fundus images by combining key point pair detection technology and linear transformation technology (such as rigid transformation, affine transformation, homography transformation, etc.). The method is simple and convenient, does not require a specific or complex network structure, and has certain reference significance for other medical / natural image registration fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart for multimodal fundus image registration; Figure 2 Schematic diagram of the original data set (CF-FFA) provided by the present invention; Figure 3 A schematic diagram of the data set labels provided by the present invention; Figure 4 A multimodal fundus image key point pair map predicted by a neural network provided by the present invention; Figure 5 The result diagram of the key point pair selection algorithm provided by the present invention; Figure 6 Color photos of fixed images and moving images provided by the present invention; Figure 7 The result diagram before and after the angiography image registration provided by the present invention; Figure 8 This is a diagram of the key point pair prediction and selection results provided by the present invention. DETAILED DESCRIPTION

[0019] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The multi-modality fundus image registration flow chart of this embodiment is as follows: Figure 1 As shown; Dataset creation: CF images and FFA images are horizontally spliced ​​to obtain the original dataset, such as Figure 2 As shown. Labelme software is used to create labels, and the bifurcation points of blood vessels are selected as key points. Several key point pairs are selected on the original data set, and they are framed with appropriately sized bounding boxes. Each pair of key point pairs and the corresponding bounding box are marked as one category. The data set labels are as follows Figure 3 As shown, key point pairs and bounding boxes of different colors in the figure represent different categories.

[0021] Dataset division and enhancement: The multimodal fundus image data is a private dataset. There are 226 pairs of CF and FFA images in total. After the dataset is made, there are 226 images in total, of which 138 are divided into training sets, 9 are divided into validation sets, and 79 are divided into test sets. The training set and validation set are enhanced by horizontal mirroring and rotation. After enhancement, the training set images are 828 and the validation set images are 54.

[0022] Network training: This embodiment uses the YOLOv8 neural network (but not limited to this type of network) as the key point detection method, sends 828 training sets and 54 validation sets to the neural network for training, and saves the optimal weights.

[0023] Result prediction: Use the optimal weights to predict key point pairs for the 79 test sets. The key point pair prediction results are as follows Figure 4 As shown in the figure, the kpi (i=0,1,2…) is the aforementioned Class i , represents the predicted key point pair category, the number after the category represents the confidence of the prediction result, and the bounding box of the same color represents a pair of key point pairs. As can be seen from the figure, the selected method can accurately detect multiple pairs of key point pairs and provide data support for linear transformations such as rigid transformation, affine transformation, and homography transformation.

[0024] Key point selection: For the convenience of calculation, select kp0 from CFPoints as the reference point CFkp f According to the fifth step of the technical solution, select (CFkp f , FFAkp f), (CFkp m , FFAkp m ), (CFkp e , FFAkp e ) Three key point pairs are used to calculate the affine matrix. The key point pair selection results are as follows Figure 5 As shown in the figure, the key point in the upper right corner of the red font indicates the selected key point, and the red straight line indicates the reference point (CFkp f / FFAkp f ) and the key point with the longest Euclidean distance from the reference point (CFkp m / FFAkp m ) (CFkpl / FFAkpl), the green solid line represents the key point CFkp e / FFAkp e The vertical distance to CFkpl / FFAkpl. The green dashed lines indicate the vertical distances of other key points to CFkpl / FFAkpl.

[0025] Image registration: Image registration is completed by calculating the affine matrix based on the selected key point pairs. The color photos of the fundus fixed image and the moving image, the results of the angiography before and after registration, and the key point pair prediction and selection results are shown in the figure below. Figure 6-Figure 8 As shown. Each row is a registration case; the first column of each row is a fixed image, the second column is a moving image, the third column is the prediction and selection result of key point pairs, the fourth column is the checkerboard display before image registration, and the fifth column is the checkerboard display after registration using the method of this embodiment. By comparing the checkerboard display before and after registration, it can be found that the method of this embodiment can achieve high-precision registration of multimodal fundus images under large difference conditions. At the same time, the fast prediction based on deep learning and the fast calculation of linear transformation make the overall time consumption of image registration low.

[0026] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A multimodal fundus image registration method based on intelligent matching of key point pairs, characterized by: The multimodal fundus image data set is divided and enhanced, and then sent to the YOLOv8 neural network for training, and the optimal weight obtained by training is selected for prediction to obtain the prediction result; the prediction result of the YOLOv8 neural network includes two parts, one part is a test set image with marked key point pairs, bounding box categories and confidence levels; the other part is a text format file containing key point pair categories, bounding box coordinates and key point pair coordinates; three key point pairs are selected according to the prediction results, and the affine transformation matrix is ​​calculated from the three key point pairs, and the image is affine transformed to obtain the registration result; Select three key point pairs from the text format file. The steps are as follows: S1-1. Set the threshold Thre and set the key points P in the text format file i ∈P for classification, P represents the key point set, each key point P i Contains attributes {Class i , box_x i , box_y i ,box_width i , box_height i , x i ,y i }, where Class i is the category of the key point pair, box_x i is the horizontal coordinate of the upper left corner of the bounding box, box_y i is the ordinate of the upper left corner of the bounding box, box_width i is the bounding box width, box_height i is the bounding box height, x i is the horizontal coordinate of the key point, y i is the ordinate of the key point; i=0,1,2,…,N, representing N key points; If the horizontal coordinate x of the key point i If it is less than the threshold Thre, the corresponding key point P i It is classified into the set CFPoints; otherwise, it is classified into the set FFAPoints, and the set CFPoints and FFAPoints form a key point pair; S1-2. Select a key point CFkp from the set CFPoints f As the reference point, calculate the key point CFkp f With other key points CFkp i Euclidean distance dis_CFkp fi , and find the distance to the key point CFkp f The key point with the longest Euclidean distance CFkp m , connect CFkp f With CFkp m , and we get the straight line CFkpl; Then calculate other key points CFkp i The vertical distance to the straight line CFkpl, select the key point CFkp with the largest vertical distance e ; In the set FFAPoints select the key point CFkp f , CFkp m and CFkp e Class with the same keypoint pair i Three key points as FFAkp f ,FFAkp m and FFAkp e ; The three key points (CFkp f ,FFAkp f ), (CFkp m ,FFAkp m ), (CFkp e ,FFAkp e ) is used to calculate the affine matrix; Image registration; based on the selected key point pair (CFkp f ,FFAkp f ), (CFkp m ,FFAkp m ), (CFkp e , FFAkp e ), transform the affine matrix to achieve image registration.

2. The multimodal fundus image registration method based on intelligent matching of key point pairs according to claim 1, characterized in that: The steps for making a multimodal fundus image dataset are as follows: The multimodal fundus images are scaled to the same size and spliced ​​to obtain the original dataset; the vascular bifurcation points are selected as key points, and several key point pairs are selected on the original dataset. They are framed with bounding boxes, and each pair of key point pairs and the corresponding bounding boxes are marked as a category to obtain images with key point pairs, bounding boxes and category information, namely labels; the original dataset and the labels together constitute the dataset.

3. The multimodal fundus image registration method based on intelligent matching of key point pairs according to claim 2, characterized in that: The steps for data set division and enhancement are as follows: The prepared data set is divided into a training set, a validation set, and a test set; and data enhancement is performed on the training set and the validation set, including but not limited to rotation, mirroring, and filtering operations, to obtain the data-enhanced training set and the validation set.

4. The multimodal fundus image registration method based on intelligent matching of key point pairs according to claim 3, characterized in that: The data-enhanced training set and validation set are sent to the YOLOv8 neural network for training to obtain the optimal weights. The steps are as follows: S4-1. Use CSPDarkNet as the Backbone, including the SPPF module and the C2f module, and send the training set to the Backbone to extract multi-level features; S4-2. The multi-level features from Backbone are further integrated using the NAS-FPN network to detect and locate key point pairs at different scales; S4-3. Output the classification results and position coordinates of the key point pairs on the verification set, save the optimal weights of the YOLOv8 neural network during the verification process, and obtain the optimal prediction results.

5. The multimodal fundus image registration method based on intelligent matching of key point pairs according to claim 1, characterized in that: The formula for affine matrix transformation is as follows: ； in, is the new coordinate of the moving image after affine transformation, is the coordinate of the moving image before affine transformation, is the affine transformation matrix calculated based on the selected key point pairs, a, b, c, d are the parameters of the affine transformation matrix, which represent the parameters after the rotation, scaling and shearing transformation are combined, respectively. , is the translation amount.

Images