Angiogram image registration method based on AKAZE and ResNet, storage medium and terminal

Abstract

The application discloses a kind of angiogram image registration method based on AKAZE and ResNet, storage medium and terminal.The image registration method includes using AKAZE algorithm to identify the feature points of the image under different scales and rotation angles, direction assignment and M-LDB descriptor generation are carried out to each detected feature point, the classification output of the ResNet neural network is converted into feature point pair output, the modified ResNet neural network is trained using the data set containing the M-LDB descriptor, feature point pairs are selected from the identified M-LDB descriptor, and the transformation relationship between images is calculated through the feature point pairs.The compatibility problem between AKAZE algorithm and ResNet neural network is well solved, and the error caused by complex attitude changes and noise between images is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image registration, and more specifically, to a method, storage medium, and terminal for angiographic image registration based on AKAZE and ResNet. Background Art

[0002] In the current medical field, vascular interventional surgery is a commonly used treatment method for cardiovascular diseases. Vascular interventional surgery is a medical surgical technique that uses interventional technology (such as catheters) to introduce therapeutic equipment into the vascular system to diagnose and treat diseases. Because in clinical practice, vascular interventional surgery has a high threshold, high difficulty, and a long learning curve for doctors, and during the operation, doctors need to wear lead-clad radiation-proof suits weighing more than ten pounds, which has a great negative impact on the doctor's surgical operation. Therefore, engineers and technicians have developed vascular interventional surgery robots to assist doctors in performing vascular interventional surgery.

[0003] When using a vascular interventional surgical robot for surgical operations, imaging assistance is often required. Angiography images are one of the important sources of information. During the specific surgical execution, it is necessary to analyze the images in time sequence together, which requires the adoption of a medical image registration strategy. Medical image registration refers to seeking a (or a series of) spatial transformations for a medical image so that it is spatially consistent with the corresponding points on another medical image. This consistency means that the same anatomical point on the human body has the same spatial position in the two matching images. The result of the registration should ensure that all anatomical points on the two images, or at least all points of diagnostic significance and points of interest and attention during surgery, are positionally matched.

[0004] To achieve registration of angiographic images, the Chinese invention patent publication number CN202111458315.7 proposes a deep learning-based method for single-modality medical image registration before and after coronary angiography. Through three steps: training a neural network model, data reading, and image registration, deep learning-based image segmentation technology is applied to single-modality image registration before and after coronary angiography, achieving end-to-end automatic completion of the single-modality registration task of coronary angiography images. However, because different angiographic image sources may have different resolutions, orientations, and sizes, the time-series images may also change due to biological motion such as the human heartbeat. Traditional vascular image registration typically requires the design of specific registration algorithms based on different image features. These algorithms may perform well on some datasets but poorly on others, lacking versatility and robustness. In addition, traditional vascular image registration algorithms are not very accurate and lack stability, making it difficult to register different angiographic images.

[0005] Therefore, developing an angiography image registration method with high accuracy and good versatility is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] The present invention provides an angiography image registration method, storage medium, and terminal based on AKAZE and ResNet. By using the AKAZE algorithm to identify feature points and using ResNet to filter out similar feature points from the identified feature points to obtain feature point pairs, accurate image registration can be achieved, avoiding errors caused by complex posture changes and noise between images, thereby compensating for the shortcomings of the existing technology.

[0007] To achieve the above objectives, the present invention provides, on the one hand, a method for angiographic image registration based on AKAZE and ResNet, characterized in that it comprises the following steps:

[0008] Step S1: pre-processing the acquired angiography image, including the following sub-steps:

[0009] Step S1.1, performing noise reduction processing on the acquired angiography image, applying mean filtering and Gaussian filtering in order to remove noise on the image;

[0010] Step S1.2: performing semantic segmentation on the current frame of the image using a deep learning model, and saving the current frame of the image;

[0011] Step S2: using the AKAZE algorithm to identify feature points of the image at different scales and rotation angles, and performing direction assignment and M-LDB descriptor generation for each detected feature point;

[0012] Step S3: modify the last layer of the ResNet neural network and convert the classification output of the ResNet neural network into a feature point pair output;

[0013] Step S4: training the modified ResNet neural network model using a data set containing the M-LDB descriptor; selecting feature point pairs from the identified M-LDB descriptors, and calculating the transformation relationship between images using the feature point pairs;

[0014] Step S5: performing registration correction on the images using the transformation relationship, thereby achieving accurate alignment between feature points between images;

[0015] The step S3 is performed before the step S4 and is not limited to being performed after the step S1 or the step S2.

[0016] Furthermore, in step S2, the AKAZE algorithm includes the following steps:

[0017] Step S2.1, use nonlinear diffusion filtering to construct the scale space, retain more edge features of the image, and use a fast implicit diffusion algorithm to speed up the construction;

[0018] Step S2.2: Within each scale layer, calculate the determinant and trace of the Hessian matrix to obtain the feature response map;

[0019] Step S2.3: Find the maximum feature point, i.e., the local extreme point, in a group of images of different scales within the same pyramid layer, and perform threshold screening and non-maximum suppression;

[0020] Step S2.4: Assign a direction to each detected feature point and generate an M-LDB descriptor.

[0021] Furthermore, the threshold screening algorithm includes the following steps:

[0022] Step S2.3.1.1. Calculate the response values ​​of all candidate feature points output by the feature point detection algorithm;

[0023] Step S2.3.1.2: Set a threshold, eliminate candidate feature points with response values ​​lower than the threshold, and retain candidate feature points with response values ​​higher than the threshold.

[0024] Furthermore, the non-maximum suppression algorithm includes the following steps:

[0025] Step S2.3.2.1, define a neighborhood window around each candidate feature point;

[0026] Step S2.3.2.2, compare the response values ​​of each candidate feature point with those of other pixels in its neighborhood window: if the response value of the processed candidate feature point is the largest, retain the candidate feature point; otherwise, delete the candidate feature point;

[0027] Step S2.3.2.3. Repeat step S2.3.2.2 until all candidate feature points have been processed.

[0028] Furthermore, the algorithm for generating the M-LDB descriptor includes the following steps:

[0029] Step S2.4.1, selecting an appropriate image layer as the input image in the nonlinear scale space according to the main direction and scale of the feature points;

[0030] Step S2.4.2, calculating the gradient magnitude and direction on the input image, and quantizing them into 8 directions;

[0031] Step S2.4.3: Divide a square area around the feature point and divide it into several small blocks;

[0032] Step S2.4.4: For each of the small blocks, calculate its gradient direction histogram, and compare the histograms of adjacent small blocks or within the same small block according to a predefined pattern to obtain a binary string;

[0033] Step S2.4.5: Connect the binary strings of all small blocks to obtain the final M-LDB descriptor.

[0034] Furthermore, in step S4, a cross-validation method is used to verify the performance of the modified ResNet neural network model.

[0035] Furthermore, in step S4, the modified ResNet neural network model outputs the probability of feature point pairs; the model selects feature point pairs with a probability of more than 90% and calculates the transformation relationship between images to achieve alignment between feature point pairs.

[0036] On the other hand, the present invention provides a computer storage medium having a computer program, wherein the computer program implements the above-mentioned angiography image registration method when executed by a processor.

[0037] In another aspect, the present invention provides an intelligent terminal, characterized by comprising: one or more memories and one or more processors;

[0038] The one or more memories are used to store computer programs;

[0039] The one or more processors are connected to the memory and are configured to run the computer program to perform the angiography image registration method as described above.

[0040] Compared with the prior art, the above invention has the following advantages or beneficial effects:

[0041] (1) The present invention uses the AKAZE algorithm to efficiently identify feature points in angiography images, and uses the ResNet neural network to filter out similar feature points from the identified key points to obtain feature point pairs, thereby achieving accurate image registration and avoiding errors caused by complex posture changes and noise between images;

[0042] (2) The function of the ResNet neural network in the present invention is modified from image classification to feature classification, and the M-LDB descriptor dataset identified by the AKAZE algorithm is used to train the modified ResNet neural network, which effectively solves the compatibility problem between the AKAZE algorithm and the ResNet neural network;

[0043] (3) The algorithm used in the present invention has low computational complexity, which not only saves computing resources but also improves the efficiency of practical applications;

[0044] (4) The present invention is robust: the present invention can not only identify feature points at different scales and rotation angles, but also has high stability and is applicable to complex angiographic images;

[0045] (5) The image registration method proposed in the present invention can help doctors perform real-time image registration during surgery, thereby more accurately identifying the location of blood vessels, shortening the operation time and reducing possible risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention and its features and advantages will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0047] Figure 1 is a flow chart of a method for angiographic image registration according to an embodiment of the present invention;

[0048] Figure 2 This is a diagram showing the effect of feature point extraction according to an embodiment of the present invention;

[0049] Figure 3 This is a feature point extraction effect diagram of another embodiment of the present invention;

[0050] Figure 4 This is a diagram showing the effect of feature point registration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0052] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that well-known algorithms and models are not shown in detail to avoid obscuring the subject matter of the present invention.

[0053] In addition, the order of execution of actions, steps, etc. in the devices and methods shown in the claims, specifications and drawings can be implemented in any order as long as there is no special explicit limitation on the order and the output of the previous processing is not used in the subsequent processing.

[0054] Example 1

[0055] See also Figure 1 This embodiment provides a method for angiography image registration based on AKAZE and ResNet, comprising the following steps:

[0056] Step S1: pre-processing the acquired angiography image, including the following sub-steps:

[0057] Step S1.1: performing noise reduction processing on the acquired angiography image, and applying mean filtering and Gaussian filtering in sequence to remove noise on the image.

[0058] Step S1.2: Use a deep learning model to perform semantic segmentation on the current frame of the image, and save the current frame of the image.

[0059] Step S2: Use the AKAZE algorithm to identify feature points of the image at different scales and rotation angles, and perform direction assignment and M-LDB descriptor generation on each detected feature point.

[0060] The Accelerated-KAZE (AKAZE for short) algorithm is a computer vision algorithm based on feature detection and descriptor matching. It is an accelerated version of the KAZE algorithm. The KAZE algorithm is an algorithm for local feature extraction and description. It uses a nonlinear diffusion filter (Nonlinear Diffusion Filtering) to construct a scale space and retain the edge information of the image. Compared with KAZE, AKAZE uses the fast implicit diffusion (FED) algorithm to speed up the construction of the nonlinear scale space and improve the calculation speed. At the same time, AKAZE uses an improved feature detection method based on the modified Hessian matrix determinant, replacing the Laplacian function, and enhancing the response and repeatability of feature points. The AKAZE algorithm can detect feature points and descriptors in images with advantages such as rotation invariance, scale invariance, and illumination invariance.

[0061] As a preferred technical solution, further: the AKAZE algorithm includes the following steps:

[0062] Step S2.1, use nonlinear diffusion filtering to construct the scale space, retain more edge features of the image, and use a fast implicit diffusion algorithm to speed up the construction;

[0063] Step S2.2: Within each scale layer, calculate the determinant and trace of the Hessian matrix to obtain the feature response map;

[0064] Step S2.3: Find the maximum feature point, i.e., the local extreme point, in a group of images of different scales within the same pyramid layer, and perform threshold screening and non-maximum suppression;

[0065] Step S2.4: Assign a direction to each detected feature point and generate an M-LDB descriptor.

[0066] The threshold screening algorithm includes the following steps:

[0067] Step S2.3.1.1. Calculate the response values ​​of all candidate feature points output by the feature point detection algorithm;

[0068] Step S2.3.1.2: Set a threshold to eliminate candidate feature points with response values ​​below the threshold and retain those with response values ​​above the threshold. As an example, the response value calculation formula for this step is defined as: the sum of the grayscale differences between the candidate feature point and the surrounding 16 pixels. In this embodiment, the threshold is set to 10.

[0069] The non-maximum suppression algorithm includes the following steps:

[0070] Step S2.3.2.1, define a neighborhood window around each candidate feature point;

[0071] Step S2.3.2.2, compare the response values ​​of each candidate feature point with those of other pixels in its neighborhood window: if the response value of the processed candidate feature point is the largest, retain the candidate feature point; otherwise, delete the candidate feature point;

[0072] Step S2.3.2.3. Repeat step S2.3.2.2 until all candidate feature points have been processed.

[0073] The M-LDB descriptor generation algorithm is an improved local difference binary descriptor with rotation invariance and scale invariance, which improves the robustness of feature extraction. The M-LDB descriptor generation algorithm includes the following steps:

[0074] Step S2.4.1, select an appropriate image layer as the input image in the nonlinear scale space according to the main direction and scale of the feature points;

[0075] Step S2.4.2, calculating the gradient magnitude and direction on the input image, and quantizing them into 8 directions;

[0076] Step S2.4.3: Divide a square area around the feature point and divide it into several small blocks;

[0077] Step S2.4.4: For each of the small blocks, calculate its gradient direction histogram, and compare the histograms of adjacent small blocks or within the same small block according to a predefined pattern to obtain a binary string;

[0078] Step S2.4.5: Connect the binary strings of all small blocks to obtain the final M-LDB descriptor.

[0079] Figure 2 and Figure 3Figure 1 shows the results of feature point extraction for different angiography images using the MLDB descriptor size of 0, the number of channels of 2, the threshold of 0.001, the number of scale space layers of 4, and the number of sublayers per layer of 4. This demonstrates that the AKAZE algorithm can effectively identify feature points in angiography images. In step S2, the average feature point recognition time for each image is 0.5 seconds, demonstrating the high efficiency of the AKAZE algorithm in identifying feature points in angiography images.

[0080] Step S3: Modify the last layer of the ResNet neural network and convert the classification output of the ResNet neural network into feature point pair output.

[0081] The ResNet neural network is a neural network model used for image classification. Here, we modify it to be used for feature point classification. For feature point classification, we modify the last layer of the ResNet neural network to convert it from classification output to feature point pair output.

[0082] Step S4: Use the data set containing the M-LDB descriptor to train the modified ResNet neural network model; select feature point pairs from the identified M-LDB descriptors, and calculate the transformation relationship between images through the feature point pairs.

[0083] Specifically, the model is first trained using the dataset prepared in step S2 and the modified ResNet neural network model. Cross-validation is used to verify the model's performance. Feature points are then input into the model, which outputs the probability of feature point pairs. For feature point pairs with a probability of 90% or greater, the model calculates the transformation relationship between the images.

[0084] Cross-validation is a statistical method used to evaluate model performance. It can maximize effective information from limited data, reduce the risk of overfitting, and select the most appropriate model and parameters. The basic idea of ​​cross-validation is to divide the dataset into training and test sets, train and test the model multiple times using different partitioning methods, and then calculate the average error or accuracy of the model on different test sets as the evaluation metric for model performance.

[0085] There are many forms of cross-validation. This embodiment adopts the leave-one-out method: only one data is used as the test set each time, and all other data are used as the training set. This is repeated n times (n is the size of the data set) to obtain n errors or accuracy rates, and then the average is taken.

[0086] The transformation relationship between the images is the spatial transformation relationship obtained using the homography matrix, which includes the following steps:

[0087] Step S11, the feature point pair x1 and x2 on two images with different viewing angles can be represented by x1=H*x2;

[0088] Step S12: Solve the homography matrix H to obtain the spatial transformation relationship; the homography matrix is ​​a non-singular matrix that can describe the mapping relationship between points on the same plane in different pixel coordinate systems.

[0089] Step S5: perform registration correction on the images according to the transformation relationship, so as to achieve accurate alignment between the feature points between the images. Figure 4 ,The model matches angiography images at different times based on the calculated ,transformation relationship between images, thereby achieving registration between ,feature point pairs.

[0090] From the above embodiments, it can be seen that by using the AKAZE algorithm to identify feature points of angiography images and using the ResNet neural network to filter out similar feature points from the identified key points to obtain feature point pairs, accurate image registration can be achieved, avoiding errors caused by complex posture changes and noise between images; the function of the ResNet neural network is modified from image classification to feature classification, and the M-LDB descriptor dataset identified by the AKAZE algorithm is used to train the modified ResNet neural network, which effectively solves the compatibility problem between the AKAZE algorithm and the ResNet neural network. The algorithm adopted by the present invention has low computational complexity, which not only saves computing resources but also improves the efficiency of practical applications; it can not only identify feature points at different scales and rotation angles, but also has high stability and is suitable for complex angiography images.

[0091] Example 2

[0092] This embodiment provides a computer storage medium having a computer program. When the computer program is executed by a processor, the angiography image registration method as described in Example 1 is implemented.

[0093] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be performed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (Compact Disc Read-Only Memory), a magneto-optical disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a magnetic card or an optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions. The computer-readable storage medium may be a product that is not connected to a computer device, or it may be a component that is connected to a computer device for use.

[0094] Example 3

[0095] This embodiment provides an intelligent terminal, including: one or more memories and one or more processors;

[0096] The one or more memories are used to store computer programs;

[0097] The one or more processors are connected to the memory and are used to run the computer program to perform the angiography image registration method as described in Example 1.

[0098] Optionally, the memory may include, but is not limited to, high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices; the processor 42 may include, but is not limited to, a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] Optionally, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0100] In summary, the application provides an angiogram image registration method based on AKAZE and ResNet, a storage medium and a terminal. The image registration method comprises the steps of identifying feature points of the image under different scales and rotation angles using the AKAZE algorithm, assigning a direction to each detected feature point and generating an M-LDB descriptor, converting the classification output of the ResNet neural network into a feature point pair output, training a modified ResNet neural network using a data set containing the M-LDB descriptor, selecting a feature point pair from the identified M-LDB descriptor, and calculating the transformation relationship between the images through the feature point pair. The application solves the compatibility problem between the AKAZE algorithm and the ResNet neural network, and avoids errors caused by complex attitude changes and noise between images.

[0101] Those skilled in the art should understand that those skilled in the art can realize variations in combination with the prior art and the above-mentioned embodiments, which are not described in detail here. Such variations do not affect the essential content of the application and are not described in detail here.

[0102] The preferred embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and that the equipment and structures not fully described should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications to the technical solutions of the application, or modify them as equivalent embodiments, without departing from the scope of the technical solutions of the application, which does not affect the essential content of the application. Therefore, any simple modification, equivalent change and modification of the above embodiments made in accordance with the technical essence of the application, without departing from the content of the technical solutions of the application, still belongs to the scope of protection of the technical solutions of the application.

[0103] Those skilled in the art can realize that the units of each example described in combination with the present embodiment, i.e. the algorithm steps, can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solutions. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered as beyond the scope of the present application.

Claims

1. A method for angiography image registration based on AKAZE and ResNet, characterized in that: The steps include: Step S1: pre-processing the acquired angiography image, including the following sub-steps: Step S1.1, performing noise reduction processing on the acquired angiography image, applying mean filtering and Gaussian filtering in order to remove noise on the image; Step S1.2: performing semantic segmentation on the current frame of the image using a deep learning model, and saving the current frame of the image; Step S2: using the AKAZE algorithm to identify feature points of the image at different scales and rotation angles, and performing direction assignment and M-LDB descriptor generation for each detected feature point; Step S3: modify the last layer of the ResNet neural network and convert the classification output of the ResNet neural network into a feature point pair output; Step S4: training the modified ResNet neural network model using a data set containing the M-LDB descriptor; selecting feature point pairs from the identified M-LDB descriptors, and calculating the transformation relationship between images using the feature point pairs; Step S5: performing registration correction on the images using the transformation relationship, thereby achieving accurate alignment between feature points between images; The step S3 is performed before the step S4 and is not limited to being performed after the step S1 or the step S2.

2. The angiography image registration method based on AKAZE and ResNet according to claim 1, characterized in that: In step S2, the AKAZE algorithm includes the following steps: Step S2.1, use nonlinear diffusion filtering to construct the scale space, retain more edge features of the image, and use a fast implicit diffusion algorithm to speed up the construction; Step S2.2: Within each scale layer, calculate the determinant and trace of the Hessian matrix to obtain the feature response map; Step S2.3: Find the maximum feature point, i.e., the local extreme point, in a group of images of different scales within the same pyramid layer, and perform threshold screening and non-maximum suppression; Step S2.4: Assign a direction to each detected feature point and generate an M-LDB descriptor.

3. The angiography image registration method based on AKAZE and ResNet according to claim 2, characterized in that: The threshold screening algorithm includes the following steps: Step S2.3.1.

1. Calculate the response values ​​of all candidate feature points output by the feature point detection algorithm; Step S2.3.1.2: Set a threshold, eliminate candidate feature points with response values ​​lower than the threshold, and retain candidate feature points with response values ​​higher than the threshold.

4. The angiography image registration method based on AKAZE and ResNet according to claim 2, characterized in that: The algorithm of non-maximum suppression comprises the following steps: Step S2.3.2.1, define a neighborhood window around each candidate feature point; Step S2.3.2.2, compare the response values ​​of each candidate feature point with those of other pixels in its neighborhood window: if the response value of the processed candidate feature point is the largest, retain the candidate feature point; otherwise, delete the candidate feature point; Step S2.3.2.

3. Repeat step S2.3.2.2 until all candidate feature points have been processed.

5. The angiography image registration method based on AKAZE and ResNet according to claim 2, characterized in that: The algorithm for generating the M-LDB descriptor includes the following steps: Step S2.4.1, select an appropriate image layer as the input image in the nonlinear scale space according to the main direction and scale of the feature points; Step S2.4.2, calculating the gradient magnitude and direction on the input image, and quantizing them into 8 directions; Step S2.4.3: Divide a square area around the feature point and divide it into several small blocks; Step S2.4.4: For each of the small blocks, calculate its gradient direction histogram, and compare the histograms of adjacent small blocks or within the same small block according to a predefined pattern to obtain a binary string; Step S2.4.5: Connect the binary strings of all small blocks to obtain the final M-LDB descriptor.

6. The angiography image registration method based on AKAZE and ResNet according to claim 1, characterized in that: In step S4, a cross-validation method is used to verify the performance of the modified ResNet neural network model.

7. The angiography image registration method based on AKAZE and ResNet according to claim 6, characterized in that: In step S4, the modified ResNet neural network model outputs the probability of feature point pairs; the model selects feature point pairs with a probability of more than 90% and calculates the transformation relationship between images to achieve alignment between feature point pairs.

8. A computer storage medium having a computer program, characterized in that: When the computer program is executed by a processor, the angiography image registration method according to any one of claims 1 to 7 is implemented.

9. An intelligent terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are configured to run the computer program to perform the angiography image registration method according to any one of claims 1 to 7.

Images