Multi-modal retina fundus image registration method and system

Abstract

The invention relates to the technical field of image processing, in particular to a multi-mode retina fundus image registration method and system. The reference image structural body passes through a first feature encoder, and reference rotation isovariant features of four stages and a total reference rotation isovariant feature graph are extracted; the floating image structural body passes through a second feature encoder, and floating rotation isovariant features of four stages and a total floating rotation isovariant feature graph are extracted; respectively inputting the reference rotation equivariant features and the floating rotation equivariant features of the four stages into a key point detection decoder for feature fusion, and outputting a reference key point prediction map and a floating key point prediction map; respectively inputting the total reference rotation equivariant feature pattern and the total floating rotation equivariant feature pattern into a key point descriptor decoder, and outputting a descriptor of each reference key point and a descriptor of each floating key point; and aligning the floating image to the reference image through a registration module to obtain a registered image. According to the invention, the multi-modal image registration precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-modal retinal fundus image registration method and system. Background Art

[0002] Image registration refers to the process of aligning similar structures or features in multiple images to the same coordinate system. Medical image registration plays an important role in the field of medical image processing and analysis. The registration of multimodal fundus images (such as fundus color photography and fluorescein fundus angiography (FFA) images, fundus color photography and optical coherence tomography angiography (OCTA) images) is conducive to the fusion of information from different imaging modalities. Blood vessels have obvious consistent structures in fundus images of different modalities and are often used to extract feature detection of multimodal fundus images.

[0003] At present, research work at home and abroad has proposed a variety of fundus image registration methods. For example, the U-Net network not only performs well in image segmentation tasks, but can also be used as a key point feature extraction network in image registration. SuperPoint is a feature point detection and description method based on deep learning. It can use the backbone network U-Net to quickly and accurately detect key points and generate corresponding local feature descriptors. These feature points and descriptors can be used to match similar areas between different images, thereby achieving image registration. The SuperRetina key point detection network adds progressive key point expansion on the basis of SuperPoint, which greatly reduces the network's demand for the number of key point gold standards, and the key points extracted by the network have certain rotational equivariance. After the key point feature extraction network obtains the key points of the image pair to be registered, an affine matrix is ​​usually fitted based on the key points. Therefore, it is a non-end-to-end rigid registration method.

[0004] However, the single-stream U-Net structure can only receive image features of a single modality during training. Therefore, it cannot recognize and correspond to the differences between different modalities when processing images of different modalities, resulting in the network being unable to effectively align images of another modality. In addition, the U-Net network mainly relies on convolutional layers for feature extraction. However, in multimodal fundus images, due to the high blood vessel density and strong background noise, the key points in the image have low contrast in local areas and are difficult to accurately separate from the background. The convolutional layer is difficult to effectively capture the detailed features in the image, especially the key point area. As a result, the U-Net network structure often finds it difficult to effectively learn key point features when processing multimodal fundus images, resulting in reduced image registration accuracy. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the defects that the existing U-Net network cannot effectively align images of different modalities and it is difficult to effectively learn key point features, which reduces the image alignment accuracy.

[0006] In order to solve the above technical problems, the present invention provides a multimodal retinal fundus image registration method, comprising the following steps: Constructing a key point detection model, the key point detection model comprising: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution, group normalization, and Relu activation function connected in sequence along the forward propagation direction; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map; Copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Copy the floating image A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; The reference rotation equivariant features of the four stages are input into the key point detection decoder for feature fusion, and the reference key point prediction map is output; The floating rotation equivariant features of the four stages are input into the key point detection decoder for feature fusion, and the floating key point prediction map is output; The total reference rotation equivariant feature map and the total floating rotation equivariant feature map are input into the key point descriptor decoder respectively, and the descriptor of each reference key point and the descriptor of each floating key point are output; The reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points are input into the registration module, and the registered image is output.

[0007] Preferably, the rotation equivariant features of the four stages are input into a key point detection decoder, feature fusion is performed, and a key point prediction map is output, including: The rotation equivariant features of the fourth stage are sequentially passed through the group equivariant dynamic snake convolution module and group upsampling to obtain the key point decoding features of the fourth level; After channel concatenation of the key point decoding features of the fourth level and the rotation equivariant features of the third stage, the key point decoding features of the third level are obtained by sequentially passing through the group equivariant dynamic snake convolution module and group upsampling; After channel concatenation of the key point decoding features of the third level and the rotation equivariant features of the second stage, the key point decoding features of the second level are obtained by sequentially passing through the group equivariant convolution module and group upsampling; After channel concatenation of the key point decoding features of the second level and the rotation equivariant features of the first stage, the key point decoding features of the first level are obtained; For the first level of key point decoding features, the feature is After all the features in the dimension are stacked to the channel dimension, they pass through the fully connected layer and the sigmoid activation function in sequence to obtain the key point prediction map.

[0008] Preferably, the input features of the group equivariant dynamic snake convolution module are processed by a feature enhancement branch to obtain enhanced features; The input features of the group equivariant dynamic snake convolution module are processed by the group equivariant dynamic snake convolution block to capture spatial information in different directions and obtain the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction; After the enhanced features, the horizontal group equivariant output features and the vertical group equivariant output features are concatenated along the channel dimension, they are processed in sequence through the third group equivariant convolution, group normalization and ReLU activation function to obtain the output features of the group equivariant dynamic snake convolution module.

[0009] Preferably, the input features of the group equivariant dynamic snake convolution module are passed through a group equivariant dynamic snake convolution block to capture spatial information in different directions, and obtain group equivariant output features in the horizontal direction and group equivariant output features in the vertical direction, including: The input features of the group equivariant dynamic snake convolution module are passed through The second group of equivariant convolution, group normalization and tanh activation function of the convolution kernel respectively obtains the horizontal bias feature and the vertical bias feature; wherein the value of each pixel in the horizontal bias feature represents the offset in the horizontal direction, and the value of each pixel in the vertical bias feature represents the offset in the vertical direction; According to the value of each pixel in the horizontal bias feature, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the horizontal direction, and the horizontal bias input feature is obtained through four-neighborhood interpolation; According to the value of each pixel in the bias feature in the vertical direction, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the vertical direction, and the bias input feature in the vertical direction is obtained through four-neighborhood interpolation; The horizontal bias input features are sequentially passed through the horizontal direction The convolution kernel, group normalization and Relu activation function are used to obtain the horizontal feature map; among them, To set parameters; The vertical offset input features are passed through the vertical direction in sequence The convolution kernel, group normalization and Relu activation function are used to obtain the feature map in the vertical direction; The horizontal feature map and the vertical feature map are respectively The dimensions are integrated to obtain the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction.

[0010] Preferably, the total rotation equivariant feature map is input into a key point descriptor decoder, and the descriptor of each key point is output, including: Based on the key point prediction map, the coordinates of each key point are obtained; according to the coordinates of each key point, the pixel value of each channel corresponding to the position of the key point in the total rotation equivariant feature map is extracted to construct the initial descriptor of each key point; The initial descriptor of each key point is passed through batch normalization, 1×1 convolution layer, Sigmoid activation function, 3×3 convolution layer and fully connected layer in sequence to obtain the descriptor of each key point.

[0011] Preferably, the step of inputting the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into a registration module and outputting the registered image comprises: The reference key point prediction map and the descriptors of all reference key points, the floating key point prediction map and the descriptors of all floating key points are matched by the Flann algorithm to obtain the key point pairs between the reference image and the floating image; The key point pairs between the reference image and the floating image are fitted using a random sampling consensus algorithm, and the key point pairs that do not meet the transformation constraints are eliminated to obtain an affine matrix; The floating image is aligned to the reference image using an affine matrix to obtain the registered image.

[0012] Preferably, the reference image is a fundus color photograph, and the floating image is any one of a fundus fluorescein angiography image and an optical coherence tomography angiography image.

[0013] Preferably, the training method of the key point detection model is: Acquire an image dataset, wherein the image dataset includes different images and a plurality of corresponding gold standard key points; divide the image dataset into a training set, a validation set, and a test set; The key point detection model is trained using the training set, and the parameters of the key point detection model are optimized using the total loss function. The training process includes: After obtaining the descriptors of each key point, freeze the network parameters, perform a random affine transformation on the image of this round, re-input the image after affine transformation into the key point detection model, and obtain the key point prediction map corresponding to the image of this round after affine transformation. and the descriptor of each key point, calculate the key point prediction map corresponding to the image in this round before and after the affine transformation The Euclidean distance of the descriptor of the key point Key point prediction map corresponding to the current round image before affine transformation Middle The descriptor of the key points and the prediction map of the key points corresponding to the image in this round after affine transformation Middle The Euclidean distance between the descriptors of the next nearest neighbor key points of a key point ,when When the ratio is greater than the set threshold, the key point is set as the gold standard key point as the gold standard key point for the next round; After the key point detection model is trained, the key points detected by the key point detection model are registered with the descriptors through the registration module.

[0014] Preferably, the total loss function includes: a Dice loss function and a ternary loss function; The calculation formula of the total loss function is: , , , in, is the total loss function of the key point detection model, is the Dice loss function, is the ternary loss function, is the key point prediction map, Rotate the image Key point prediction map corresponding to the angle, is the soft label obtained by the Gaussian Boolean function of the key point graph of the gold standard, is element-wise multiplication, is the maximum value function, For the image Descriptors of key points and image rotation The corresponding angle The Euclidean distance between the descriptors of key points, is the threshold for dividing positive samples and negative samples, For the image Descriptors of key points and image rotation Any other key point corresponding to the angle The Euclidean distance between the descriptors of For the image Descriptors of key points and image rotation The maximum Euclidean distance between the descriptors of all other key points corresponding to the angle.

[0015] The present invention also provides a multimodal retinal fundus image registration system, comprising: A model construction module is used to construct a key point detection model, wherein the key point detection model includes: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution, group normalization, and Relu activation function connected in sequence along the forward propagation direction; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map; Reference rotation equivariant feature acquisition module, used to copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Floating rotation equivariant feature acquisition module, used to copy floating images A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; The reference key point prediction map acquisition module is used to input the reference rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the reference key point prediction map; The floating key point prediction map acquisition module is used to input the floating rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the floating key point prediction map; A key point descriptor acquisition module is used to input the total reference rotation equivariant feature map and the total floating rotation equivariant feature map into a key point descriptor decoder, and output a descriptor of each reference key point and a descriptor of each floating key point; The image registration module is used to input the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module, and output the registered image.

[0016] The above technical solution of the present invention has the following beneficial effects compared with the prior art: The multimodal retinal fundus image registration method and system described in the present invention respectively extracts features of a reference image and a floating image through a first feature encoder and a second feature encoder, thereby solving the defect that a traditional single-stream structure can only receive image features of a single modality in cross-modal registration; and replaces the traditional convolution layer with a group equivariant dynamic serpentine convolution module, each group equivariant dynamic serpentine convolution module comprising: a feature enhancement branch, a group equivariant dynamic serpentine convolution block, a third group equivariant convolution, group normalization, and a Relu activation function; the feature enhancement branch comprises: a first group equivariant convolution, group normalization, and a Relu activation function sequentially connected along the forward propagation direction; the feature enhancement branch processes and enhances the input features through the first group equivariant convolution, group normalization, and a Relu activation function, thereby improving the key information expression capability in the image and obtaining enhanced features; the group The equivariant dynamic snake convolution block significantly improves the robustness and accuracy of the model to spatial transformations between images (such as translation, rotation, etc.) through offset modeling and directional processing. By extracting features in the horizontal and vertical directions respectively, it helps the feature encoding module capture the directional information in the image, improves the feature extraction ability of local key point areas, and thus improves the image registration accuracy and robustness, and obtains the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction; by splicing the enhanced features, the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction along the channel dimension, and then processing them in turn through the third group equivariant convolution and group normalization, the output of the group equivariant dynamic snake convolution module is obtained; the separation of the feature encoder for low-contrast key point areas is effectively improved, thereby improving the registration accuracy of multimodal retinal fundus images.

[0017] In addition, the present invention inputs the reference rotation equivariant features of the four stages and the floating rotation equivariant features of the four stages into the key point detection decoder respectively, performs feature fusion, outputs the reference key point prediction map and the floating key point prediction map, and sequentially passes the rotation equivariant features of the fourth stage through the group equivariant dynamic serpentine convolution module and the group upsampling to obtain the key point decoding features of the third level. In order to reduce the overfitting phenomenon caused by the network being too deep, the key point decoding features of the third level are channel-spliced ​​with the rotation equivariant features of the second stage, and sequentially pass through the group equivariant convolution module and the group upsampling to obtain the key point decoding features of the second level; the key point decoding features of the second level are channel-spliced ​​with the rotation equivariant features of the first stage through the group upsampling to obtain the key point decoding features of the first stage; for the key point decoding features of the first stage, the feature is After all the features in the dimension are stacked to the channel dimension, they are passed through the fully connected layer and the sigmoid activation function in turn to obtain the key point prediction map. The key point detection decoder proposed in the present invention uses group equivariant dynamic snake convolution blocks and group equivariant convolution modules to perform channel splicing and high-dimensional feature stacking, so that the model can better capture multi-scale and multi-directional feature information, and make the model more rotationally robust to rotational transformations, thereby improving the robustness and accuracy of the key point detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 It is the structural diagram of the key point detection model and registration module.

[0019] Figure 2 It is the structural diagram of the feature encoder.

[0020] Figure 3 It is the structural diagram of the key point detection decoder.

[0021] Figure 4 It is the structural diagram of the key point descriptor decoder.

[0022] Figure 5 It is the structural diagram of the group equivariant dynamic snake convolution module.

[0023] Figure 6 It is a structural diagram of the group equivariant dynamic snake convolution block.

[0024] Figure 7 It is a registration chessboard diagram of fundus color photography and fluorescein fundus angiography images. Figure 7 (a) is a color fundus photograph. Figure 7 (b) is the fluorescein fundus angiography image. Figure 7 (c) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through scale-invariant feature transformation. Figure 7 (d) is the registration result of fundus color photo and fluorescein fundus angiography image through Orb. Figure 7 (e) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the Harris corner detector. Figure 7 (f) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the SuperRetina key point detection network. Figure 7 (g) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the dynamic snake network. Figure 7(h) in the figure is the registration result of the fundus color photograph and the fluorescein fundus angiography image through the key point detection model.

[0025] Figure 8 It is a registration chessboard diagram of fundus color photography and optical coherence tomography angiography images. Figure 8 (a) is a color fundus photograph. Figure 8 (b) is an optical coherence tomography image. Figure 8 (c) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through scale-invariant feature transformation. Figure 8 (d) is the registration result of fundus color photo and fluorescein fundus angiography image through Orb. Figure 8 (e) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the Harris corner detector. Figure 8 (f) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the SuperRetina key point detection network. Figure 8 (g) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the dynamic snake network. Figure 8 (h) in the figure is the registration result of the fundus color photograph and the fluorescein fundus angiography image through the key point detection model. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0027] Embodiment 1 of the present invention provides a multimodal retinal fundus image registration method, including: Step S11: construct a key point detection model (G_DSCNet), wherein the key point detection model includes: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; like Figure 1 As shown, Figure 1 This is the structural diagram of the key point detection model and registration module.

[0028] like Figure 2 As shown, Figure 2It is a structural diagram of the feature encoder. Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution connected in sequence along the forward propagation direction, group normalization, Relu activation function; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map.

[0029] Step S12: Copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Step S13: Copy the floating image A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; In this embodiment, specifically, the reference image is a fundus color photograph, and the floating image is any one of a fundus fluorescein angiography image and an optical coherence tomography angiography image.

[0030] Step S14: input the reference rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output a reference key point prediction map; Step S15: input the floating rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output a floating key point prediction map; like Figure 3 As shown, Figure 3 In this embodiment, preferably, the rotation equivariant features of the four stages are input into the key point detection decoder, feature fusion is performed, and a key point prediction map is output, including: Step 151: Rotational equivariant features of the fourth stage The key point decoding features of the fourth level are obtained by sequentially passing through the group equivariant dynamic snake convolution module and group upsampling. ; Step 152: Decode the key points of the fourth level Rotational equivariance with the third stage After channel splicing, the key point decoding features of the third level are obtained by passing through the group equivariant dynamic snake convolution module and group upsampling in sequence. ; Step 153: Decode the key points of the third level Rotational equivariance with the second stage After channel splicing, the key point decoding features of the second level are obtained by passing through the group equivariant convolution module and group upsampling in sequence. ; Step 154: Decode the key points of the second level Rotational equivariance with the first stage After channel splicing, the key point decoding features of the first level are obtained; Step 155: Decode the key point features of the first level and convert them into After all the features in the dimension are stacked to the channel dimension, they pass through the fully connected layer and the sigmoid activation function in sequence to obtain the key point prediction map.

[0031] The key point detection decoder proposed in the present invention utilizes group equivariant dynamic snake convolution blocks and group equivariant convolution modules to perform channel splicing and high-dimensional feature stacking, so that the model can better capture multi-scale and multi-directional feature information and is robust to rotational transformations, thereby improving the robustness and accuracy of the key point detection model.

[0032] Step S16: inputting the total reference rotation equivariant feature map and the total floating rotation equivariant feature map into the key point descriptor decoder respectively, and outputting the descriptor of each reference key point and the descriptor of each floating key point; like Figure 4 As shown, Figure 4 is a structural diagram of the key point descriptor decoder. In this embodiment, specifically, the total rotation equivariant feature map Input key point descriptor decoder, output each key point descriptor, including: Step 161: Based on the key point prediction graph, obtain the coordinates of each key point; according to the coordinates of each key point , extract the pixel value of each channel corresponding to the key point position in the total rotation equivariant feature map, and construct the initial descriptor of each key point , is the number of channels of the total rotation equivariant feature map, For the The initial descriptor of the key points, is the total number of key points; Step 162: The initial descriptor of each key point is passed through batch normalization and 1×1 convolution layer in turn to flatten the initial descriptor of each key point into , through the Sigmoid activation function, 3×3 convolution layer and fully connected layer, the descriptor of each key point is obtained .

[0033] Step S17: input the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module, and output the registered image.

[0034] In this embodiment, preferably, the step of inputting the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module and outputting the registered image comprises: Step S171: The reference key point prediction image and the descriptors of all reference key points, the floating key point prediction image and the descriptors of all floating key points are matched by the Flann algorithm to obtain key point pairs between the reference image and the floating image; Step S172: using a random sampling consensus algorithm to fit the key point pairs between the reference image and the floating image, and removing the key point pairs that do not meet the transformation constraints to obtain an affine matrix; Step S173: using an affine matrix to align the floating image to the reference image to obtain a registered image.

[0035] In the registration of multimodal retinal fundus images, the registration methods based on the Flann algorithm, the Random Sample Consensus Algorithm (RANSAC) and the affine transformation have significant advantages. Retinal fundus images usually have different imaging modes, and the images under these modes may have illumination differences, contrast changes and local deformations. Through efficient Flann feature matching, the corresponding key points can be accurately found between different modalities, avoiding the problem that the traditional matching methods based on pixel intensity or grayscale value are easily disturbed by illumination and contrast changes. The Random Sample Consensus Algorithm (RANSAC) further eliminates false matches, improves the robustness of the registration, and ensures that the final registration results are more accurate. In addition, the affine transformation can effectively handle the rotation, translation, scale transformation and other situations in the image, and ensure the accurate alignment of images of different modalities. Compared with other registration methods, this method can not only stably perform image registration under different modalities, but also has high computational efficiency. It is particularly suitable for scenarios in clinical practice where fast and accurate registration of different types of fundus images is required.

[0036] In this embodiment, in addition to the above-mentioned registration method based on the flann algorithm, the random sampling consensus algorithm (RANSAC) and the affine transformation, a spectrum-based registration method can also be used. The spectrum-based registration method is specifically as follows: Fourier transform is performed on the reference image and the floating image respectively to obtain the spectrum representation of the reference image in the frequency domain and the spectrum representation of the floating image in the frequency domain, the phase difference between the spectrum of the reference image and the floating image is calculated, and the image translation amount is obtained based on the phase difference between the spectrum of the reference image and the floating image; the image translation amount is used to align the floating image to the reference image to obtain the registered image. A convolutional neural network (CNN) can also be used to predict the registration parameters of the reference image and the floating image, and to predict the deformation field or affine matrix between the two images, and the predicted deformation field or affine matrix is ​​used to align the floating image to the reference image to obtain the registered image.

[0037] like Figure 5 As shown, Figure 5 is a structural diagram of a group equivariant dynamic snake convolution module. In this embodiment, preferably, the input feature of the group equivariant dynamic snake convolution module is Through feature enhancement branch processing, enhanced features are obtained ,include: Input features of group equivariant dynamic snake convolution module Pass by The first group of equivariant convolution with convolution kernel and padding step size of 1 expands the number of channels to , after group normalization and Relu activation function, the enhanced features are obtained ;in, is the input feature of the group equivariant dynamic snake convolution module The number of channels after the first group of equivariant convolution expansion.

[0038] The input features of the group equivariant dynamic snake convolution module Through group equivariant dynamic snake convolution block processing, spatial information in different directions is captured to obtain group equivariant output features in the horizontal direction Group equivariant output characteristics with perpendicular direction ; Enhanced features , horizontal group equivariant output characteristics Group equivariant output characteristics with perpendicular direction After splicing along the channel dimension, the output features of the group equivariant dynamic snake convolution module are obtained by sequentially passing through the third group equivariant convolution, group normalization, and Relu activation function. .

[0039] The present invention replaces the traditional convolutional layer in the traditional encoder with a group equivariant dynamic snake convolution module (G_DSC convolution module), adopts group equivariant convolution as the basic convolution kernel, and adopts a pooling layer and an upsampling layer with group equivariance, so that the network has rotation equivariance, ensuring that the attention of the network remains consistent before and after rotation, and can adapt to the rotation change of the input image, ensuring the consistency of the attention of the network before and after rotation. The feature enhancement branch improves the sensitivity of the network to key features and reduces noise interference. By splicing the group equivariant output features in the horizontal and vertical directions with the enhanced features along the channel dimension, after further processing, the group equivariant dynamic snake convolution module (G_DSC convolution module) performs well in terms of rotation equivariance, spatial feature capture and multimodal information fusion.

[0040] like Figure 6 As shown, Figure 6 : is a structural diagram of a group equivariant dynamic snake convolution block. In this embodiment, preferably, the input features of the group equivariant dynamic snake convolution module are passed through a group equivariant dynamic snake convolution block to capture spatial information in different directions, and obtain group equivariant output features in the horizontal direction and group equivariant output features in the vertical direction, including: The input features of the group equivariant dynamic snake convolution module Pass The second group of equivariant convolution, group normalization and tanh activation function of the convolution kernel obtains the horizontal bias features respectively. Offset feature from the vertical direction ; Wherein, the value of each pixel in the horizontal bias feature represents the offset in the horizontal direction, and the value of each pixel in the vertical bias feature represents the offset in the vertical direction; According to the value of each pixel in the horizontal bias feature, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the horizontal direction, and the horizontal bias input feature is obtained through four-neighborhood interpolation. ; According to the value of each pixel in the vertical bias feature, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the vertical direction, and the vertical bias input feature is obtained by four-neighborhood interpolation. ; The horizontal bias input features are sequentially passed through the horizontal direction The convolution kernel, group normalization and Relu activation function are used to obtain the horizontal feature map; among them, To set parameters; The vertical offset input features are passed through the vertical direction in sequence The convolution kernel, group normalization and Relu activation function are used to obtain the vertical feature map ; The horizontal feature map and the vertical feature map are respectively The dimensions are integrated to obtain the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction. .

[0041] Although the dynamic snake convolution (DSCNet) has good flexibility in capturing the spatial structure in the image, the traditional convolution operation lacks inherent adaptability to geometric transformations such as image rotation and scale transformation. When processing rotated, translated or otherwise deformed images, ordinary convolution may cause feature loss or inconsistency, thus affecting the performance of the model. In particular, when faced with large-scale rotation, tilt and other deformations, ordinary convolution cannot automatically adapt to the geometric transformation of the image, which limits the generalization ability of the model.

[0042] The present invention replaces the ordinary convolution in the dynamic snake network with the second group equivariant convolution. Unlike ordinary convolution, the second group equivariant convolution can only process local features of fixed direction and scale. Instead, it can capture features at different rotation angles and directions by introducing group symmetry (such as rotation, translation, reflection and other geometric transformations) in the convolution operation. This enables the network to learn feature representations of multiple directions and scales at the same time, thereby improving the diversity of features and enhancing the adaptability to complex visual patterns (such as rotation, tilt, scaling, etc.). Combined with the flexibility of dynamic snake convolution, the key point detection model can more efficiently capture diverse spatial information and improve the overall expression ability.

[0043] In the first embodiment, the blood vessels of fundus images of different modalities are extracted respectively, and the coordinates of each key point are obtained through the same key point detection decoder, and the descriptors are allocated on the key point coordinates through the rotation and other variability features integrated by the feature encoder. Finally, the flann algorithm and the random sampling consensus algorithm (RANSAC) are used to fit the affine matrix of the detected key points and descriptors to achieve the registration of multi-modal fundus images, which can effectively improve the registration accuracy, especially when processing fundus images with deformations such as rotation and scaling, it can accurately extract and match stable vascular features and enhance the robustness of image registration.

[0044] Based on the above embodiment 1, in embodiment 2 of the present invention, the rotation angle dimension of the group equivariant convolution is is set to 8, the feature encoder is set to four stages, and the number of channels of the rotation equivariant features in each stage is set to 16, 32, 64, and 128 respectively. Set to 9; the reference image or floating image is used as the input image and processed by the feature encoder and key point decoder, specifically including the following steps: Step S21: Input image copy Pass through and get the input image structure ,in, Indicates the batch size, is the initial number of channels of the input image, set to 1, and Represent the height and width of the input image respectively.

[0045] Step S22: Input image structure Through the feature encoder, four stages of rotational equivariant features are extracted Equivariant characteristic map with total rotation , , For the The rotational equivariant characteristics of the stages, For the The number of channels of the rotational equivariant features at each stage, It is the stage index, including: Step S221: Input image structure go through The first group of equivariant convolution with convolution kernel and padding step size of 1 expands the number of channels to After group normalization and Relu activation function, the enhanced features of the first stage are obtained. .

[0046] Step S222: Input image structure After the group equivariant dynamic snake convolution block, the horizontal group equivariant output features of the first stage are obtained. Group equivariant output characteristics with perpendicular direction ,include: Step S2221: Input feature structure go through The second group of equivariant convolution, group normalization and tanh activation function obtain the horizontal and vertical bias features respectively. , ; Step S2222: According to the value of each pixel in the horizontal offset feature, the input feature structure After each pixel is offset in the horizontal direction, the horizontal bias input feature is obtained through four-neighborhood interpolation ; Step S2223: According to the value of each pixel in the vertical bias feature, the input feature structure After each pixel is offset in the vertical direction, the vertical bias input feature is obtained through four-neighborhood interpolation ; Step S2224: Input the horizontal offset into the feature Through the horizontal direction The convolution kernel, group normalization and Relu activation function are used to obtain the horizontal feature map; Step S2225: Input the vertical offset into the feature Through the vertical direction The convolution kernel, group normalization and Relu activation function are used to obtain the feature map in the vertical direction; Step S2226: The horizontal feature map and the vertical feature map are respectively The dimensions are integrated to obtain the horizontal group equivariant output features of the first stage Group equivariant output characteristics with perpendicular direction .

[0047] Step S223: The enhanced features of the first stage , the horizontal group equivariant output characteristics of the first stage and the group equivariant output characteristics in the vertical direction of the first stage Splicing along the channel dimension, the number of channels is Features , Through the third group equivariant convolution of 1×1 convolution kernel and group normalization, the number of channels is increased from Adjust to , we get the rotational equivariant characteristics of the first stage , After group equivariant downsampling, it serves as the input feature of the next group equivariant dynamic snake convolution module.

[0048] Step S224: The rotation equivariant features of other stages except the rotation equivariant features of the first stage are resized by group transpose convolution to make their sizes consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are concatenated along the channel dimension to obtain a total rotation equivariant feature map.

[0049] Step S23: Input the rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output a key point prediction map, including: Step S231: The rotational equivariant features of the fourth stage After group equivariant dynamic snake convolution module and group upsampling, the key point decoding features of the fourth level are obtained ; Step S232: Decode the key points of the fourth level Rotational equivariance with the third stage Perform channel splicing to obtain the third fusion feature ,in, , The third fusion feature The number of channels; Step S233: The third fusion feature The key point decoding features of the third level are obtained by sequentially passing through the group equivariant dynamic snake convolution module and group upsampling. ; Step S234: Decode the key points of the third level Rotational equivariance with the second stage Perform channel splicing to obtain the second fusion feature ,in, , The second fusion feature The number of channels; Step S235: The second fusion feature The key point decoding features of the second level are obtained by sequentially passing through the group equivariant convolution module and group upsampling. ; Step S236: Decode the key points of the second level Rotational equivariance with the first stage Perform channel splicing to obtain the key point decoding features of the first level ,in, , Decode features for key points at the first level The number of channels; Step S237: Decode the key point features of the first level and After all the features in the dimension are stacked to the channel dimension, they are passed through the fully connected layer and the sigmoid activation function in turn to obtain the key point prediction graph .

[0050] In this embodiment 2, the rotation angle dimension of the group equivariant convolution is Set to 8. Dividing the feature encoder into four stages can effectively balance the computational complexity and the performance of the key point detection model. The number of rotational equivariant feature channels in the four stages is set to 16, 32, 64, and 128, which increases exponentially. It can extract increasingly rich feature information at different levels while ensuring that the model's expressive power is gradually enhanced. With the increase in the number of channels, the feature encoder can capture more detailed and high-order image information. Setting it to 9 can capture more extensive contextual information, reduce the depth of the model, and improve the efficiency and accuracy of the key point detection model.

[0051] In this embodiment, preferably, the training method of the key point detection model is: Acquire an image dataset, wherein the image dataset includes different images and a plurality of corresponding gold standard key points; divide the image dataset into a training set, a validation set, and a test set; The key point detection model is trained using the training set, and the parameters of the key point detection model are optimized using the total loss function. The training process includes: After obtaining the descriptors of each key point, freeze the network parameters, perform a random affine transformation on the image of this round, re-input the image after affine transformation into the key point detection model, and obtain the key point prediction map corresponding to the image of this round after affine transformation. and the descriptor of each key point, calculate the key point prediction map corresponding to the image in this round before and after the affine transformation The Euclidean distance of the descriptor of the key point Key point prediction map corresponding to the current round image before affine transformation Middle The descriptor of the key points and the prediction map of the key points corresponding to the image in this round after affine transformation Middle The Euclidean distance between the descriptors of the next nearest neighbor key points of a key point ,when When the ratio is greater than the set threshold, the key point is set as the gold standard key point as the gold standard key point for the next round; After the key point detection model is trained, the key points detected by the key point detection model are registered with the descriptors through the registration module.

[0052] In the multimodal fundus image registration task, the gold standard of key points is scarce and difficult to make, mainly because the complex structure, high noise and differences between different modalities in fundus images make manual annotation of key points very challenging. The present invention expands the gold standard key points based on the Euclidean distance calculation of affine transformation and key point descriptors in each round of training, and screens the key points by ratio judgment, which can effectively avoid mismatching and improve the accuracy of image registration.

[0053] In this embodiment, preferably, the total loss function includes: a Dice loss function and a ternary loss function; The calculation formula of the total loss function is: , , , in, is the total loss function of the key point detection model, is the Dice loss function, is the ternary loss function, is the key point prediction map, Rotate the image Key point prediction map corresponding to the angle, is the soft label obtained by the Gaussian Boolean function of the key point graph of the gold standard, is element-wise multiplication, is the maximum value function, For the image Descriptors of key points and image rotation The corresponding angle The Euclidean distance between the descriptors of key points, is the threshold for dividing positive samples and negative samples, For the image Descriptors of key points and image rotation Any other key point corresponding to the angle The Euclidean distance between the descriptors of For the image Descriptors of key points and image rotation The maximum Euclidean distance between the descriptors of all other key points corresponding to the angle.

[0054] Embodiment 3 of the present invention In order to verify the effectiveness and versatility of the method of the present invention, the method was verified by using multimodal fundus image key point detection and registration tasks such as fundus color photography and fluorescein fundus angiography (FFA) image registration, and fundus color photography and optical coherence tomography angiography (OCTA) image registration.

[0055] In this embodiment, a diabetic retinopathy dataset containing 400 pairs of fundus color photos and fluorescein fundus angiography (FFA) images was used to verify the method. The dataset was divided into 240 pairs of training sets, 80 pairs of validation sets, and 80 pairs of test sets. Online random contrast transformation, brightness transformation, left-right flipping, and up-down flipping were used for data amplification. The success rate of registration was used as an evaluation indicator. All experiments used the flann method to pair key points, and the random sampling consensus algorithm (RANSAC) was used to fit the affine matrix to achieve image registration.

[0056] In the comparative experiment, the method of the present invention is compared with traditional key point detection methods including scale invariant feature transform (SIFT), ORB (Oriented FAST and Rotated BRIEF), Harris corner detector, SuperRetina key point detection network, dynamic snake convolution network (DSCNet) and other key point detection and description networks based on convolutional neural networks. Among them, the dynamic snake convolution network (DSCNet) is a blood vessel segmentation network based on dynamic snake convolution rather than a key point detection and description network. In order to verify the effectiveness of the present invention and ensure fairness, the network structure is applied to the autonomous training of SuperRetina's training strategy.

[0057] Since the modality difference between fundus color photography and fluorescein fundus angiography (FFA) images is small, they can be regarded as images of the same distribution after data enhancement by limited contrast adaptive histogram equalization (CLAHE). Therefore, all deep learning methods use a single-stream encoder structure and only use fundus color photography images for training. Traditional methods and deep learning methods use the same data enhancement method. As shown in Table 1, Table 1 shows the comparative experimental results of key point detection and registration of fundus color photography and fluorescein fundus angiography (FFA) images.

[0058] Table 1

[0059] As shown in Table 1, the traditional key point detection methods, Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), and Harris corner detector can detect more key points (here, SIFT and Orb key points are manually set to 1000, and Harris corner points are set to 200. Due to different implementation methods, the number of Harris corner points is usually less than that of SIFT and Orb key points), and the registration success rates are 36.02%, 30.14%, and 37.02%, respectively. Compared with traditional methods, SuperRetina self-supervisedly learns the key points on the blood vessels and uses the features of the fourth stage of the encoder to extract key points and descriptors. It detects fewer key points, but its robustness and effectiveness are improved, and the registration success rate reaches 45.02%. The dynamic snake convolution network (DSCNet) uses dynamic snake convolution to guide the network to effectively focus on the blood vessel area, reducing the number of key points, but further improving robustness and effectiveness, and the registration success rate is increased to 55.00%. The present invention utilizes the rotation equivariance of group equivariant convolution to ensure that the extracted descriptor features are rotationally robust when the input image is rotated at random angles for self-supervised learning, and integrates the rotation equivariant features of multiple stages of the feature encoder to extract key points and descriptors. Therefore, the robustness of the key points and their descriptors is improved, and the registration success rate reaches 63.00%, which illustrates the effectiveness of the method of the present invention.

[0060] like Figure 7 As shown, Figure 7 This is a chessboard diagram for the registration of fundus color photography and fluorescein fundus angiography images. Figure 7 (a) is a color fundus photograph. Figure 7 (b) is the fluorescein fundus angiography image. Figure 7 (c) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through scale-invariant feature transformation. Figure 7 (d) is the registration result of fundus color photo and fluorescein fundus angiography image through Orb. Figure 7 (e) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the Harris corner detector. Figure 7 (f) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the SuperRetina key point detection network. Figure 7 (g) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the dynamic snake network. Figure 7 (h) in the figure is the registration result of the fundus color photograph and the fluorescein fundus angiography image through the key point detection model. Figure 7The registration results of different methods are shown. The traditional method identifies many key points, but they are not key points of vascular features, so the key point matching effect is poor. SuperRetina and the dynamic snake convolution network (DSCNet) successfully matched the key points and fitted the affine matrix, but from the chessboard diagram, it can be seen that the overlapping parts of the blood vessels appear connected, while the other parts do not overlap and appear interrupted. The final affine matrix has a large deviation. The key point detection model (G_DSCNet) proposed in the present invention has the highest blood vessel overlap and the most connected areas.

[0061] This embodiment also uses a data set containing 100 pairs of pathological myopia fundus color photographs and optical coherence tomography angiography (OCTA) images to evaluate the performance of the present invention. Due to the small number of samples, a five-fold cross validation method was used. Online random contrast transformation, brightness transformation, left-right flipping and up-down flipping were used for data amplification. The number of key points detected and the success rate of registration were used as evaluation indicators. All experiments used the flann method to pair key points, and the random sampling consensus algorithm (RANSAC) was used to fit the affine matrix to achieve image registration.

[0062] In the comparative experiment, all deep learning networks use dual-stream encoders. However, due to the large difference in modality, the vascular density of optical coherence tomography angiography (OCTA) images is much higher than that of fundus color photography and there is noise. The image distribution is also completely different. Therefore, the traditional key point detection algorithm cannot correctly fit the affine matrix and apply it to registration. The results are shown in Table 2, which shows the comparative experimental results of key point detection and registration of fundus color photography and optical coherence tomography angiography (OCTA) images.

[0063] Table 2

[0064] As shown in Table 2, for SuperRetina, because it uses the original U-type codec structure, it is not possible to effectively learn the vascular features of OCTA data when there are few training samples. The key point detection model (G_DSCNet) proposed in this invention retains the effectiveness of key point detection of vascular key points in the optical coherence tomography angiography (OCTA) modality, and compared with the dynamic snake network (DSCNet), it can make the descriptor more accurate and rotationally robust, so the registration success rate is higher.

[0065] like Figure 8 As shown, Figure 8 This is a chessboard diagram for the registration of fundus color photography and optical coherence tomography angiography images. Figure 8 (a) is a color fundus photograph. Figure 8 (b) is an optical coherence tomography image. Figure 8 (c) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through scale-invariant feature transformation. Figure 8 (d) is the registration result of fundus color photo and fluorescein fundus angiography image through Orb. Figure 8 (e) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the Harris corner detector. Figure 8 (f) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the SuperRetina key point detection network. Figure 8 (g) in the figure is the registration result of the fundus color photo and the fluorescein fundus angiography image through the dynamic snake network. Figure 8 (h) in the figure is the registration result of the fundus color photograph and the fluorescein fundus angiography image through the key point detection model. Figure 8 The results of fundus color photography and optical coherence tomography angiography (OCTA) image registration using different methods are shown. The results show that traditional methods cannot correctly match key points. SuperRetina cannot learn the vascular key point information of the OCTA modality, so the registration is wrong. In addition, the key point detection model (G_DSCNet) proposed in the present invention has smaller differences in the misaligned parts than the dynamic snake network (DSCNet), that is, the key point RMSE is smaller.

[0066] The present invention performs well in the registration tasks of fundus color photography and fluorescein fundus angiography (FFA) images, and fundus color photography and optical coherence tomography angiography (OCTA) images, indicating that the method is applicable to multimodal image registration tasks.

[0067] Embodiment 4 of the present invention provides a multimodal retinal fundus image registration system, including: A model construction module is used to construct a key point detection model, wherein the key point detection model includes: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution, group normalization, and Relu activation function connected in sequence along the forward propagation direction; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map; Reference rotation equivariant feature acquisition module, used to copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Floating rotation equivariant feature acquisition module, used to copy floating images A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; The reference key point prediction map acquisition module is used to input the reference rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the reference key point prediction map; The floating key point prediction map acquisition module is used to input the floating rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the floating key point prediction map; A key point descriptor acquisition module is used to input the total reference rotation equivariant feature map and the total floating rotation equivariant feature map into a key point descriptor decoder, and output a descriptor of each reference key point and a descriptor of each floating key point; The image registration module is used to input the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module, and output the registered image.

[0068] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A multimodal retinal fundus image registration method, characterized in that: The following steps are involved: Constructing a key point detection model, the key point detection model comprising: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution, group normalization, and Relu activation function connected in sequence along the forward propagation direction; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map; Copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Copy the floating image A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; The reference rotation equivariant features of the four stages are input into the key point detection decoder for feature fusion, and the reference key point prediction map is output; The floating rotation equivariant features of the four stages are input into the key point detection decoder for feature fusion, and the floating key point prediction map is output; The total reference rotation equivariant feature map and the total floating rotation equivariant feature map are input into the key point descriptor decoder respectively, and the descriptor of each reference key point and the descriptor of each floating key point are output; The reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points are input into the registration module, and the registered image is output.

2. A multimodal retinal fundus image registration method according to claim 1, characterized in that: The rotation equivariant features of the four stages are input into the key point detection decoder for feature fusion, and the key point prediction map is output, including: The rotation equivariant features of the fourth stage are sequentially passed through the group equivariant dynamic snake convolution module and group upsampling to obtain the key point decoding features of the fourth level; After channel concatenation of the key point decoding features of the fourth level and the rotation equivariant features of the third stage, the key point decoding features of the third level are obtained by sequentially passing through the group equivariant dynamic snake convolution module and group upsampling; After channel concatenation of the key point decoding features of the third level and the rotation equivariant features of the second stage, the key point decoding features of the second level are obtained by sequentially passing through the group equivariant convolution module and group upsampling; After channel concatenation of the key point decoding features of the second level and the rotation equivariant features of the first stage, the key point decoding features of the first level are obtained; For the first level of key point decoding features, the feature is After all the features in the dimension are stacked to the channel dimension, they pass through the fully connected layer and the sigmoid activation function in sequence to obtain the key point prediction map.

3. A multimodal retinal fundus image registration method according to claim 1 or 2, characterized in that: The input features of the group equivariant dynamic snake convolution module are processed through the feature enhancement branch to obtain enhanced features; The input features of the group equivariant dynamic snake convolution module are processed by the group equivariant dynamic snake convolution block to capture spatial information in different directions and obtain the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction; After the enhanced features, the horizontal group equivariant output features and the vertical group equivariant output features are concatenated along the channel dimension, they are processed in sequence through the third group equivariant convolution, group normalization and ReLU activation function to obtain the output features of the group equivariant dynamic snake convolution module.

4. A multimodal retinal fundus image registration method according to claim 3, characterized in that: The input features of the group equivariant dynamic snake convolution module are passed through the group equivariant dynamic snake convolution block to capture spatial information in different directions, and obtain group equivariant output features in the horizontal direction and group equivariant output features in the vertical direction, including: The input features of the group equivariant dynamic snake convolution module are passed through The second group of equivariant convolution, group normalization and tanh activation function of the convolution kernel respectively obtains the horizontal bias feature and the vertical bias feature; wherein the value of each pixel in the horizontal bias feature represents the offset in the horizontal direction, and the value of each pixel in the vertical bias feature represents the offset in the vertical direction; According to the value of each pixel in the horizontal bias feature, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the horizontal direction, and the horizontal bias input feature is obtained through four-neighborhood interpolation; According to the value of each pixel in the bias feature in the vertical direction, each pixel in the input feature of the group equivariant dynamic snake convolution module is offset in the vertical direction, and the bias input feature in the vertical direction is obtained through four-neighborhood interpolation; The horizontal bias input features are sequentially passed through the horizontal direction The convolution kernel, group normalization and Relu activation function are used to obtain the horizontal feature map; among them, To set parameters; The vertical offset input features are passed through the vertical direction in sequence The convolution kernel, group normalization and Relu activation function are used to obtain the feature map in the vertical direction; The horizontal feature map and the vertical feature map are respectively The dimensions are integrated to obtain the group equivariant output features in the horizontal direction and the group equivariant output features in the vertical direction.

5. A multimodal retinal fundus image registration method according to claim 1, characterized in that: The total rotation equivariant feature map is input into the key point descriptor decoder, and the descriptor of each key point is output, including: Based on the key point prediction map, the coordinates of each key point are obtained; according to the coordinates of each key point, the pixel value of each channel corresponding to the position of the key point in the total rotation equivariant feature map is extracted to construct the initial descriptor of each key point; The initial descriptor of each key point is passed through batch normalization, 1×1 convolution layer, Sigmoid activation function, 3×3 convolution layer and fully connected layer in sequence to obtain the descriptor of each key point.

6. A multimodal retinal fundus image registration method according to claim 1, characterized in that: The step of inputting the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module and outputting the registered image comprises: The reference key point prediction map and the descriptors of all reference key points, the floating key point prediction map and the descriptors of all floating key points are matched by the Flann algorithm to obtain the key point pairs between the reference image and the floating image; The key point pairs between the reference image and the floating image are fitted using a random sampling consensus algorithm, and the key point pairs that do not meet the transformation constraints are eliminated to obtain an affine matrix; The floating image is aligned to the reference image using an affine matrix to obtain the registered image.

7. A multimodal retinal fundus image registration method according to claim 1, characterized in that: The reference image is a fundus color photograph, and the floating image is any one of a fundus fluorescein angiography image and an optical coherence tomography angiography image.

8. A multimodal retinal fundus image registration method according to claim 1, characterized in that: The training method of the key point detection model is: Acquire an image dataset, wherein the image dataset includes different images and a plurality of corresponding gold standard key points; Divide the image dataset into training set, validation set and test set; The key point detection model is trained using the training set, and the parameters of the key point detection model are optimized using the total loss function. The training process includes: After obtaining the descriptors of each key point, freeze the network parameters, perform a random affine transformation on the image of this round, re-input the image after affine transformation into the key point detection model, and obtain the key point prediction map corresponding to the image of this round after affine transformation. and the descriptor of each key point, calculate the key point prediction map corresponding to the image in this round before and after the affine transformation The Euclidean distance of the descriptor of the key point Key point prediction map corresponding to the current round image before affine transformation Middle The descriptor of the key points and the prediction map of the key points corresponding to the image in this round after affine transformation Middle The Euclidean distance between the descriptors of the next nearest neighbor key points of a key point ,when When the ratio is greater than the set threshold, the key point is set as the gold standard key point as the gold standard key point for the next round; After the key point detection model is trained, the key points detected by the key point detection model are registered with the descriptors through the registration module.

9. A multimodal retinal fundus image registration method according to claim 8, characterized in that: The total loss function includes: Dice loss function and ternary loss function; The calculation formula of the total loss function is: ， ， ， in, is the total loss function of the key point detection model, is the Dice loss function, is the ternary loss function, is the key point prediction map, Rotate the image Key point prediction map corresponding to the angle, is the soft label obtained by the Gaussian Boolean function of the key point graph of the gold standard, is element-wise multiplication, is the maximum value function, For the image Descriptors of key points and image rotation The corresponding angle The Euclidean distance between the descriptors of key points, is the threshold for dividing positive samples and negative samples, For the image Descriptors of key points and image rotation Any other key point corresponding to the angle The Euclidean distance between the descriptors of For the image Descriptors of key points and image rotation The maximum Euclidean distance between the descriptors of all other key points corresponding to the angle.

10. A multimodal retinal fundus image registration system, characterized in that: include: A model construction module is used to construct a key point detection model, wherein the key point detection model includes: a first feature encoder, a second feature encoder, a key point detection decoder and a key point descriptor decoder; Among them, each feature encoder includes four stages connected in sequence along the forward propagation direction, each stage is a group equivariant dynamic snake convolution module, and a group equivariant dynamic snake convolution module includes: feature enhancement branch, group equivariant dynamic snake convolution block, third group equivariant convolution, group normalization, Relu activation function; the feature enhancement branch includes: the first group equivariant convolution, group normalization, and Relu activation function connected in sequence along the forward propagation direction; each group equivariant dynamic snake convolution module outputs a stage of rotation equivariant features respectively; for the rotation equivariant features of other stages except the rotation equivariant features of the first stage, the size is adjusted by group transpose convolution respectively, so that its size is consistent with the rotation equivariant features of the first stage; the rotation equivariant features of all stages are spliced ​​along the channel dimension to obtain the total rotation equivariant feature map; Reference rotation equivariant feature acquisition module, used to copy the reference image A reference image structure is generated, and the reference image structure is passed through the first feature encoder to extract the reference rotation equivariant features of the four stages and the total reference rotation equivariant feature map; wherein, is the rotation angle dimension of the group equivariant convolution; Floating rotation equivariant feature acquisition module, used to copy floating images A floating image structure is generated, and the floating image structure is passed through a second feature encoder to extract floating rotation equivariant features of four stages and a total floating rotation equivariant feature map; The reference key point prediction map acquisition module is used to input the reference rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the reference key point prediction map; The floating key point prediction map acquisition module is used to input the floating rotation equivariant features of the four stages into the key point detection decoder, perform feature fusion, and output the floating key point prediction map; A key point descriptor acquisition module is used to input the total reference rotation equivariant feature map and the total floating rotation equivariant feature map into a key point descriptor decoder, and output a descriptor of each reference key point and a descriptor of each floating key point; The image registration module is used to input the reference image, the floating image, the reference key point prediction map, the floating key point prediction map, the descriptors of all reference key points, and the descriptors of all floating key points into the registration module, and output the registered image.

Images