Deep learning based cross-modality non-rigid registration method, system and medium

Abstract

This invention provides a deep learning-based cross-modal non-rigid body registration method, system, and medium, comprising: acquiring three-dimensional images of different modalities, including at least one reference image and at least one floating image; acquiring the registration region of the three-dimensional image, and detecting image feature points within the registration region of the reference image; obtaining image patches of a preset size centered on each image feature point; inputting the image patches into a similarity network to obtain a similarity map within the corresponding range of the floating image; inputting the coordinate information of the image feature points and the corresponding similarity map into a displacement network to obtain a displacement vector, interpolating the displacement vector field for regions without image feature points; and performing a spatial transformation on the floating image based on the displacement vector field to obtain the registration result. This invention reduces interference from low-information points by extracting image feature points, thus improving registration efficiency, especially when the image size is large.

Description

[0001] Technology Neighborhood

[0002] This invention relates to the field of image processing technology, and more specifically, to a cross-modal non-rigid body registration method, system, and medium based on deep learning. Background Technology

[0003] Modern medical diagnosis relies heavily on various medical imaging modalities. Medical multimodal imaging generally includes computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound imaging (US), each with its own characteristics. CT provides clear imaging of high-density tissues such as bones, liver, lungs, and kidneys, demonstrating a clear imaging advantage; MRI offers high resolution for soft tissue imaging. The fusion of multiple imaging modalities can provide complementary information, better achieving diagnostic, assessment, or interventional goals. For example, in computer-aided diagnosis, the fusion of multimodal images can fully combine the tissue characteristics reflected in different modalities, providing a more accurate interpretation of the presence, nature, and extent of lesions. In minimally invasive surgical navigation, the fusion of preoperative and intraoperative images allows for the overlay of preoperative planning and intraoperative images, providing surgeons with richer and more intuitive information, improving the quality of image guidance during interventional procedures, thereby enhancing surgical quality and improving clinical outcomes. However, images of different modalities are usually acquired at different times using different scanning instruments, which is accompanied by changes in patient posture and internal anatomical structures. Therefore, the prerequisite for achieving multimodal image fusion is to register multimodal medical images, and the accuracy of registration directly determines the fusion effect.

[0004] Cross-modal medical image registration is a challenging problem. The relationships between grayscale distributions in different modalities of medical images are often complex and unknown. Furthermore, structures and features present in one modality may be missing in another. Traditional cross-modal registration methods can be broadly categorized into grayscale similarity-based methods and anatomical feature-based methods. Grayscale-based methods primarily utilize cross-modal grayscale similarity measures, such as mutual information and cross-correlation; anatomical feature-based methods rely mainly on landmarks identified in different modalities. In recent years, deep learning technology has developed rapidly and has seen increasing research and application in the field of image registration, promising to solve the problems of slow registration speed and insufficient registration accuracy in traditional methods. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the purpose of this invention is to provide a cross-modal non-rigid body registration method, system and medium based on deep learning.

[0006] A deep learning-based cross-modal non-rigid body registration method according to the present invention includes:

[0007] Acquire three-dimensional images of different modalities, the three-dimensional images including at least one reference image and at least one floating image; acquire the registration region of the three-dimensional image, detect image feature points within the registration region of the reference image, the image feature points being points that can be distinguished from the image features of other points in their neighborhood; obtain image patches of a preset size centered on each image feature point; input the image patches into a similarity network to obtain a similarity map within the corresponding range of the floating image; input the coordinate information of the image feature points and the corresponding similarity map into a displacement network to obtain a displacement vector; interpolate the regions without image feature points based on the displacement vector to obtain a displacement vector field; perform spatial transformation on the floating image according to the displacement vector field to obtain the registration result.

[0008] Preferably, the region to be registered is determined through manual interaction, or based on the grayscale threshold of the image, or by automatic detection and segmentation of specific structures in the image.

[0009] Preferably, the method for acquiring the image feature points includes:

[0010] Voxel points are sampled from the region to be registered in the reference image. The feature score is determined based on the gray variance and gradient value in the neighborhood of the sampled point. Points with feature scores higher than the preset value are taken as image feature points.

[0011] Alternatively, specific structures in the region to be registered in the reference image can be segmented, and feature scores can be calculated based on the positional relationship between each boundary point of the specific structure and its surrounding boundary points. Points with feature scores greater than a preset value can be used as image feature points.

[0012] Preferably, the feature score S(p) is calculated based on the voxel point located at coordinate p in image I determined by the Foerstner operator, and its expression is:

[0013]

[0014] Among them, K σ This represents the Gaussian kernel function with variance σ. Let be the value of the spatial gradient of image I at coordinate p, and Tr(·) denotes finding the trace of the matrix.

[0015] Preferably, the registration method further includes:

[0016] Image feature point optimization steps: Optimize the number and distribution of the image feature points, using any of the following optimization methods:

[0017] Optimize the distribution of image feature points: Scan the reference image using a sampling window of a set size. When two or more image feature points appear in the sampling window, only retain the image feature point with the largest feature score.

[0018] Alternatively, optimize the number and distribution of the influence feature points: when the number of image feature points is greater than a preset value, randomly select a point from the detected image feature points as the initial optimization point set. Each time, select the point farthest from the remaining image feature points and add it to the optimization point set until the number of image feature points in the optimization point set reaches the preset value. The distance between the image feature point and the optimization point set is the minimum value of the Euclidean distance from the point to all image feature points in the optimization point set.

[0019] Alternatively, optimize the number and distribution of image feature points: when the number of image feature points is greater than the preset value, construct an octree structure using all image feature points, and add the point with the largest feature score in the current octet to the optimization point set according to the width priority principle, until the number of image feature points in the optimization point set reaches the preset value.

[0020] Preferably, the input of the similarity network is image patches corresponding to a reference image and a floating image. The image patch of the reference image has a size of W1×H1×D1, and the image patch of the floating image contains a specified detection range with a size of W2×H2×D2, and satisfies W1≤W2, H1≤H2, D1≤D2.

[0021] The output of the similarity network is a similarity map of the corresponding image feature points. The size of the similarity map is (W2-W1) / q+1×(H2-H1) / q+1×(D2-D1) / q+1, where q is the downsampling coefficient.

[0022] Preferably, the displacement network includes an encoder, a graph convolutional network, and a decoder; the input of the encoder is a similarity map, the graph convolutional network receives the output of the encoder and the coordinate information of the corresponding image feature points, and the output of the decoder is a displacement heatmap; the values ​​in the displacement heatmap are used as weights of the corresponding displacement vectors for weighted averaging to obtain the final displacement vector; the encoder and decoder are connected by a jumper.

[0023] Preferably, the registration method further includes:

[0024] The similarity of local structures between the reference image and the floating image is calculated using specified features; an objective function is constructed based on the similarity and smoothness constraints; and the displacement vector field is locally optimized by minimizing the objective function.

[0025] According to the present invention, a cross-modal non-rigid registration system based on deep learning is provided, comprising:

[0026] Registration Region Acquisition Module: Acquires three-dimensional images of different modalities, wherein the three-dimensional images include at least one reference image and at least one floating image, and acquires the registration region of the three-dimensional images;

[0027] Image feature point detection module: Detects image feature points in the registration area of ​​the reference image, wherein the image feature points are points that can be distinguished from the image features of other points in the neighborhood;

[0028] Similarity map acquisition module: Taking each of the image feature points as the center, an image block of a preset size is obtained; the image block is input into a similarity network to obtain a similarity map within the corresponding range of the floating image;

[0029] Displacement vector field acquisition module: Input the coordinate information of the image feature points and the corresponding similarity map into the displacement network to obtain the displacement vector; Based on the displacement vector, interpolate the region without image feature points to obtain the displacement vector field;

[0030] Registration module: Performs spatial transformation on the floating image based on the displacement vector field to obtain the registration result.

[0031] According to a medium provided by the present invention, the medium is a computer-readable storage medium, on which a deep learning-based cross-modal non-rigid body registration program is stored. When the deep learning-based cross-modal non-rigid body registration program is executed by a processor, it implements the above-described deep learning-based cross-modal non-rigid body registration method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention reduces interference from low-information points by extracting image feature points, thereby improving registration efficiency, especially when the image size is large.

[0034] 2. This invention uses a similarity graph of all feature points for global optimization, which removes the prerequisite of accurately detecting corresponding points in two modalities. At the same time, it considers the spatial distribution information of all feature points, thus improving the robustness of the algorithm.

[0035] 3. The present invention introduces deep learning to fully extract the corresponding information of the same anatomical structure in different modalities, and avoids the large time overhead caused by iterative solutions in traditional methods. Attached Figure Description

[0036] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0037] Figure 1This is a flowchart of the registration method of the present invention;

[0038] Figure 2 This is a schematic diagram of the similarity network structure of the present invention;

[0039] Figure 3 This is a schematic diagram of the displacement network structure of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0041] This invention discloses a cross-modal non-rigid body registration method based on deep learning, referring to... Figure 1 As shown, it includes the following steps:

[0042] Step S1: Acquire three-dimensional images of different modalities, including at least one reference image and at least one floating image. By reading in three-dimensional images of different modalities containing the same region of the same patient, designate modality 1 as the reference image and modality 2 as the floating image and interpolate it to the same spatial resolution as modality 1. Obtain the registration regions of the two images.

[0043] Three-dimensional images can be CT, MRI, ultrasound (three-dimensional ultrasound or a three-dimensional ultrasound image reconstructed from a series of two-dimensional ultrasound images), etc. During the registration process, the optimal spatial transformation of the floating image is sought, mapping it to the coordinate system of the reference image, so that the corresponding human anatomical points in the two modalities achieve spatial consistency.

[0044] The region to be registered can be determined manually, based on the image's grayscale threshold, or automatically detected and segmented from specific structures within the image. A special case is when the region to be registered is the entire image.

[0045] Step S2: Detect image feature points within the registration area of ​​the reference image. Sampling is performed on the registration area of ​​the reference image, and feature scores are obtained based on the neighborhood information of the sampled points. Points with feature scores greater than a set threshold are designated as image feature points. The method for obtaining image feature points is as follows:

[0046] Grid sampling or random sampling is performed on the region to be registered from the reference image. A 3D operator, constructed based on the gray-level variance and gradient values ​​in the neighborhood of the sampled points, is used to determine the feature score. Points with feature scores higher than a preset value are designated as image feature points. For example, the Foerstner operator is a commonly used 3D feature point detection operator that can be used to calculate the feature score S(p) of a pixel located at coordinate p in image I. Its expression is as follows:

[0047]

[0048] Among them, K σ This represents the Gaussian kernel function with variance σ. Let be the value of the spatial gradient of image I at coordinate p, and Tr(·) denotes finding the trace of the matrix.

[0049] Another way to obtain image feature points is to segment specific structures in the region to be registered in the reference image, calculate feature scores based on the positional relationship between each boundary point of the specific structure and its surrounding boundary points, and use points with feature scores greater than a preset value as image feature points. For example, curvature values ​​can be calculated by calculating the positional relationship between each boundary point and its surrounding boundary points, and points with curvature values ​​greater than a set threshold can be used as image feature points.

[0050] Step 3: Optimize the data and distribution of image feature points. To prevent image feature points from concentrating in the same area, optimize the number and distribution of image feature points to avoid a large number of image feature points in the subsequently obtained image patches. Any of the following optimization methods can be used:

[0051] Optimize the distribution of image feature points: Scan the reference image using a sampling window of a set size. When two or more image feature points appear in the sampling window, only retain the image feature point with the largest feature score.

[0052] Alternatively, optimize the number and distribution of influence feature points: when the number of image feature points is greater than a preset value, randomly select a point from the detected image feature points as the initial optimization point set. Each time, select the point farthest from the remaining image feature points and add it to the optimization point set until the number of image feature points in the optimization point set reaches the preset value. The distance between the image feature point and the optimization point set is the minimum value of the Euclidean distance from the point to all image feature points in the optimization point set.

[0053] Alternatively, optimize the number and distribution of image feature points: when the number of image feature points is greater than the preset value, construct an octree structure using all image feature points, and add the point with the largest feature score in the current octet to the optimization point set according to the width priority principle, until the number of image feature points in the optimization point set reaches the preset value.

[0054] Step S4: Using each image feature point as the center, take an image patch containing a specified range of neighborhood centered on that feature point, input the image patch into the similarity network, and obtain the similarity map within the corresponding range of the floating image.

[0055] Reference Figure 2 As shown, the input to the similarity network is image patches corresponding to a reference image and a floating image. The image patch of the reference image has a size of W1×H1×D1, and the image patch of the floating image contains a specified detection range with a size of W2×H2×D2, satisfying W1≤W2, H1≤H2, and D1≤D2. The output of the similarity network is a similarity map of corresponding image feature points. The size of the similarity map is (W2-W1) / q+1×(H2-H1) / q+1×(D2-D1) / q+1, where q is the downsampling coefficient. The value of any point in the similarity map represents the probability that the position of that value predicted based on local image features in the floating image corresponds to the same anatomical point as the feature point in the reference image.

[0056] The aforementioned similarity network is a self-supervised training-based convolutional neural network that uses similarity map peaks to construct a contrast loss function to determine whether the floating image patch contains an anatomical structure corresponding to the feature points of the reference image.

[0057] Step S5: Input the coordinate information of the image feature points and the corresponding similarity map into the displacement network to obtain the displacement vector.

[0058] Reference Figure 3 As shown, the displacement network includes an encoder, a graph convolutional network, and a decoder. The encoder's input is a similarity map. The graph convolutional network receives the encoder's output and the coordinate information of corresponding image feature points. The decoder's output is a displacement heatmap. The value of any point in the displacement heatmap represents the probability that, based on the distribution of all feature point positions and the prediction of the corresponding similarity map, the position of that value in the floating image corresponds to the same anatomical point as the feature point in the reference image. The values ​​in the displacement heatmap are used as weights for the corresponding displacement vectors and weighted averaged to obtain the final displacement vector. Are there jumpers between the encoder and decoder? In practical applications, multiple displacement network structures can also be cascaded.

[0059] Step S6: Interpolate the region without image feature points based on the displacement vector to obtain the displacement vector field.

[0060] One way to store the displacement vector field is as a 6-dimensional matrix, where the first 3 dimensions are the same size as the modality 1 image, and the last 3 dimensions represent the displacement vector that maps the corresponding pixel to the modality 2 image.

[0061] To ensure the smoothness of the displacement vector field, cubic linear interpolation can be used.

[0062] Step S7: Perform local optimization on the displacement vector field to obtain the final displacement vector field.

[0063] One specific optimization approach is to calculate the similarity of local structures between the reference image and the floating image using specified features; construct an objective function based on the similarity and smoothness constraints; and optimize the displacement vector field locally by minimizing the objective function. For example, the modality-independent neighborhood descriptor (MIND) is a common cross-modal image feature, which can be used as a specified feature to extract from the images of the two modalities, and the squared difference of the MIND descriptors of the two modalities can be used to measure the similarity of the local structures. Alternatively, a similarity network can be used, with its output replacing the specified feature similarity to construct the objective function for local optimization of the displacement vector field.

[0064] Step S8: Perform spatial transformation on the floating image based on the optimized displacement vector field to obtain the registration result.

[0065] This invention also discloses a deep learning-based cross-modal non-rigid registration system, comprising:

[0066] Registration Region Acquisition Module: Acquires three-dimensional images of different modalities, wherein the three-dimensional images include at least one reference image and at least one floating image, and acquires the registration region of the three-dimensional images;

[0067] Image feature point detection module: Detects image feature points in the registration area of ​​the reference image, wherein the image feature points are points that can be distinguished from the image features of other points in the neighborhood;

[0068] Similarity map acquisition module: Taking each of the image feature points as the center, an image block of a preset size is obtained; the image block is input into a similarity network to obtain a similarity map within the corresponding range of the floating image;

[0069] Displacement vector field acquisition module: Input the coordinate information of the image feature points and the corresponding similarity map into the displacement network to obtain the displacement vector; Based on the displacement vector, interpolate the region without image feature points to obtain the displacement vector field;

[0070] Registration module: Performs spatial transformation on the floating image based on the displacement vector field to obtain the registration result.

[0071] The present invention also discloses a medium, which is a computer-readable storage medium, such as a computer hard disk, wherein a deep learning-based cross-modal non-rigid body registration program is stored on the computer-readable storage medium, and the deep learning-based cross-modal non-rigid body registration program implements the above-mentioned deep learning-based cross-modal non-rigid body registration method when executed by a processor.

[0072] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, enabling the system and its various devices, modules, and units to function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0073] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in these embodiments can be arbitrarily combined with each other.

Claims

1. A cross-modal non-rigid body registration method based on deep learning, characterized in that, include: Acquire three-dimensional images of different modalities, wherein the three-dimensional images include at least one reference image and at least one floating image; A registration region to be obtained from a 3D image is acquired. Image feature points are detected within the registration region of the reference image. These image feature points are those that can be distinguished from the image features of other points in their neighborhood. Image patches of a preset size are obtained, centered on each image feature point. The image patches are input into a similarity network to obtain a similarity map within the corresponding range of the floating image. The coordinate information of the image feature points and the corresponding similarity map are input into a displacement network to obtain a displacement vector. Interpolation is performed on regions without image feature points based on the displacement vector to obtain a displacement vector field. The floating image is then spatially transformed according to the displacement vector field to obtain the registration result. The displacement network includes an encoder, a graph convolutional network, and a decoder; the input of the encoder is a similarity map, the graph convolutional network receives the output of the encoder and the coordinate information of the corresponding image feature points, and the output of the decoder is a displacement heatmap. The values ​​in the displacement heatmap are used as weights for the corresponding displacement vectors and weighted averaged to obtain the final displacement vector; the encoder and decoder are connected by a jumper.

2. The deep learning-based cross-modal non-rigid registration method according to claim 1, characterized in that: The region to be registered is determined through manual interaction, or based on the grayscale threshold of the image, or by automatic detection and segmentation of specific structures in the image.

3. The deep learning-based cross-modal non-rigid registration method according to claim 1, characterized in that: The methods for acquiring the image feature points include: Voxel points are sampled from the region to be registered in the reference image. The feature score is determined based on the gray variance and gradient value in the neighborhood of the sampled point. Points with feature scores higher than the preset value are taken as image feature points. Alternatively, specific structures in the region to be registered in the reference image can be segmented, and feature scores can be calculated based on the positional relationship between each boundary point of the specific structure and its surrounding boundary points. Points with feature scores greater than a preset value can be used as image feature points.

4. The deep learning-based cross-modal non-rigid registration method according to claim 3, characterized in that: Image determined based on Foerstner operator I The middle is located in the coordinate p Feature scoring of voxel points Its expression is: in, The variance is expressed as Gaussian kernel function, For image I Spatial gradient in coordinates p The value at that location, This indicates finding the trace of a matrix.

5. The deep learning-based cross-modal non-rigid registration method according to claim 3, characterized in that, Also includes: Image feature point optimization steps: Optimize the number and distribution of the image feature points, using any of the following optimization methods: Optimize the distribution of image feature points: Scan the reference image using a sampling window of a set size. When two or more image feature points appear in the sampling window, only retain the image feature point with the largest feature score. Alternatively, optimize the number and distribution of the influence feature points: when the number of image feature points is greater than a preset value, randomly select a point from the detected image feature points as the initial optimization point set. Each time, select the point farthest from the remaining image feature points and add it to the optimization point set until the number of image feature points in the optimization point set reaches the preset value. The distance between the image feature point and the optimization point set is the minimum value of the Euclidean distance from the point to all image feature points in the optimization point set. Alternatively, optimize the number and distribution of image feature points: when the number of image feature points is greater than the preset value, construct an octree structure using all image feature points, and add the point with the largest feature score in the current octet to the optimization point set according to the width priority principle, until the number of image feature points in the optimization point set reaches the preset value.

6. The deep learning-based cross-modal non-rigid registration method according to claim 1, characterized in that: The input to the similarity network is image patches corresponding to a reference image and a floating image, wherein the image patch size of the reference image is W1. H1 D1, the image patch of the floating image contains a specified detection range, and its size is W2. H2 D2, and satisfies W1 W2,H1 H2,D1 D2; The output of the similarity network is a similarity map of the corresponding image feature points, and the size of the similarity map is ((W2-W1) / q+1). ((H2-H1) / q+1) ((D2-D1) / q+1), where q is the downsampling coefficient.

7. The deep learning-based cross-modal non-rigid registration method according to claim 1, characterized in that, Also includes: The similarity of local structures between the reference image and the floating image is calculated using specified features; An objective function is constructed based on the similarity and smoothness constraints, and the displacement vector field is locally optimized by minimizing the objective function.

8. A cross-modal non-rigid registration system based on deep learning, characterized in that, include: Registration Region Acquisition Module: Acquires three-dimensional images of different modalities, wherein the three-dimensional images include at least one reference image and at least one floating image, and acquires the registration region of the three-dimensional images; Image feature point detection module: Detects image feature points in the registration area of ​​the reference image, wherein the image feature points are points that can be distinguished from the image features of other points in the neighborhood; Similarity map acquisition module: Taking each of the image feature points as the center, an image block of a preset size is obtained; the image block is input into a similarity network to obtain a similarity map within the corresponding range of the floating image; Displacement vector field acquisition module: Input the coordinate information of the image feature points and the corresponding similarity map into the displacement network to obtain the displacement vector; Based on the displacement vector, interpolate the region without image feature points to obtain the displacement vector field; Registration module: Performs spatial transformation on the floating image based on the displacement vector field to obtain the registration result; The displacement network includes an encoder, a graph convolutional network, and a decoder; the input of the encoder is a similarity map, the graph convolutional network receives the output of the encoder and the coordinate information of the corresponding image feature points, and the output of the decoder is a displacement heatmap. The values ​​in the displacement heatmap are used as weights for the corresponding displacement vectors and weighted averaged to obtain the final displacement vector; the encoder and decoder are connected by a jumper.

9. A medium, characterized in that: The medium is a computer-readable storage medium, on which a deep learning-based cross-modal non-rigid body registration program is stored. When the deep learning-based cross-modal non-rigid body registration program is executed by a processor, it implements the deep learning-based cross-modal non-rigid body registration method as described in any one of claims 1-7.

Images