Fast reversible three-dimensional thin plate spline deformation method and device based on deep learning

Abstract

The invention relates to a rapid reversible three-dimensional thin-plate spline deformation method based on deep learning. The method comprises the following steps: obtaining a thin-plate spline displacement field psi < tps >; constructing a deep neural network to obtain a floating image Im and a fixed image If; four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions are obtained; obtaining a final displacement field psi; and obtaining a final registration result. According to the method, a traditional TPS deformation algorithm is improved based on a deep learning method, the complex non-rigid deformation relation between different control points can be accurately captured, meanwhile, the complex calculation process originally depending on matrix inversion and kernel function operation in TPS is converted into the single forward propagation process capable of being trained from end to end, and the method has the advantages of being high in robustness and high in robustness. Therefore, the computing resource consumption and the time complexity are remarkably reduced, and the real-time response capability of the system and the efficiency of processing large-scale data are improved.

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and computer vision technology, and in particular to a fast and reversible three-dimensional thin plate spline deformation method and device based on deep learning. Background Art

[0002] Non-rigid deformation is an important technology in computer vision and image processing, designed to deform objects or images to achieve precise shape adjustment and structural migration. Traditional non-rigid deformation methods mainly include elastic deformation, template-based deformation, and thin plate splines. Elastic deformation usually simulates the deformation of elastic bodies to make local adjustments and maintains the smoothness of the deformation based on the principle of energy minimization. However, it has certain limitations when dealing with complex structures and large-scale data. Template-based deformation methods rely on known template images or shapes and adjust the target shape through the mapping relationship of control points. However, they are prone to instability when dealing with nonlinear and large-scale deformations.

[0003] Thin Plate Spline (TPS) is a classic non-rigid deformation method that achieves smooth deformation by minimizing bending energy. The core advantage of TPS is that it can ensure the smoothness of the deformation results and avoid abrupt or discontinuous deformations. In addition, TPS can generate globally consistent deformation effects by globally optimizing the interaction between control points, which is particularly suitable for processing tasks with complex structures and large-scale deformations. Although TPS has shown excellent performance in many applications, it still has some limitations, especially when processing large-scale data. The computational complexity of TPS is O(n 3 ), where n is the number of control points. As the number of control points increases, the computational complexity increases cubically, making TPS very slow to apply to large datasets and complex scenarios. For example, in high-dimensional data or image processing tasks, TPS may not meet the requirements of real-time computing. Furthermore, TPS suffers from the drawback of irreversible deformation, which limits its application in some tasks. TPS deforms between control points through interpolation, but points outside the control points (i.e., points within the interpolation region) are typically not mapped one-to-one. Interpolation is essentially an estimation method used to infer the values ​​of other unknown points based on known data (control points). During this process, TPS does not define how to reverse the deformation result to the original data. This leads to a problem: although TPS maintains smoothness during deformation, points outside the control points do not have a fixed, inverse mapping relationship. Without additional information, the deformed image or shape cannot be restored to its original state through a simple inverse operation. Summary of the Invention

[0004] In order to solve the problems of low computational efficiency and irreversible deformation in existing technologies when processing large-scale data, the primary purpose of the present invention is to provide a fast and reversible three-dimensional thin plate spline deformation method based on deep learning that can reduce computing resource consumption and time complexity, improve the system's real-time response capability and efficiency in processing large-scale data, and ensure that the deformation process meets the topology preservation and reversibility.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a fast and reversible three-dimensional thin plate spline deformation method based on deep learning, which includes the following steps in sequence:

[0006] (1) Obtain the target image feature point set P tar And the feature point set P of the image to be registered src , target image feature point set P tar And the feature point set P of the image to be registered src is a one-to-one matching point set, and the thin plate spline displacement field ψ is calculated by the TPS thin plate spline deformation model tps ;

[0007] (2) Construct a deep neural network and transform the target image feature point set P tar And the feature point set P of the image to be registered src Input deep neural network, deep neural network transforms the target image feature point set P tar And the feature point set P of the image to be registered src The images are converted into binary images as floating images I m and fixed image I f ;

[0008] (3) The floating image I m and fixed image I f Input to the multi-scale feature encoder to obtain four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions

[0009] (4) Four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions The feature maps of each layer in the image are passed through the correlation-aware multi-window MLP module to perform four coarse-to-fine registration steps, and the resulting displacement field ψ4 is the final displacement field ψ of the nonlinear deformation from the image to be registered to the target image.

[0010] (5) Using the final displacement field ψ, the target image feature point set P tar Accurately mapped to the feature point set P of the image to be registered srcThe spatial coordinate system is used to achieve accurate alignment of feature points; and based on the final displacement field ψ, the nonlinear geometric transformation of the image to be registered is performed to obtain the final registration result;

[0011] (6) The loss function uses the mean square error to supervise the final displacement field ψ, and combines it with the distance loss function to optimize the alignment accuracy of the final registration result.

[0012] In step (1), the target image feature point set P is obtained tar And the feature point set P of the image to be registered src The specific steps include:

[0013] (1a) A set of coordinate points is randomly generated in a spatial range with a resolution of 160*96*128 as the original point set. The number of original point sets is M;

[0014] (1b) Construct a three-dimensional control grid with a resolution of 5*5*5 in a spatial range of 160*96*128, and generate a random displacement at each grid node with a displacement amplitude not exceeding 16;

[0015] (1c) constructing a 3D displacement interpolator based on the 3D control grid to calculate the smooth displacement of any point;

[0016] (1d) using a three-dimensional displacement interpolator to calculate the displacement corresponding to the original point set, adding the corresponding displacement to the coordinates of each point in the original point set to generate a deformed point set;

[0017] (1e) The original point set is used as the target image feature point set P tar , the deformed point set is used as the feature point set O of the image to be registered src .

[0018] The step (3) specifically includes the following steps:

[0019] (3a) Construct a multi-scale feature encoder. The multi-scale feature encoder consists of four cascaded convolutional modules, which include the first to fourth convolutional modules in sequence. Each convolutional module is connected by a 3D average pooling layer with a stride of 2 to achieve step-by-step dimensionality reduction of the feature space dimension.

[0020] (3b) Each convolution module adopts a unified architecture design, which consists of two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1. Each convolutional layer is followed by a LeakyReLU activation function and an instance normalization layer.

[0021] (3c) The floating image I m and fixed image I fFeature extraction is performed sequentially through four convolution modules. The feature map output by each convolution module is input into the next convolution module after average pooling and downsampling, and finally four layers of floating feature maps with different resolutions are output. and four layers of fixed feature maps with different resolutions

[0022] The step (4) specifically includes the following steps:

[0023] (4a) Using the extracted four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions Four coarse-to-fine registration steps are performed, where the first registration step is: first, the floating feature map and fixed feature maps Feed into the first multi-window MLP block to explore the floating feature map and fixed feature maps Then, the features output by the first multi-window MLP block are fed into the first deformable registration head, which generates an initial velocity field v1. The initial velocity field V1 passes through the first integrator to obtain the initial displacement field ψ1.

[0024] (4b) The initial displacement field ψ1 is used to guide the second step of registration. After upsampling, the initial displacement field ψ1 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The initial velocity field v1 is upsampled together with the features generated by the second multi-window MLP module to obtain the first upsampled features. The first upsampled features and the features generated by the second multi-window MLP module are input into the third multi-window MLP module. The output features of the third multi-window MLP module are sent to the second deformable registration head. The second deformable registration head generates a residual velocity field v2. The residual velocity field v2 is fused with the upsampled initial displacement field ψ1 after passing through the second integrator to form a displacement field ψ2.

[0025] (4c) The displacement field ψ2 is used to guide the third step of registration. After upsampling, the displacement field ψ2 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field V2 is upsampled together with the second upsampled features, and the features generated by the fourth multi-window MLP module and the second upsampled features are input into the fifth multi-window MLP module. The output features of the fifth multi-window MLP module are sent to the third deformable registration head, and the third deformable registration head generates a residual velocity field V3. The residual velocity field V3 is fused with the upsampled displacement field ψ2 after passing through the third integrator to form a displacement field ψ3.

[0026] (4d) The displacement field ψ3 is used to guide the fourth step of registration. After upsampling, the displacement field ψ3 is applied to the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field v3 is sent together to the sixth multi-window MLP module, the residual velocity field v3 is upsampled to obtain the third upsampling feature, the feature generated by the sixth multi-window MLP module and the third upsampling feature are input into the seventh multi-window MLP module together, the output feature of the seventh multi-window MLP module is sent to the fourth deformable registration head, and the fourth deformable registration head generates a residual velocity field v4. The residual velocity field v4 is fused with the upsampled displacement field ψ3 after passing through the fourth integrator to form the displacement field ψ4, which is the final displacement field ψ of the nonlinear deformation of the image to be registered to the target image.

[0027] The step (5) specifically includes the following steps:

[0028] (5a) The final displacement field ψ contains three channels, corresponding to the displacement in the x, y, and z directions respectively; the target image feature point set P is calculated using the trilinear interpolation method. tar The displacement of each point in the x, y, and z directions is used to obtain the transformed point set P warped ;

[0029] (5b) Generate grid coordinates corresponding to the image to be registered, and add the grid coordinates to the displacements in the x, y, and z directions in the final displacement field ψ to obtain the transformed coordinates; calculate the grayscale value of each grid point by interpolation to achieve a nonlinear geometric transformation of the image to be registered, and the nonlinear geometric transformation uses image distortion to obtain the final registration result.

[0030] The step (6) specifically includes the following steps:

[0031] (6a) In terms of displacement field optimization, a displacement field supervision mechanism is constructed to minimize the predicted final displacement field ψ and the thin plate spline displacement field ψ. tps The mean square error between The formula is as follows:

[0032]

[0033] Where n represents the total number of integer coordinates of the displacement field, y i Represents ψ tps The displacement of each point in represents the displacement of each point in the final displacement field ψ predicted by the network;

[0034] (6b) In terms of feature point alignment, we construct the average Euclidean distance loss and the target image feature point set Ptar After the displacement field transformation, the transformed point set P is obtained warped , by minimizing the transformed point set P warped and the feature point set P of the image to be registered src The average Euclidean distance between them makes the feature points accurately aligned; the average Euclidean distance loss The formula is as follows:

[0035]

[0036] Among them, N represents the total number of feature point pairs, ||.||2 is the Euclidean norm operator, Represents the target image feature point set P tar The spatial coordinates of the i-th feature point in , Represents the feature point set P of the image to be registered src The spatial coordinates of the i-th feature point in R 3 →R 3 Parameterized displacement field mapping function;

[0037] (6c) Total loss function for:

[0038]

[0039] Among them, λ is a regularization parameter used to balance the accuracy of displacement field and feature point alignment.

[0040] The step (4a) specifically comprises the following steps:

[0041] (4a1) Input the two input feature maps F1 and F2 into a 3D correlation layer, which is used to calculate the spatial correlation between them and obtain the 3D correlation map C F , where the maximum displacement is set to d = 3, and the correlation map is generated by calculating the local correlation within a 3 × 3 × 3 neighborhood; the generated C F The shape of the graph is the same as the input feature graph F1 and F2, and the number of channels is d 3 =27;

[0042] (4a2) F1, F2 and C F These three feature maps are connected to form a comprehensive feature map, and the comprehensive feature map is sent to a 3×3×3 convolution layer to obtain a correlation perception feature map F corr , F corr Including F1, F2 and C F Spatial relationship information between them;

[0043] (4a3) F corrInput to a first multi-window MLP module for processing; the first multi-window MLP module includes multiple window-based MLP branches, where each branch uses a different window size for processing;

[0044] (4a4) The output features of the multi-window MLP branches are fused to obtain the fused features: the output features of each branch are weighted and merged by using the channel weighted summation method. In the weighted summation process, the weight coefficients are dynamically calculated through global average pooling, two-layer MLP and softmax function; in order to highlight the important feature channels, the residual channel attention module is used to process the fused features to obtain the output feature F S ;

[0045] (4a5) will output feature F S The initial velocity field v1 is generated by the first deformable registration head. The initial velocity field v1 undergoes seven identical scaling and squaring operations to obtain the initial displacement field ψ1. The scaling and squaring formulas are as follows:

[0046]

[0047] Where v represents the static velocity field, ψ (t) represents the integration of v over t∈[0,1].

[0048] Another object of the present invention is to provide an electronic device, comprising:

[0049] processor; and

[0050] A memory having computer program instructions stored therein, wherein the computer program instructions, when executed by the processor, enable the processor to perform the deep learning-based fast reversible three-dimensional thin plate spline deformation method as described above.

[0051] The present invention also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the deep learning-based fast reversible three-dimensional thin plate spline deformation method as described above.

[0052] The above technical solutions demonstrate the following advantages: First, the present invention improves the traditional TPS deformation algorithm, accurately capturing the complex non-rigid deformation relationships between different control points. It also transforms the complex computational process in TPS, which originally relied on matrix inversion and kernel function operations, into a single forward propagation process that can be trained end-to-end. This significantly reduces computational resource consumption and time complexity, and improves the system's real-time response capability and efficiency in processing large-scale data. Second, the present invention introduces a differential homeomorphism constraint mechanism, namely an integrator, into the network structure. This mechanism ensures that the deformation function is continuous, injective, and reversible globally by constraining the first-order derivative properties of the deformation field, thereby ensuring that the deformation process satisfies topology preservation and reversibility. Third, the training point sets used in the present invention are all randomly generated. The wide coverage of the random point sets enables the network to learn general deformation laws in the latent deformation space, rather than being limited to the local statistical characteristics of a few typical samples. This training mechanism not only gives the model stronger generalization ability, but also enables it to have good robustness and transferability in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of the method of the present invention;

[0054] Figure 2 This is a structural diagram of the deep neural network in the present invention;

[0055] Figure 3 The slice of the image to be registered, the image after the thin plate spline deformation of the image to be registered selected by the present invention are respectively used as the target image, the image to be registered and the blend result of the target image;

[0056] Figure 4 are the feature points of the image to be registered and the feature points of the target image;

[0057] Figure 5 The forward displacement field of the present invention is respectively Figure 3 Register to Figure 4 The result image and the target image are blended together; the inverse displacement field of the present invention is used to Figure 4 Register to Figure 3 The result image is blended with the image to be registered;

[0058] Figure 6 A result image of registering the target image feature points to the to-be-registered image feature points using the forward displacement field of the present invention;

[0059] Figure 7 This is a result image obtained by using the inverse displacement field of the present invention to register the feature points of the image to be registered to the feature points of the target image. DETAILED DESCRIPTION

[0060] like Figure 1 As shown, a fast and reversible three-dimensional thin plate spline deformation method based on deep learning includes the following steps in sequence:

[0061] (1) Obtain the target image feature point set P tar And the feature point set P of the image to be registered src , target image feature point set P tar And the feature point set P of the image to be registered src is a one-to-one matching point set, and the thin plate spline displacement field ψ is calculated by the TPS thin plate spline deformation model tps ;

[0062] (2) Constructing a deep neural network, such as Figure 2 As shown, the target image feature point set P tar And the feature point set P of the image to be registered src Input deep neural network, deep neural network transforms the target image feature point set P tar And the feature point set P of the image to be registered src The images are converted into binary images as floating images I m and fixed image I f ;

[0063] (3) The floating image I m and fixed image I f Input to the multi-scale feature encoder to obtain four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions

[0064] (4) Four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions The feature maps of each layer in the image are passed through the correlation-aware multi-window MLP module to perform four coarse-to-fine registration steps, and the resulting displacement field ψ4 is the final displacement field ψ of the nonlinear deformation from the image to be registered to the target image.

[0065] (5) Using the final displacement field ψ, the target image feature point set P tar Accurately mapped to the feature point set P of the image to be registered src The spatial coordinate system is used to achieve accurate alignment of feature points; and based on the final displacement field ψ, the nonlinear geometric transformation of the image to be registered is performed to obtain the final registration result;

[0066] (6) The loss function uses the mean square error to supervise the final displacement field ψ, and combines it with the distance loss function to optimize the alignment accuracy of the final registration result.

[0067] In step (1), the target image feature point set P is obtained tarAnd the feature point set P of the image to be registered src The specific steps include:

[0068] (1a) A set of coordinate points is randomly generated in a spatial range with a resolution of 160*96*128 as the original point set. The number of original point sets is M;

[0069] (1b) Construct a three-dimensional control grid with a resolution of 5*5*5 in a spatial range of 160*96*128, and generate a random displacement at each grid node with a displacement amplitude not exceeding 16;

[0070] (1c) constructing a 3D displacement interpolator based on the 3D control grid to calculate the smooth displacement of any point;

[0071] (1d) using a three-dimensional displacement interpolator to calculate the displacement corresponding to the original point set, adding the corresponding displacement to the coordinates of each point in the original point set to generate a deformed point set;

[0072] (1e) The original point set is used as the target image feature point set P tar , the deformed point set is used as the feature point set P of the image to be registered src .

[0073] The step (3) specifically includes the following steps:

[0074] (3a) Construct a multi-scale feature encoder. The multi-scale feature encoder consists of four cascaded convolutional modules, which include the first to fourth convolutional modules in sequence. Each convolutional module is connected by a 3D average pooling layer with a stride of 2 to achieve step-by-step dimensionality reduction of the feature space dimension.

[0075] (3b) Each convolution module adopts a unified architecture design, which consists of two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1. Each convolutional layer is followed by a LeakyReLU activation function and an instance normalization layer.

[0076] (3c) The floating image I m and fixed image I f Feature extraction is performed sequentially through four convolution modules. The feature map output by each convolution module is input into the next convolution module after average pooling and downsampling, and finally four layers of floating feature maps with different resolutions are output. and four layers of fixed feature maps with different resolutions

[0077] The step (4) specifically includes the following steps:

[0078] (4a) Using the extracted four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions Four coarse-to-fine registration steps are performed, where the first registration step is: first, the floating feature map and fixed feature maps Feed into the first multi-window MLP block to explore the floating feature map and fixed feature maps Then, the features output by the first multi-window MLP block are fed into the first deformable registration head, which generates an initial velocity field v1. The initial velocity field v1 passes through the first integrator to obtain the initial displacement field ψ1.

[0079] (4b) The initial displacement field ψ1 is used to guide the second step of registration. After upsampling, the initial displacement field ψ1 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The initial velocity field v1 is upsampled together with the features generated by the second multi-window MLP module to obtain the first upsampled features. The first upsampled features and the features generated by the second multi-window MLP module are input into the third multi-window MLP module. The output features of the third multi-window MLP module are sent to the second deformable registration head. The second deformable registration head generates a residual velocity field v2. The residual velocity field v2 is fused with the upsampled initial displacement field ψ1 after passing through the second integrator to form a displacement field ψ2.

[0080] (4c) The displacement field ψ2 is used to guide the third step of registration. After upsampling, the displacement field ψ2 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field v2 is upsampled together with the second upsampled features, and the features generated by the fourth multi-window MLP module and the second upsampled features are input into the fifth multi-window MLP module. The output features of the fifth multi-window MLP module are sent to the third deformable registration head, and the third deformable registration head generates a residual velocity field v3. The residual velocity field v3 is fused with the upsampled displacement field ψ2 after passing through the third integrator to form a displacement field ψ3.

[0081] (4d) The displacement field ψ3 is used to guide the fourth step of registration. After upsampling, the displacement field ψ3 is applied to the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field v3 is sent together to the sixth multi-window MLP module, the residual velocity field v3 is upsampled to obtain the third upsampling feature, the feature generated by the sixth multi-window MLP module and the third upsampling feature are input into the seventh multi-window MLP module together, the output feature of the seventh multi-window MLP module is sent to the fourth deformable registration head, and the fourth deformable registration head generates a residual velocity field v4. The residual velocity field v4 is fused with the upsampled displacement field ψ3 after passing through the fourth integrator to form the displacement field ψ4, which is the final displacement field ψ of the nonlinear deformation of the image to be registered to the target image.

[0082] The step (5) specifically includes the following steps:

[0083] (5a) The final displacement field ψ contains three channels, corresponding to the displacement in the x, y, and z directions respectively; the target image feature point set P is calculated using the trilinear interpolation method. tar The displacement of each point in the x, y, and z directions is used to obtain the transformed point set P warped ;

[0084] (5b) Generate grid coordinates corresponding to the image to be registered, and add the grid coordinates to the displacements in the x, y, and z directions in the final displacement field ψ to obtain the transformed coordinates; calculate the grayscale value of each grid point by interpolation to achieve a nonlinear geometric transformation of the image to be registered, and the nonlinear geometric transformation uses image distortion to obtain the final registration result.

[0085] The step (6) specifically includes the following steps:

[0086] (6a) In terms of displacement field optimization, a displacement field supervision mechanism is constructed to minimize the predicted final displacement field ψ and the thin plate spline displacement field ψ. tps The mean square error between The formula is as follows:

[0087]

[0088] Where n represents the total number of integer coordinates of the displacement field, y i Represents ψ tps The displacement of each point in represents the displacement of each point in the final displacement field ψ predicted by the network;

[0089] (6b) In terms of feature point alignment, we construct the average Euclidean distance loss and the target image feature point set P tar After the displacement field transformation, the transformed point set P is obtained warped , by minimizing the transformed point set P warped and the feature point set P of the image to be registered src The average Euclidean distance between them makes the feature points accurately aligned; the average Euclidean distance loss The formula is as follows:

[0090]

[0091] Among them, N represents the total number of feature point pairs, ||.||2 is the Euclidean norm operator, Represents the target image feature point set P tar The spatial coordinates of the i-th feature point in , Represents the feature point set P of the image to be registered src The spatial coordinates of the i-th feature point in R 3 →R 3 Parameterized displacement field mapping function;

[0092] (6c) Total loss function for:

[0093]

[0094] Among them, λ is a regularization parameter used to balance the accuracy of displacement field and feature point alignment.

[0095] The step (4a) specifically comprises the following steps:

[0096] (4a1) Input the two input feature maps F1 and F2 into a 3D correlation layer, which is used to calculate the spatial correlation between them and obtain the 3D correlation map C F , where the maximum displacement is set to d = 3, and the correlation map is generated by calculating the local correlation within a 3 × 3 × 3 neighborhood; the generated C F The shape of the graph is the same as the input feature graph F1 and F2, and the number of channels is d 3 =27;

[0097] (4a2) F1, F2 and C F These three feature maps are connected to form a comprehensive feature map, and the comprehensive feature map is sent to a 3×3×3 convolution layer to obtain a correlation perception feature map F corr , F corr Including F1, F2 and C F Spatial relationship information between them;

[0098] (4a3) F corr Input to a first multi-window MLP module for processing; the first multi-window MLP module includes multiple window-based MLP branches, where each branch uses a different window size for processing;

[0099] (4a4) The output features of the multi-window MLP branches are fused to obtain the fused features: the output features of each branch are weighted and merged by using the channel weighted summation method. In the weighted summation process, the weight coefficients are dynamically calculated by global average pooling, two-layer MLP and softmax function; in order to highlight the important feature channels, the residual channel attention module is used to process the fused features to obtain the output feature F S ;

[0100] (4a5) will output feature F S The initial velocity field v1 is generated by the first deformable registration head. The initial velocity field v1 undergoes seven identical scaling and squaring operations to obtain the initial displacement field ψ1. The scaling and squaring formulas are as follows:

[0101]

[0102] Where v represents the static velocity field, ψ (t) represents the integration of v over t∈[0,1].

[0103] To verify the performance of the present invention, we tested the runtime of the Thin Plate Spline (TPS) algorithm on a 568*320*456 image with 4809 feature points on a computing platform equipped with an i7-7700K CPU and an RTX-1080 GPU, and evaluated the computational efficiency of the present invention. As shown in Table 1, the runtime of the present invention in the same scenario was 1.45 seconds, a 3559.84-fold improvement compared to 5161.77 seconds on the CPU platform, fully demonstrating the significant computational efficiency advantage of the present invention.

[0104] Table 1 Comparison of running time between the present invention and TPS on different computing platforms

[0105] model Image size (pixels) Number of feature points Run time (seconds) Speed ​​improvement ratio TPS(CPU) 568*320*456 4809 5161.77 1 TPS (GPU) 568*320*456 4809 99.12 52.07 The present invention 568*320*456 4809 1.45 3559.84

[0106] Figure 3 The present invention selects a mouse brain image of the STPT modality as the image to be registered; the image of the image to be registered after thin plate spline deformation is used as the target image; and the image to be registered and the target image are blended.

[0107] Figure 4 are the feature points of the image to be registered and the feature points of the target image.

[0108] Figure 5 The following two blend images of the registration results are shown: Figure 5 The left picture shows Figure 3 The result of registering the image to be registered to the target image through the forward displacement field of the present invention and the blend result of the target image; Figure 5 The right picture is Figure 3 The target image in the image to be registered is registered to the image to be registered using the inverse displacement field of the present invention, and the blended result of the image to be registered. As can be seen from the blended result diagram, the present invention can achieve the same registration accuracy as the thin plate spline deformation in both forward and reverse registration processes, and is reversible, fully demonstrating the superiority of the present invention in image registration accuracy and reversibility.

[0109] Figure 6 This is the result of using the forward displacement field of the present invention to align the target image feature points with the feature points of the image to be registered. It can be seen that the present invention can achieve accurate registration of the target image feature points with the feature points of the image to be registered.

[0110] Figure 7 From the result of using the inverse displacement field of the present invention to align the feature points of the image to be aligned to the feature points of the target image, it can be seen that the present invention can achieve accurate alignment of the feature points of the image to be aligned to the feature points of the target image, which also reflects the reversibility of the present invention.

[0111] In summary, the present invention improves the traditional TPS deformation algorithm based on a deep learning method. It can accurately capture the complex non-rigid deformation relationship between different control points. At the same time, it transforms the complex computational process in TPS, which originally relied on matrix inversion and kernel function operations, into a single forward propagation process that can be trained end-to-end, thereby significantly reducing computing resource consumption and time complexity, and improving the system's real-time response capability and efficiency in processing large-scale data. The present invention introduces a differential homeomorphism constraint mechanism, namely an integrator, into the network structure. This mechanism ensures that the deformation function is continuous, injective, and reversible on a global scale by constraining the first-order derivative properties of the deformation field, thereby ensuring that the deformation process satisfies topology preservation and reversibility. The training point sets used in the present invention are all randomly generated. The wide coverage of the random point sets enables the network to learn the general deformation laws in the potential deformation space, rather than being limited to the local statistical characteristics of a few typical samples. This training mechanism not only gives the model stronger generalization ability, but also enables it to have good robustness and transferability in practical applications.

[0112] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A fast and reversible three-dimensional thin plate spline deformation method based on deep learning, characterized by: The method comprises the following steps in sequence: (1) Obtain the target image feature point set P tar And the feature point set P of the image to be registered src , target image feature point set P tar And the feature point set P of the image to be registered src is a one-to-one matching point set, and the thin plate spline displacement field ψ is calculated by the TPS thin plate spline deformation model tps ; (2) Construct a deep neural network and transform the target image feature point set P tar And the feature point set P of the image to be registered src Input deep neural network, deep neural network transforms the target image feature point set P tar And the feature point set P of the image to be registered src The images are converted into binary images as floating images I m and fixed image I f ; (3) The floating image I m and fixed image I f Input to the multi-scale feature encoder to obtain four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions (4) Four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions The feature maps of each layer in the image are passed through the correlation-aware multi-window MLP module to perform four coarse-to-fine registration steps, and the resulting displacement field ψ4 is the final displacement field ψ of the nonlinear deformation from the image to be registered to the target image. (5) Using the final displacement field ψ, the target image feature point set P tar Accurately mapped to the feature point set P of the image to be registered src spatial coordinate system to achieve precise alignment of feature points; And based on the final displacement field ψ, the image to be registered is subjected to nonlinear geometric transformation to obtain the final registration result; (6) The loss function uses the mean square error to supervise the final displacement field ψ, and combines it with the distance loss function to optimize the alignment accuracy of the final registration result.

2. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 1 is characterized by: In step (1), the target image feature point set P is obtained tar And the feature point set P of the image to be registered src The specific steps include: (1a) A set of coordinate points is randomly generated in a spatial range with a resolution of 160*96*128 as the original point set. The number of original point sets is M; (1b) Construct a three-dimensional control grid with a resolution of 5*5*5 in a spatial range of 160*96*128, and generate a random displacement at each grid node with a displacement amplitude not exceeding 16; (1c) constructing a 3D displacement interpolator based on the 3D control grid to calculate the smooth displacement of any point; (1d) using a three-dimensional displacement interpolator to calculate the displacement corresponding to the original point set, adding the corresponding displacement to the coordinates of each point in the original point set to generate a deformed point set; (1e) The original point set is used as the target image feature point set P tar , the deformed point set is used as the feature point set P of the image to be registered src .

3. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 1 is characterized in that: The step (3) specifically includes the following steps: (3a) Construct a multi-scale feature encoder. The multi-scale feature encoder consists of four cascaded convolutional modules, which include the first to fourth convolutional modules in sequence. Each convolutional module is connected by a 3D average pooling layer with a stride of 2 to achieve step-by-step dimensionality reduction of the feature space dimension. (3b) Each convolution module adopts a unified architecture design, which consists of two 3D convolutional layers with a kernel size of 3×3×3 and a stride of 1. Each convolutional layer is followed by a LeakyReLU activation function and an instance normalization layer. (3c) The floating image I m and fixed image I f Feature extraction is performed sequentially through four convolution modules. The feature map output by each convolution module is input into the next convolution module after average pooling and downsampling, and finally four layers of floating feature maps with different resolutions are output. and four layers of fixed feature maps with different resolutions 4. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 1 is characterized in that: The step (4) specifically includes the following steps: (4a) Using the extracted four layers of floating feature maps with different resolutions and four layers of fixed feature maps with different resolutions Four coarse-to-fine registration steps are performed, where the first registration step is: first, the floating feature map and fixed feature maps Feed into the first multi-window MLP block to explore the floating feature map and fixed feature maps Then, the features output by the first multi-window MLP block are fed into the first deformable registration head, which generates an initial velocity field v1. The initial velocity field v1 passes through the first integrator to obtain the initial displacement field ψ1. (4b) The initial displacement field ψ1 is used to guide the second step of registration. After upsampling, the initial displacement field ψ1 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The initial velocity field v1 is upsampled together with the features generated by the second multi-window MLP module to obtain the first upsampled features. The first upsampled features and the features generated by the second multi-window MLP module are input into the third multi-window MLP module. The output features of the third multi-window MLP module are sent to the second deformable registration head. The second deformable registration head generates a residual velocity field v2. The residual velocity field v2 is fused with the upsampled initial displacement field ψ1 after passing through the second integrator to form a displacement field ψ2. (4c) The displacement field ψ2 is used to guide the third step of registration. After upsampling, the displacement field ψ2 is used to adjust the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field v2 is upsampled together with the second upsampled features, and the features generated by the fourth multi-window MLP module and the second upsampled features are input into the fifth multi-window MLP module. The output features of the fifth multi-window MLP module are sent to the third deformable registration head, and the third deformable registration head generates a residual velocity field v3. The residual velocity field v3 is fused with the upsampled displacement field ψ2 after passing through the third integrator to form a displacement field ψ3. (4d) The displacement field ψ3 is used to guide the fourth step of registration. After upsampling, the displacement field ψ3 is applied to the floating feature map. The deformed feature map is generated by distortion deformation, and the deformed feature map is different from the fixed feature map The residual velocity field V3 is sent together to the sixth multi-window MLP module, the residual velocity field V3 is upsampled to obtain the third upsampling feature, the feature generated by the sixth multi-window MLP module and the third upsampling feature are input into the seventh multi-window MLP module together, the output feature of the seventh multi-window MLP module is sent to the fourth deformable registration head, and the fourth deformable registration head generates a residual velocity field v4. The residual velocity field v4 is fused with the upsampled displacement field ψ3 after passing through the fourth integrator to form the displacement field ψ4, which is the final displacement field ψ of the nonlinear deformation of the image to be registered to the target image.

5. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 1 is characterized in that: The step (5) specifically includes the following steps: (5a) The final displacement field ψ contains three channels, corresponding to the displacement in the x, y, and z directions respectively; the target image feature point set P is calculated using the trilinear interpolation method. tar The displacement of each point in the x, y, and z directions is used to obtain the transformed point set P warped ; (5b) Generate grid coordinates corresponding to the image to be registered, and add the grid coordinates to the displacements in the x, y, and z directions in the final displacement field ψ to obtain the transformed coordinates; calculate the grayscale value of each grid point by interpolation to achieve a nonlinear geometric transformation of the image to be registered, and the nonlinear geometric transformation uses image distortion to obtain the final registration result.

6. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 1, characterized in that: The step (6) specifically includes the following steps: (6a) In terms of displacement field optimization, a displacement field supervision mechanism is constructed to minimize the predicted final displacement field ψ and the thin plate spline displacement field ψ. tps The mean square error between The formula is as follows: Where n represents the total number of integer coordinates of the displacement field, y i Represents ψ tps The displacement of each point in represents the displacement of each point in the final displacement field ψ predicted by the network; (6b) In terms of feature point alignment, we construct the average Euclidean distance loss and the target image feature point set P tar After the displacement field transformation, the transformed point set P is obtained warped , by minimizing the transformed point set P warped and the feature point set P of the image to be registered src The average Euclidean distance between them makes the feature points accurately aligned; the average Euclidean distance loss The formula is as follows: Among them, N represents the total number of feature point pairs, ||.||2 is the Euclidean norm operator, Represents the target image feature point set P tar The spatial coordinates of the i-th feature point in , Represents the feature point set P of the image to be registered src The spatial coordinates of the i-th feature point in R 3 →R 3 Parameterized displacement field mapping function; (6c) Total loss function for: Among them, λ is a regularization parameter used to balance the accuracy of displacement field and feature point alignment.

7. The fast and reversible three-dimensional thin plate spline deformation method based on deep learning according to claim 4, characterized in that: The step (4a) specifically comprises the following steps: (4a1) Input the two input feature maps F1 and F2 into a 3D correlation layer, which is used to calculate the spatial correlation between them and obtain the 3D correlation map C F , where the maximum displacement is set to d = 3, and the correlation map is generated by calculating the local correlation within a 3 × 3 × 3 neighborhood; the generated C F The shape of the graph is the same as the input feature graph F1 and F2, and the number of channels is d 3 =27; (4a2) F1, F2 and C F These three feature maps are connected to form a comprehensive feature map, and the comprehensive feature map is sent to a 3×3×3 convolution layer to obtain a correlation perception feature map F corr , F corr Including F1, F2 and C F Spatial relationship information between them; (4a3) F corr Input to a first multi-window MLP module for processing; the first multi-window MLP module includes multiple window-based MLP branches, where each branch uses a different window size for processing; (4a4) The output features of the multi-window MLP branches are fused to obtain the fused features: the output features of each branch are weighted and merged by using the channel weighted summation method. In the weighted summation process, the weight coefficients are dynamically calculated by global average pooling, two-layer MLP and softmax function; in order to highlight the important feature channels, the residual channel attention module is used to process the fused features to obtain the output feature F S ; (4a5) will output feature F S The initial velocity field v1 is generated by the first deformable registration head. The initial velocity field v1 undergoes seven identical scaling and squaring operations to obtain the initial displacement field ψ1. The scaling and squaring formulas are as follows: Where v represents the static velocity field, ψ (t) represents the integration of v over t∈[0,1].

8. An electronic device comprising: processor; as well as A memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the deep learning-based fast reversible three-dimensional thin plate spline deformation method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor executes the deep learning-based fast reversible three-dimensional thin plate spline deformation method as described in any one of claims 1 to 7.

Images