Medical image registration method and system based on multi-dimensional loss function

Abstract

The application provides a medical image registration method and system based on a multi-dimensional loss function. The method comprises: preprocessing 3D-CT medical image data, including fixed images and floating images; constructing an initial model of a registration model using a transformer-based deep learning architecture, first performing coarse registration on the preprocessed 3D-CT medical image data to obtain coarse registration rigid transformation parameters; further performing fine registration on the initial model of the registration model based on the coarse registration rigid transformation parameters; and further optimizing the initial model of the registration model using a multi-dimensional adaptive loss function to obtain a final registration model capable of obtaining optimal results. The application comprehensively uses deep learning network technology, a coarse-to-fine two-stage registration structure based on a transformer deep learning network framework, and a multi-dimensional adaptive loss function, thereby improving the registration adaptation range and registration accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image registration processing, in particular to a medical image registration method and system based on a multi-dimensional loss function, and also relates to a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] Medical image refers to the technology and process of obtaining internal tissue images of human body or part of human body in a non-invasive manner for medical treatment or medical research, including X-ray, computed tomography (CT), magnetic resonance imaging (MRI), etc. In the medical field, image registration is a very important task. Medical images usually contain a large amount of information, including tumors, lesions, organs, etc., and these information can be obtained through image registration to obtain more accurate detection information.

[0003] In image registration, two or more groups of images need to be aligned so that their corresponding pixel positions are consistent in the same coordinate system. The goal of image registration is to minimize the differences between images so that they can be compared and fused in subsequent processing and analysis. However, traditional medical image registration techniques still have certain limitations, such as long computation time, poor effect of processing complex medical images, etc. With the continuous development of deep learning and computer vision technology, the research on medical image registration is more in-depth, and the application of deep learning in medical image registration can provide better support for medical diagnosis and treatment.

[0004] Compared with traditional medical image registration schemes, medical image registration technology based on deep learning has some significant advantages: it can adaptively extract features without the need for manual design of feature extractors; it can handle more complex data; it has higher accuracy, better robustness, faster speed, etc. Therefore, it is widely used in the fields of medical image registration, segmentation, detection, etc.

[0005] Miao, et al. constructed a CNN regressor to directly estimate the transformation parameters (Miao, Shun, Z. Jane Wang, and Rui Liao. "A CNN regression approach for real-time 2D / 3D registration." IEEE transactions on medical imaging 35.5 (2016): 1352-1363.). Liao, et al. built a POINT^2 network by tracking points of interest (POI) and triangulation layers to achieve more reliable 2D / 3D registration (Liao, Haofu, et al. "Multiview 2D / 3D rigid registration via a point-of-interest network for tracking and triangulation." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019.). Kausch, et al. proposed a sequential deep learning-based C-arm pose automatic estimation process, which achieved end-to-end automatic registration and made the network more robust through the constraints of segmented images and landmark points (Kausch, Lisa, et al. "C-arm positioning for standard projections during spinal implant placement." Medical Image Analysis 81 (2022): 102557.). Czolbe, et al. introduced semantic similarity evidence to make the transformation matrix smoother (Czolbe, Steffen, et al. "Semantic similarity metrics for image registration." Medical Image Analysis 87 (2023): 102830.). Zhu, et al. used multi-scale feature extraction and similarity attention models to further obtain more accurate and robust registration results (Zhu, Fei, et al. "Similarity attention-based CNN for robust 3D medical image registration." Biomedical Signal Processing and Control 81 (2023): 104403.).

[0006] Although deep learning has been widely used in medical image registration, there are still many deficiencies after its introduction: first, the high demand for computer power of the metrological calculation of three-dimensional data, direct processing of the original data requires a large amount of computing power and is time-consuming, and the accuracy and part of the semantic information are lost by downsampling or dimensionality reduction, both of which need to be balanced. In addition, the existing direct registration algorithm generally uses a single loss function, which often produces the problem of incompatible registration accuracy and adaptation range during training. SUMMARY

[0007] The present application provides a medical image registration method and system based on a multi-dimensional loss function to address the above deficiencies in the prior art, as well as a corresponding computer terminal and computer readable storage medium.

[0008] According to one aspect of the present application, a medical image registration method based on a multi-dimensional loss function is provided, comprising:

[0009] Pretreating 3D-CT medical image data to obtain a training data set and medical image data to be registered;

[0010] Providing a transformer-based deep learning network model, which includes a coarse registration network part and a fine registration network part;

[0011] Using the coarse registration network part to perform coarse registration on the training data set to obtain coarse registration transformation parameters;

[0012] Based on the coarse registration transformation parameters, using the fine registration network part to perform fine registration on the training data set to obtain fine registration transformation parameters;

[0013] Using a multi-dimensional adaptive loss function to optimize the fine registration transformation parameters to train a registration model;

[0014] Using the registration model to perform registration processing on the medical image data to be registered.

[0015] Preferably, the pretreatment of 3D-CT medical image data to obtain a training data set comprises:

[0016] Converting the image format, normalizing the image size, and processing the simulated floating data of the 3D-CT medical image data to obtain preprocessed data; wherein the preprocessed data includes fixed images and floating images;

[0017] Dividing the preprocessed data into a training set, a validation set, and a test set to obtain a training data set.

[0018] Preferably, the coarse registration network part and the fine registration network part each comprise a transformer module and a classification head module; wherein:

[0019] The transformer module is configured to extract features of input data to obtain attention scores.

[0020] The classification head module comprises two consecutive multi-layer perceptron layers and a layer of hyperbolic tangent activation function layers; the attention scores are taken as inputs of the classification head module, and after linear mapping and activation function, corresponding transformation parameter results are finally output.

[0021] Preferably, the coarse registration of the training data set by the coarse registration network part comprises:

[0022] The fixed images and the floating images in the training data set are respectively processed by dimension reduction and blocking to form corresponding low-resolution 3D feature maps.

[0023] The low-resolution 3D feature maps are taken as inputs of the transformer module of the coarse registration network part to generate low-resolution patch attention scores.

[0024] The low-resolution patch attention scores are outputted as coarse registration transformation parameters by the classification head module of the coarse registration network part.

[0025] Preferably, the fine registration of the training data set by the fine registration network part based on the coarse registration transformation parameters comprises:

[0026] The coarse registration transformation parameters are combined with the floating images in the training data set to obtain coarse registration transformed images.

[0027] The coarse registration transformed images and the fixed images in the training data set are respectively processed by dimension reduction and blocking to form corresponding high-resolution 3D feature maps.

[0028] The high-resolution 3D feature maps are taken as inputs of the transformer module of the fine registration network part to generate high-resolution patch attention scores.

[0029] The high-resolution patch attention scores are outputted as fine registration transformation parameters by the classification head module of the fine registration network part.

[0030] Preferably, the fine registration transformation parameters are optimized by a multi-dimensional adaptive loss function to train a registration model, comprising:

[0031] The fine registration transformation parameters are optimized by using a multi-dimensional adaptive loss function comprising a parameter domain and an image domain, wherein the multi-dimensional adaptive loss function L comprising the parameter domain and the image domain is total is:

[0032] L total = L MSE + μL NCC

[0033] In the formula, L MSE is a pose estimation loss function, μ is a sigmoid function, and L NCC is an inter-pixel loss function.

[0034] The pose estimation loss function L MSE is:

[0035]

[0036] In the formula, is a predicted value, is a standard value, and n is the number of fine registration transformation parameters.

[0037] The inter-pixel loss function L MCC is:

[0038]

[0039] In the formula, x is an original image, x * is a predicted image, Cov(·) is the covariance of two images, and Var(·) is the variance of an image itself.

[0040] Preferably, the registration of the medical image data to be registered by using the registration model comprises:

[0041] The medical image data to be registered is subjected to a rigid body transformation by the registration model to generate corresponding fixed medical image data, thereby completing the registration of the medical image to be registered.

[0042] According to another aspect of the present application, a medical image registration system based on a multi-dimensional loss function is provided, comprising:

[0043] A data processing module is configured to pre-process 3D-CT medical image data to obtain a training data set and medical image data to be registered, respectively.

[0044] A model training module is configured to provide a transformer-based deep learning network model, wherein the deep learning network model comprises a coarse registration network part and a fine registration network part.

[0045] The coarse registration network part is configured to perform coarse registration on the training data set to obtain coarse registration transformation parameters;

[0046] The fine registration network part is configured to perform fine registration on the training data set based on the coarse registration transformation parameters to obtain fine registration transformation parameters.

[0047] The fine registration transformation parameters are optimized by using a multi-dimensional adaptive loss function to obtain a registration model.

[0048] A registration module is configured to perform registration processing on the medical image data to be registered.

[0049] According to a third aspect of the present application, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program to perform the method according to any one of the preceding aspects of the present application, or to run the system according to any one of the preceding aspects of the present application.

[0050] According to a fourth aspect of the present application, a computer-readable storage medium is provided, which stores a computer program executable by a processor to perform the method according to any one of the preceding aspects of the present application, or to run the system according to any one of the preceding aspects of the present application.

[0051] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0052] The medical image registration method and system based on a multi-dimensional loss function provided by the present application implement a two-step 3D registration technology, use a deep learning method to replace a traditional method to achieve medical image three-dimensional registration, improve registration accuracy, and thus improve detection efficiency.

[0053] The medical image registration method and system based on a multi-dimensional loss function provided by the present application maximize the accuracy based on coarse and fine registration combination of multi-scale input under the condition of the same computer computing power.

[0054] The medical image registration method and system based on a multi-dimensional loss function provided by the present application use a multi-dimensional adaptive loss function to comprehensively consider pose estimation and pixel loss, and realize simultaneous compatibility of a wide range of adaptive range and high-precision registration. BRIEF DESCRIPTION OF DRAWINGS

[0055] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, with reference to the accompanying drawings:

[0056] Figure 1A workflow diagram of a medical image registration method based on a multi-dimensional loss function in an embodiment of the present application.

[0057] Figure 2 A working schematic diagram of a medical image registration model in a preferred embodiment of the present application.

[0058] Figure 3 A working schematic diagram of a scaled dot product attention module in a preferred embodiment of the present application.

[0059] Figure 4 A schematic diagram of a multi-head attention mechanism in a preferred embodiment of the present application.

[0060] Figure 5 A schematic diagram of two sets of registration results in a preferred embodiment of the present application; (a) and (b) are a set of registration result schematic diagrams, and (c) and (d) are another set of registration result schematic diagrams.

[0061] Figure 6 A schematic diagram of the component modules of a medical image registration system based on a multi-dimensional loss function in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application will be described in detail below: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation modes and specific operation processes are given. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0063] An embodiment of the present application provides a medical image registration method based on a multi-dimensional loss function, which comprehensively uses deep learning network technology, a coarse-to-fine two-stage registration structure based on a transformer deep learning network framework, and a multi-dimensional adaptive loss function, thereby improving the adaptation range and registration accuracy of registration.

[0064] Specifically, as shown in Figure 1 the medical image registration method based on a multi-dimensional loss function provided by the embodiment can include the following operations:

[0065] S1, pre-processing 3D-CT medical image data to obtain a training data set and medical image data to be registered;

[0066] S2, providing a transformer-based deep learning network model, the deep learning network model including a coarse registration network part and a fine registration network part;

[0067] S3, using the coarse registration network part to perform coarse registration on the training data set and obtaining coarse registration transformation parameters;

[0068] S4, performing fine registration on the training data set based on the coarse registration transformation parameter, to obtain a fine registration transformation parameter by using a fine registration network part;

[0069] S5, optimizing the fine registration transformation parameter by using a multi-dimensional adaptive loss function, to train a registration model;

[0070] S6, performing registration processing on the medical image data to be registered by using the registration model.

[0071] In some preferred embodiments, S1 above, the pre-processing of the 3D-CT medical image data to obtain the training data set can further include the following operations:

[0072] S11, performing image format conversion, image size normalization and simulated floating data processing on the 3D-CT medical image data to obtain pre-processed data; wherein the pre-processed data includes fixed images and floating images;

[0073] S12, dividing the pre-processed data into a training set, a validation set and a test set to obtain the training data set.

[0074] In some preferred embodiments, S2 above, the coarse registration network part and the fine registration network part can each further include a transformer module and a classification head module; wherein:

[0075] The transformer module is configured to extract features of the input data to obtain attention scores;

[0076] The classification head module includes two consecutive multi-layer perceptron layers and a layer of hyperbolic tangent activation function; the attention scores are taken as inputs of the classification head module, and after linear mapping and activation function, the corresponding transformation parameter results are finally output.

[0077] In some preferred embodiments, S3 above, the coarse registration of the training data set by using the coarse registration network part to obtain the coarse registration transformation parameter can further include the following operations:

[0078] S31, performing dimensionality reduction and block processing on the fixed images and the floating images in the training data set respectively to form corresponding low-resolution 3D feature maps;

[0079] S32, taking the low-resolution 3D feature maps as inputs of the transformer module of the coarse registration network part to generate low-resolution patch attention scores;

[0080] S33, outputting the coarse registration transformation parameter (6DOF rigid transformation parameter) by passing the low-resolution patch attention scores through the classification head module of the coarse registration network part.

[0081] In some preferred embodiments, S4, based on the coarse registration transformation parameters, uses a fine registration network part to fine register the training data set to obtain fine registration transformation parameters, and can further include the following operations:

[0082] S41, combine the coarse registration transformation parameters with the floating images in the training data set to obtain coarse registration transformed images;

[0083] S42, respectively reduce the dimensions and block process the coarse registration transformed images and the fixed images in the training data set to form corresponding high-resolution 3D feature maps;

[0084] S43, take the high-resolution 3D feature maps as the input of the transformer module of the fine registration network part to generate high-resolution patch attention scores;

[0085] S44, output the fine registration transformation parameters (6DOF rigid transformation parameters) through the classification head module of the fine registration network part from the high-resolution patch attention scores.

[0086] In some preferred embodiments, S5, uses a multi-dimensional adaptive loss function to optimize the fine registration transformation parameters to train the registration model, and can further include the following operations:

[0087] The multi-dimensional adaptive loss function containing the parameter domain and the image domain is used to optimize the fine registration model, wherein the multi-dimensional adaptive loss function containing the parameter domain and the image domain is:

[0088] L total =L MSE +μL NCC

[0089] In the formula, L MSE is the loss function of pose estimation, μ is a sigmoid function, and L NCC is the inter-pixel loss function.

[0090] In some preferred embodiments, the loss function L MSE of pose estimation is:

[0091]

[0092] In the formula, y is the predicted value, y is the standard value, and n is the number of fine registration transformation parameters (6DOF rigid transformation parameters, including: 3 displacement parameters and 3 rotation parameters);

[0093] The inter-pixel loss function L NCC is:

[0094]

[0095] where x is the original image, x * is the predicted image, Cov(·) is the covariance of two images, and Var(·) is the variance of an image itself.

[0096] In some preferred embodiments, the S6 is further configured to perform the registration of the medical image data to be registered by using the registration model, and the method further comprises:

[0097] performing rigid transformation on the medical image data to be registered by using the registration model to generate corresponding fixed medical image data, and completing the registration of the medical image data to be registered.

[0098] The technical solutions provided by the above embodiments of the application are further described in detail in combination with a preferred embodiment.

[0099] The medical image registration method provided by the preferred embodiment is based on deep learning technology and a multi-dimensional adaptive loss function, and realizes accurate registration of medical images through two-step training.

[0100] As shown in Figure 1 The medical image registration method based on a multi-dimensional loss function provided by the preferred embodiment mainly comprises the following steps:

[0101] Step one: acquisition and preprocessing of 3D-CT data;

[0102] Step two: coarse registration of 3D-CT data by using a coarse registration network part based on a transformer deep learning architecture;

[0103] Step three: further fine registration by using a fine registration network part based on a transformer deep learning architecture based on the updated parameters of the coarse registration result;

[0104] Step four: further optimization of the network architecture by using a multi-dimensional adaptive loss function, and training of a registration model for obtaining an optimal registration result.

[0105] In step one, the acquisition and preprocessing of 3D-CT data comprises:

[0106] Select an experimental data set and uniformly save it in NifTI format, and perform image size normalization, simulated floating data processing, and division of the obtained data into a training set, a validation set and a test set.

[0107] In step two, the coarse registration of 3D data by using a transformer deep learning architecture comprises:

[0108] Step 2.1, first, the standard image (fixed image) and the floating image are down-sampled to obtain low-resolution images, which are input into the registration network. The registration network is based on the ViT architecture under the transformer, which is a coarse registration network. In this registration network, the convolution idea is also used in the ViT architecture. For the input low-resolution 3D feature map As the input of the first stage, a function mapping relationship f is learned, which is a 3D convolution with a convolution kernel size of sxsxs, a step size of s-o, and zero padding of p. It maps the low-resolution 3D feature map x0 to a new feature f(x0) with C1 channels, and then expands f(x0) to H1W1D1xC1. Finally, the feature is input into the following transformer network after layer normalization.

[0109] Step 2.2: In the transformer network, the misalignment and global relationship between the fixed image and the moving image are captured and modeled through the similarity between the projected query-key pairs, so as to generate attention scores for each patch embedding, and finally a multi-head attention mechanism is adopted.

[0110] Step 2.3: Finally, a classification head is added at the end of each stage, which is implemented by two consecutive multi-layer perceptron (MLP) layers with hyperbolic tangent (Tanh) activation function. The classification head takes the features averaged by the transformer module as input, and finally outputs the rigid transformation parameters after linear mapping and activation function, which are the coarse registration 6DOF transformation parameters.

[0111] In step three, the coarse registration result update parameters are further fine registered by a transformer-based deep learning architecture, including:

[0112] Step 3.1, after the end of the first stage, the output rigid transformation parameters are converted into a rigid transformation matrix, which is then used to transform the images for the next stage. The second stage images are high-resolution images of the standard image (fixed image) and the floating image obtained by down-sampling. The second stage input is the high-resolution fixed image and the transformed image obtained by combining the coarse registration result and the high-resolution floating image.

[0113] Step 3.2: For the input transformed image 3D feature map as the input of the registration network of the second stage, a function mapping relationship f is learned, which is a 3D convolution with a convolution kernel size of sxsxs, a step size of s-o, and zero padding of p. It maps the feature x1 to a new feature f(x1) with C2 channels, and then expands f(x1) to H2W2D2xC2, and finally normalizes the feature.

[0114] Step 3.3: The new features are sent into the second stage transformer network. The structure of the second stage transformer network is the same as that of step 2.2.

[0115] Step 3.4: The classification head of the second stage is also implemented by two consecutive multi-layer perceptron (MLP) layers with a hyperbolic tangent (Tanh) activation function. The classification head takes the features averaged by the transformer module as input, and finally outputs the rigid transformation parameters after linear mapping and activation function. Finally, the corresponding rigid transformation parameters and floating CT data are subjected to rigid transformation to generate the corresponding standard CT data.

[0116] In step four, the loss function of the network is composed of the 6DOF error in the pose estimation dimension and the pixel loss in the image domain. The rigid transformation parameters can measure the error between the predicted pose and the standard pose, and the most common way is to calculate the difference between each parameter and try to minimize the difference to make the predicted value close to the standard value. In the preferred embodiment, MSE is used as the loss function L MSE of pose estimation. The predicted pose is further combined with the original 3D-CT to obtain the predicted 3D-CT, and the pixel-level error between the predicted 3D-CT and the standard 3D-CT is calculated. The pixel loss function L NCC is used to measure the difference between the predicted image and the target image. Finally, an adaptive multi-dimensional loss function is proposed, and the specific formula is as follows:

[0117] L total =L MSE +μL NCC

[0118] Where μ is a sigmoid-like function that adaptively allocates the weight of the 6-dimensional loss and the image domain loss based on the total number of network training set. The network initially gives priority to dimension error and then gives priority to image domain error. In this way, it is compatible with wide-range adaptation and high-precision registration.

[0119] The technical solutions provided by the above embodiments of the application will be further described in detail with reference to a specific application example.

[0120] The medical image registration method based on multi-dimensional loss function used in the specific application example uses computer-aided clinical multi-modal image data for registration, thereby realizing the registration of three-dimensional CT medical images.

[0121] As shown in Figure 1 , it is a workflow diagram of the medical image registration method based on multi-dimensional loss function used in the specific application example, including the following steps:

[0122] Step one, 3D-CT data acquisition and preprocessing;

[0123] Step two, coarse registration of 3D-CT data using a coarse registration network part based on a transformer-based deep learning architecture;

[0124] Step three, further fine registration by a fine registration network part based on a transformer-based deep learning architecture based on the update of parameters based on the coarse registration result, to obtain a registration model;

[0125] Step four, further optimization of the network architecture using a multi-dimensional adaptive loss function, training to obtain a registration model, to obtain an optimal registration result.

[0126] 3D-CT registration is to match the 3D standard image with the 3D floating image, and the registration is divided into rigid registration and non-rigid registration. The present application is based on rigid registration, and studies the rigid registration of 3D standard image and 3D floating image. First, the standard image F CT is taken as the starting point, and the corresponding floating image M CT is made by rigid transformation. The registration target is to learn the rigid transformation matrix from M CT to F CT . Specifically, the designed network parameterizes the rigid registration problem as a function f θ (M CT ,F CT )=T, where θ is a set of rigid transformation parameters, and T represents the predicted rigid transformation matrix. The purpose is to optimize the matrix T to make it closer to the true rigid transformation matrix, so that the DOF parameter after the transformation of the matrix T is closest to the actual DOF parameter.

[0127] In geometric transformation, rigid transformation is a transformation that keeps shape and size unchanged, which includes translation and rotation as two basic operations. Rigid transformation can be represented by six parameters, which describe the translation and rotation components. Specifically, the six parameters of rigid transformation are the displacement components of translation in three axial directions (t x , t y , t z ) and the rotation components around three axial directions (r x , r y , r z ). These parameters describe the movement and rotation operations of rigid transformation in three-dimensional space. The rigid transformation matrix is a 4x4 matrix that represents the rigid transformation, which transforms a point from the initial position to the new position. In the present application, the rigid transformation matrix is usually represented as T, which has the following form:

[0128]

[0129] where t is a 3x1 translation vector R is a 3x3 rotation matrix, which is obtained by multiplying three rotation matrices R x , R y and R z , i.e. R = R x * R y * R z where

[0130]

[0131]

[0132]

[0133] In step one, the experimental data set collection and preprocessing includes:

[0134] The largest publicly available annotated spine CT data set CTSpine1K is selected as the experimental data set, all data are saved in NIfTI format, the slice size is 512*512, the slice number is 512 after center cropping, so the overall size of all data is 5123, 279 cases in CTSpine1K are selected as the training set, and 81 cases are selected as the test set.

[0135] For each CT data, 6 different combinations of parameters are randomly generated in the range of t x,y,z ∈(-45mm, 45mm), r x,y,z ∈(-45°, 45°), then the DOF parameters are converted into R x , R y , R z and t in turn according to the above formula, and combined into a 4x4 rigid transformation matrix, finally the different rigid transformation parameters and the original CT data are generated through rigid transformation to generate the corresponding floating CT data according to the affine_transform method in the ndimage library of the scipy toolkit, a total of 1674 groups of data. Combined with the actual situation in the clinic, the parameter range in the test set is t x,y,z ∈(-20±15mm, 20±15mm), r x,y,z ∈(-20±15°, 20±15°), 4 different combinations of data are generated for each case, a total of 324 groups of data.

[0136] This specific application example solves the problem of 3D-CT registration from coarse to fine, which can be divided into two stages i = 1, 2, when i = 1, it is coarse registration, when i = 2, it is fine registration.

[0137] In step two, a transformer-based deep learning architecture is used to coarsely register the 3D data:

[0138] Step 2.1: First, the standard image and the floating image with a size of 5123 are down-sampled to low-resolution images with a size of 1283, which are input into the coarse registration network. The registration network is based on the vit architecture under the transformer, and the convolution idea is also used in the ViT architecture in the network. For the input 3D feature map As the input of the first stage, a function mapping relationship f is learned, f is a 3D convolution with a convolution kernel size of s x s x s, a step of s-o, and a zero padding of p. At this time It maps the feature x0 to a new feature f(x0) with C1 channels. The size of the new feature is Then f(x0) is unfolded into H1W1D1 x C1, and in this method H1W1D1 is 4096 and C1 is 256. Finally, the feature is input into the transformer module of the first stage after layer normalization.

[0139] Step 2.2: In the transformer network as shown in Figure 2 , the misalignment and global relationship between the fixed image and the moving image are captured and modeled through the similarity between the projected query-key pairs, thereby generating attention scores for each patch embedding. Specifically, given the input embedding patch block X = {x1, x2...x n-1 ,x n}, where n is the number of patch blocks.

[0140] Q(x k ) = x k W Q k = 1, 2... n-1, n

[0141] K(x k ) = x k W K k = 1, 2... n-1, n

[0142] V(x k ) = x k W V k = 1, 2... n-1, n

[0143] Where Q(x k ), K(x k ) and V(x k ) represent the query, key and value of the kth input patch respectively. W Q , W K and WV It is a learnable parameter matrix used to map the input patch to the query, key, and value space. Then, Q(x) is... k ), K(x k ) and V(x k Input to, for example Figure 3 The attention score is calculated in the scaled dot product attention module shown below:

[0144]

[0145] Where, d h This refers to the dimension of the key space. Finally, a multi-head attention mechanism with h=2 is employed, such as... Figure 4 As shown, the outputs of all scaled dot product attention modules are combined across channels and passed through matrix W. O ∈R d×d Perform a linear projection. In this study, d = 256.

[0146] Step 2.3: Finally, a classification head is appended at the end of each stage. This head is implemented by two consecutive multilayer perceptron (MLP) layers using the hyperbolic tangent (Tanh) activation function. The classification head takes the features averaged by the transformer module as input, and after linear mapping and activation functions, it finally outputs six rigid body transformation parameters.

[0147] In step three, the parameters updated from the coarse registration result are further refined through a registration network for fine registration:

[0148] Step 3.1: After the first stage, the output rigid body transformation parameters are converted into a rigid body transformation matrix using a formula before being used to transform the image in the next stage. The image in the second stage is a 512 pixel image. 3 The standard image and the floating image were downsampled to obtain a size of 256. 3 The high-resolution image is used as the first input, and the second-stage input is the "moving" image resulting from the combination of the high-resolution image and the coarse registration rigid body transformation matrix.

[0149] Step 3.2: For the input high-resolution "moving" 3D feature map As input for stage two, a function mapping f is learned, where f is a 3D convolution with kernel size s×s×s, stride so, and zero padding p. It maps feature x1 to a new feature f(x1) with channel C2, and the size of the new feature is... Then f(x1) is expanded into H2W2D2×C2. In this method, H2W2D2 is 4096 and C2 is 256. Finally, the features are normalized.

[0150] Step 3.3: The new features are sent into the second stage transformer module. The second stage transformer module has the same structure as the one in step 2.2.

[0151] Step 3.4: The second stage classification head is also implemented by two consecutive multi-layer perceptron (MLP) layers with a hyperbolic tangent (Tanh) activation function. The classification head takes the features averaged by the transformer module as input, and outputs 6 rigid transformation parameters after linear mapping and activation function. Finally, the corresponding standard CT data is generated by rigid transformation of the floating CT data according to the affine_transform method in the ndimage library of the scipy toolkit.

[0152] Compared to the requirement of initial registration error range for one-time registration, multi-resolution registration has higher robustness and universality for large-scale registration. Features may not be easily matched at a single scale. By performing two-stage registration at different scales, the robustness of matching can be improved, and the possibility of mismatching can be reduced. Moreover, multi-scale registration can capture more detailed information in the image, thereby improving the accuracy of registration. Performing coarse large-scale registration at a smaller scale and then performing detailed small-scale registration at a larger scale can ensure that the final registration result is more accurate and reduce the amount of calculation. The two-stage multi-scale resolution registration is adopted in the present application. Specifically, the input of the first stage is a low-resolution image of 128 3 , and the input of the second stage is a high-resolution image of 256 3 .

[0153] The loss function of the network in step four of the specific application example is composed of 6DOF error in pose estimation dimension and pixel loss in image domain.

[0154] The 6 rigid transformation parameters can measure the error between the predicted pose and the standard pose. The most common way is to calculate the difference of each parameter and try to reduce the difference so that the predicted value is close to the standard value In this experiment, MSE is used as the loss function L MSE of pose estimation:

[0155]

[0156] In this specific application example, the predicted pose obtained is further combined with the original 3D-CT to obtain the predicted 3D-CT, and the pixel level error with the standard 3D-CT is calculated. The pixel loss is used to measure the difference between the predicted image and the target image, and the optimization algorithm L NCC is optimized by making the predicted image closer to the target image.

[0157]

[0158] In the formula, x represents the original image, x * represents the predicted image, Cov(·) represents the covariance of two images, and Var(·) represents the variance of the image itself.

[0159] According to the conclusions of previous work and experimental experience, the loss based on 6 dimensions has greater robustness for the registration of two images with large errors. However, when the error of two images is small, the error value in each dimension is negligible compared with the initial error value, which will lead to difficulty in further gradient descent of the network after a certain time, and finally lead to underfitting. On the contrary, the loss based on images is very sensitive to the images to be registered, and a large initial error will have a high probability to lead the network to be unable to fit or fall into a local minimum. When the error is small, each pixel will add error to the loss based on images, which can further guide the gradient descent during network training and obtain a more optimized result. In summary, a multi-dimensional adaptive loss function is proposed in the specific application example, and the specific formula is as follows:

[0160] L total =L MSE +μL NCC

[0161] In the formula, μ is a sigmoid function, which adaptively allocates the weight of the loss based on 6 dimensions and the loss based on the image domain according to the set total number of network training rounds, so that the network initially gives priority to the dimensional error and then gives priority to the image domain error. In this way, the wide range of adaptation and high-precision registration are compatible.

[0162] In summary, the medical image registration method provided by the above-mentioned embodiment of the application inputs the moving position 3D-CT, outputs the rigid transformation matrix from the moving position to the standard position, and further obtains the standard position 3D-CT. The final registration result three-dimensional effect diagram is as shown in Figure 5 It can be seen that, compared with the large spatial misalignment before registration, the method provided by the above-mentioned embodiment of the application can realize spatial alignment, thereby realizing rigid registration of three-dimensional image data, so as to guide the clinician to make relevant judgments and operations.

[0163] An embodiment of the application provides a medical image registration system based on a multi-dimensional loss function, as shown in Figure 6 The system can include the following modules:

[0164] A data processing module, which is used for pre-processing 3D-CT medical image data to obtain a training data set and medical image data to be registered, respectively;

[0165] a model training module configured to provide a transformer-based deep learning network model, the deep learning network model comprising a coarse registration network part and a fine registration network part; wherein:

[0166] the coarse registration network part is configured to perform coarse registration on a training data set to obtain coarse registration transformation parameters;

[0167] the fine registration network part is configured to perform fine registration on the training data set based on the coarse registration transformation parameters to obtain fine registration transformation parameters;

[0168] the fine registration transformation parameters are optimized using a multi-dimensional adaptive loss function to obtain a registration model;

[0169] a registration module configured to perform registration processing on medical image data to be registered.

[0170] In the model training module, when selecting a loss function, according to the conclusions and experimental experience of previous work, a loss function based on six dimensions has greater robustness for registration of two images with large errors. When the error of the two images is small, the error value in each dimension is negligible compared to the initial error value, which will make it difficult for the network to continue gradient descent after a certain period of time, ultimately leading to underfitting. On the contrary, the image-based loss is very sensitive to the image to be registered. A large initial error will have a high probability of causing the network to fail to fit or fall into a local minimum. When the error is small, each pixel will add error to the image-based loss, which can further guide the gradient descent during network training and obtain a more optimized result. Therefore, an adaptive multi-dimensional loss function that considers both pose estimation and pixel loss is designed.

[0171] It should be noted that the steps in the method provided by the present application can be implemented by corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solution of the method to realize the composition of the system, i.e., the embodiments in the method can be understood as preferred examples of constructing the system, which will not be described here.

[0172] An embodiment of the present application is a computer terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, can be used to execute the method of any one of the above embodiments of the present application, or run the system of any one of the above embodiments of the present application.

[0173] Optionally, the memory is configured to store a program; the memory can include volatile memory (e.g., random-access memory (RAM) such as static random-access memory (SRAM), Double Data Rate SDRAM (DDR SDRAM), and the like) or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, and the like for implementing the above-described methods), computer instructions, and the like. The computer programs, computer instructions, and the like described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, and the like described above can be invoked by the processor.

[0174] The computer programs, computer instructions, and the like described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, and the like described above can be invoked by the processor.

[0175] The processor is configured to execute the computer programs stored in the memory to implement each step in the methods or each module of the systems described in the above embodiments. For details, refer to the related descriptions in the above method and system embodiments.

[0176] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0177] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is configured to be executed by a processor to implement the method of any one of the above embodiments of the present application or to run the system of any one of the above embodiments of the present application.

[0178] The medical image registration method and system based on a multi-dimensional loss function provided by the above embodiments of the present application implement a two-step 3D registration technology, use a deep learning method to replace a traditional method to implement medical image three-dimensional registration, improve registration accuracy, and thus improve detection efficiency. The coarse-fine registration combination based on multi-scale input maximizes the accuracy under the condition of the same computer computing power. The multi-dimensional adaptive loss function comprehensively considers the posture estimation and the loss between pixels, and realizes the simultaneous compatibility of a wide range of adaptive range and high-precision registration.

[0179] Those skilled in the art will appreciate that, in addition to implementing the system provided by the present application and each device thereof in the form of a pure computer readable program code, the system provided by the present application and each device thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device thereof can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0180] The details of the above embodiments of the present application are known to those skilled in the art.

[0181] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essential content of the present application.

Claims

1. A medical image registration method based on a multidimensional loss function, characterized in that, include: Preprocessing of 3D-CT medical image data yields a training dataset and medical image data to be registered. A transformer-based deep learning network model is provided, which includes a coarse registration network part and a fine registration network part; The coarse registration network part is used to perform coarse registration on the training dataset to obtain coarse registration transformation parameters; Based on the coarse registration transformation parameters, the fine registration network part is used to perform fine registration on the training dataset to obtain the fine registration transformation parameters; The fine registration transformation parameters are optimized using a multi-dimensional adaptive loss function, and a registration model is trained, including: The fine registration transform parameters are optimized using a multi-dimensional adaptive loss function that includes both parameter and image domains. for: In the formula, Let be the loss function for pose prediction. For sigmoid-like functions, The inter-pixel loss function; The loss function for posture prediction is calculated from the error between the predicted posture and the standard posture based on the six rigid body transformation parameters. The registration model is used to perform registration processing on the medical image data to be registered.

2. The medical image registration method based on a multidimensional loss function according to claim 1, characterized in that, The preprocessing of 3D-CT medical image data yields a training dataset, including: The 3D-CT medical image data undergoes image format conversion, image size normalization, and simulated floating data processing to obtain preprocessed data; wherein, the preprocessed data includes: fixed images and floating images; The preprocessed data is divided into a training set, a validation set, and a test set to obtain the training dataset.

3. The medical image registration method based on a multidimensional loss function according to claim 1, characterized in that, Both the coarse registration network and the fine registration network include: a transformer module and a classification head module; wherein: The transformer module is used to extract features from the input data and obtain an attention score; The classification head module includes two consecutive multilayer perceptron layers and a hyperbolic tangent activation function layer; the attention score is used as the input of the classification head module, and after linear mapping and activation function, the corresponding transformation parameter results are finally output.

4. The medical image registration method based on a multidimensional loss function according to claim 3, characterized in that, The step of using the coarse registration network part to perform coarse registration on the training dataset to obtain coarse registration transformation parameters includes: The fixed and floating images in the training dataset are subjected to dimensionality reduction and block segmentation respectively to form corresponding low-resolution 3D feature maps. The low-resolution 3D feature map is used as input to the transformer module of the coarse registration network to generate a low-resolution patch attention score. The low-resolution patch attention score is used to output coarse registration transformation parameters through the classification head module of the coarse registration network.

5. The medical image registration method based on a multidimensional loss function according to claim 3, characterized in that, The step of performing fine registration on the training dataset based on the coarse registration transformation parameters, using the fine registration network component, to obtain the fine registration transformation parameters includes: The coarse registration transformation parameters are combined with the floating images in the training dataset to obtain the coarse registration transformed image. The coarsely registered image and the fixed image in the training dataset are respectively subjected to dimensionality reduction and block segmentation to form corresponding high-resolution 3D feature maps. The high-resolution 3D feature map is used as input to the transformer module of the fine registration network to generate a high-resolution patch attention score. The high-resolution patch attention score is used to output fine registration transformation parameters through the classification head module of the fine registration network.

6. The medical image registration method based on a multidimensional loss function according to claim 1, characterized in that, The registration process of the medical image data to be registered using the registration model includes: The medical image data to be registered is subjected to rigid body transformation through the registration model to generate corresponding fixed medical image data, thus completing the registration of the medical image to be registered.

7. A medical image registration system based on a multidimensional loss function, characterized in that, include: The data processing module is used to preprocess 3D-CT medical image data to obtain training datasets and medical image data to be registered. The model training module provides a transformer-based deep learning network model, which includes a coarse registration network and a fine registration network. The coarse registration network part is used to perform coarse registration on the training dataset to obtain coarse registration transformation parameters; The fine registration network part performs fine registration on the training dataset based on the coarse registration transformation parameters to obtain the fine registration transformation parameters; The fine registration transformation parameters are optimized using a multi-dimensional adaptive loss function, and a registration model is trained, including: The fine registration transform parameters are optimized using a multi-dimensional adaptive loss function that includes both parameter and image domains. for: In the formula, Let be the loss function for pose prediction. For sigmoid-like functions, The inter-pixel loss function; The loss function for posture prediction is calculated from the error between the predicted posture and the standard posture based on the six rigid body transformation parameters. The registration module is used to perform registration processing on the medical image data to be registered.

8. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-6, or to run the system of claim 7.

Images