Medical image registration method, device and equipment of motion decomposition model and medium

Abstract

The application relates to a medical image registration method and device based on a motion decomposition model, equipment and a medium, wherein the method comprises the following steps: collecting a fixed medical image and a moving medical image to be registered, and preprocessing the fixed medical image and the moving medical image to be registered to obtain initial medical image data; extracting structured image features in the initial medical image data through a shared weight coding module to obtain a moving feature map and a fixed feature map, wherein the moving feature map and the fixed feature map each comprise five layers of feature maps; inputting each layer of the moving feature map and the fixed feature map into a corresponding estimator in sequence, and fusing the generated residual deformation field with a total residual deformation field after upsampling in each estimator to obtain a target total residual deformation field; and registering the moving feature map according to the target total residual deformation field to obtain a target registration image. The application improves the accuracy of medical image registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a medical image registration method and device based on a motion decomposition model, equipment and a medium. BACKGROUND

[0002] With the development of deep learning, computer-aided diagnosis technology is also becoming more and more mature. Among them, medical image registration based on deep learning has become one of the research hotspots. Medical image registration has important application value in medical image analysis, and has many applications in medical fields such as imaging diagnosis, surgical navigation and radiotherapy planning. Traditional medical image registration methods have been widely used in many medical fields, such as image navigation, non-invasive navigation in tumor resection surgery, etc. In the future, with the continuous development and application of medical technology and software, the application of traditional medical image registration methods in medical image analysis will be more extensive and in-depth. However, traditional medical image registration methods also have some problems and challenges, such as the influence of registration accuracy by noise and deformation, low registration efficiency of large-scale data sets, etc. To solve these challenges, many new methods and technologies have appeared in recent years, such as medical image registration methods combined with deep learning, registration methods based on supervised and unsupervised learning, etc.

[0003] The existing medical image registration method is to incorporate the Transformer into the registration framework, mainly to enhance the features, without further exploring their relationship with registration or deformation estimation, and few studies consider the relationship between the attention mechanism and deformation estimation. Therefore, the medical image registration accuracy of the existing image registration method is low. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a medical image registration method, device, equipment and medium based on a motion decomposition model to improve the accuracy of medical image registration.

[0005] To solve the above technical problems, the embodiments of the present application provide a medical image registration method based on a motion decomposition model, comprising:

[0006] Collecting fixed medical images and mobile medical images to be registered, and pre-processing the fixed medical images and mobile medical images to be registered to obtain initial medical image data;

[0007] Extracting structured image features in the initial medical image data through a shared weight encoding module to obtain a mobile feature map and a fixed feature map, wherein the mobile feature map and the fixed feature map each include five layers of feature maps;

[0008] The mobile feature map and the fixed feature map of each layer are sequentially input into a corresponding estimator, and in each estimator, the generated residual deformation field is fused with a total residual deformation field after upsampling to obtain a target total residual deformation field, wherein each estimator includes a motion decomposition Transformer module and a regeneration module.

[0009] The mobile feature map is registered according to the target total residual deformation field to obtain a target registration image.

[0010] To solve the above technical problems, the embodiment of the application provides a medical image registration device of a motion decomposition model, which comprises:

[0011] A medical image data acquisition unit is configured to acquire fixed medical images and mobile medical images to be registered, and to pre-process the fixed medical images and the mobile medical images to be registered to obtain initial medical image data.

[0012] An image feature extraction unit is configured to extract structured image features in the initial medical image data by a shared weight encoding module to obtain a mobile feature map and a fixed feature map, wherein the mobile feature map and the fixed feature map each comprise five layers of feature maps.

[0013] A residual deformation field generation unit is configured to sequentially input the mobile feature map and the fixed feature map of each layer into a corresponding estimator, and in each estimator, the generated residual deformation field is fused with a total residual deformation field after upsampling to obtain a target total residual deformation field, wherein each estimator includes a motion decomposition Transformer module and a regeneration module.

[0014] A registration image generation unit is configured to register the mobile feature map according to the target total residual deformation field to obtain a target registration image.

[0015] To solve the above technical problems, the application adopts a technical scheme, which is to provide a computer device, comprising one or more processors, and a memory for storing one or more programs, so that the one or more processors implement the medical image registration method of the motion decomposition model according to any one of the above.

[0016] To solve the above technical problems, the application adopts a technical scheme, which is a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the medical image registration method of the motion decomposition model according to any one of the above.

[0017] The embodiment of the application provides a medical image registration method, device and equipment based on a motion decomposition model and a medium. The method comprises the following steps: collecting a fixed medical image and a moving medical image to be registered, and preprocessing the fixed medical image and the moving medical image to obtain initial medical image data; extracting structured image features in the initial medical image data through a shared weight coding module to obtain a moving feature map and a fixed feature map, wherein the moving feature map and the fixed feature map each comprise five layers of feature maps; inputting each layer of the moving feature map and the fixed feature map into a corresponding estimator in sequence, and fusing a generated residual deformation field with a total residual deformation field after upsampling in each estimator to obtain a target total residual deformation field, wherein each estimator comprises a motion decomposition Transformer module and a regeneration module; and registering the moving feature map according to the target total residual deformation field to obtain a target registration image. The embodiment of the application extracts features from medical image data through a shared weight coding module, generates a target total residual deformation field for a moving feature map and a fixed feature map through a motion decomposition Transformer module and a regeneration module, and finally registers the moving feature map according to the target total residual deformation field to obtain a target registration image, thereby improving the accuracy of medical image registration. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the scheme in the present application, the drawings required in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 is the implementation flowchart of the medical image registration method based on the motion decomposition model provided by the embodiment of the present application;

[0020] Figure 2 is the overall architecture schematic diagram provided by the embodiment of the present application;

[0021] Figure 3 is the implementation flowchart of the sub-process in the medical image registration method based on the motion decomposition model provided by the embodiment of the present application;

[0022] Figure 4 is the implementation flowchart of the sub-process in the medical image registration method based on the motion decomposition model provided by the embodiment of the present application;

[0023] Figure 5 is the implementation flowchart of the sub-process in the medical image registration method based on the motion decomposition model provided by the embodiment of the present application;

[0024] Figure 6 is a regenerated module structure schematic diagram provided by an embodiment of the application;

[0025] Figure 7 is a motion decomposition Transformer module structure schematic diagram provided by an embodiment of the application;

[0026] Figure 8 is a medical image registration method sub-process implementation flowchart of a motion decomposition model provided by an embodiment of the application;

[0027] Figure 9 is a medical image registration method sub-process implementation flowchart of a motion decomposition model provided by an embodiment of the application;

[0028] Figure 10 is a CUDA operator call structure schematic diagram provided by an embodiment of the application;

[0029] Figure 11 is a medical image registration device schematic diagram of a motion decomposition model provided by an embodiment of the application;

[0030] Figure 12 is a schematic diagram of a computer device provided by an embodiment of the application. DETAILED DESCRIPTION

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and the drawings are to be regarded as illustrative in nature and are not intended to limit the application; the terminology used in the description and the claims of the present application and the above description of the drawings includes the terms specifically mentioned above as well as their derivatives.

[0032] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that the embodiments described herein are merely examples from a much larger number of embodiments that can be claimed.

[0033] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings.

[0034] The application will be described in detail in conjunction with the drawings and embodiments.

[0035] It should be noted that the medical image registration method of the motion decomposition model provided in the embodiments of the present application is generally executed by a server, and accordingly, the medical image registration device of the motion decomposition model is generally configured in the server.

[0036] Please refer to Figure 1 and Figure 2 , Figure 1 a specific embodiment of the medical image registration method of the motion decomposition model is shown, Figure 2 is a schematic diagram of the overall architecture provided by the embodiments of the present application.

[0037] It should be noted that the method of the present application is not limited to the order of the flowchart shown, and the method includes the following steps: Figure 1

[0038] S1: Collect fixed medical images and moving medical images to be registered, and pre-process the fixed medical images and moving medical images to be registered to obtain initial medical image data.

[0039] In the embodiments of the present application, the medical images can be brain magnetic resonance images and abdominal CT images, or other medical images. The medical images are randomly divided into fixed medical images and moving medical images, and are used as fixed medical images and moving medical images to be registered; the fixed medical images and moving medical images to be registered are respectively image cropped to obtain cropped medical image data, and the cropped medical image data is image normalized to make the gray value of the image between 0 and 1, to obtain initial medical image data, wherein the initial medical image data includes initial fixed medical image data and initial moving medical image data.

[0040] S2: Extract structured image features in the initial medical image data through a shared weight encoding module to obtain a moving feature map and a fixed feature map. The moving feature map and the fixed feature map each include five layers of feature maps.

[0041] In the embodiments of the present application, the encoding module includes a pyramid structure encoder (L=1, 2, 3, 4, 5) with five layers of shared weights, which is used to extract structured image features in the initial medical image data.

[0042] Please refer to Figure 3 , Figure 3 a specific embodiment of step S2 is shown, described as follows:

[0043] S21: Select two images from the initial medical image data in turn as a moving medical image and a fixed medical image, respectively.

[0044] ​S22: input the mobile medical image and the fixed medical image into the shared weight encoding module in sequence, and perform feature extraction on the input mobile medical image and the fixed medical image by a first layer encoder to obtain an initial first layer mobile feature map and an initial first layer fixed feature map.

[0045] S23: perform feature processing on the initial first layer mobile feature map and the initial first layer fixed feature map by two convolution blocks to obtain a first layer mobile feature map and a first layer fixed feature map.

[0046] In the embodiment of the application, two images are sequentially selected from the initial medical image data as a mobile medical image and a fixed medical image; and the mobile medical image and the fixed medical image are sequentially input into the shared weight encoding module. For the first layer encoder (L = 1), the image feature extraction is performed on the input fixed medical image I f and the mobile medical image I m , and the initial first layer mobile feature map and the initial first layer fixed feature map, which includes convolution processing of a convolution layer and activation processing of a leaky ReLU (LReLU) activation function. After the first feature extraction, the channel dimension of the feature map is increased to C. Then, two additional convolution layers, instance normalization and LReLU processing are used on the initial first layer mobile feature map and the initial first layer fixed feature map, and finally the first layer mobile feature map F1 and the first layer fixed feature map M1 with a 2C dimension are obtained.

[0047] S24: sequentially perform feature extraction on the mobile medical image and the fixed medical image by the last four layer encoders to obtain a last four layer mobile feature map and a last four layer fixed feature map.

[0048] Please refer to Figure 4 , Figure 4 a specific implementation of step S24 is shown as follows:

[0049] S241: sequentially perform feature extraction on the mobile medical image and the fixed medical image by the last four layer encoders to obtain an initial last four layer mobile feature map and an initial last four layer fixed feature.

[0050] S242: perform average pooling downsampling processing on any of the initial last four layer mobile feature map and the initial last four layer fixed feature to obtain a last four layer mobile sampling map and a last four layer fixed sampling map.

[0051] S243: perform feature extraction on the last four layer mobile sampling map and the last four layer fixed sampling map by two convolution blocks respectively to obtain the last four layer mobile feature map and the last four layer fixed feature map.

[0052] In the embodiments of the present application, for the feature extraction process of the last four layers of the encoder (L=2, 3, 4, 5), the difference from the feature extraction process of the first layer of encoding is that the feature map is down-sampled by average pooling before being processed into 2 times the channel, to obtain the last four layers of the moving sampling map and the last four layers of the fixed sampling map. Finally, the last four layers of the moving sampling map and the last four layers of the fixed sampling map are subjected to feature extraction through two convolutional blocks respectively, to obtain the last four layers of the moving feature map (M2, M3, M4, M5) and the last four layers of the fixed feature map (F2, F3, F4, F5).

[0053] S3: sequentially input the moving feature map and the fixed feature map of each layer into the corresponding estimator, and in each estimator, fuse the generated residual deformation field after up-sampling with the total residual deformation field to obtain a target total residual deformation field. Each estimator includes a motion decomposition Transformer module and a regeneration module.

[0054] In the embodiments of the present application, a pyramid registration structure is adopted, which has the advantage of reducing the attention calculation range required by each decoding layer, thereby reducing the calculation consumption. Two feature maps F1, F2, F3, F4, F5 and M1, M2, M3, M4, M5 are generated according to the shared weight encoder. The feature map M5 and F5 are sent to the first estimator to obtain the initial residual deformation field φ1 of the coarsest decoding layer as the initial value of the total residual deformation field φ. The estimator is composed of a motion decomposition Transformer module and a regeneration module. The moving feature map M4 is deformed using the up-sampled φ deformation, and the deformed moving feature map is sent into the estimator together with F4 to obtain φ2. Then φ2 is fused with the previous total residual deformation field to obtain the updated φ, and the fusion method is as follows:

[0055]

[0056] Wherein, after the residual sub-field is generated in the s stage, the deformation field is fused with the generated deformation field. The feature maps M3, M2, M1 and F3, F2, F1 undergo similar operations. Finally, the obtained total residual deformation field φ is used to register the moving image Im, thereby obtaining the target registered image.

[0057] Please refer to Figure 5 , Figure 6 and Figure 7 , Figure 5 shows a specific implementation of step S3, Figure 6 is a schematic diagram of the regeneration module structure provided in the embodiments of the present application, Figure 7 is a schematic diagram of the motion decomposition Transformer module structure provided in the embodiments of the present application, which is described in detail as follows:

[0058] S31: input the moving feature map and the fixed feature map of each layer into the corresponding estimator in sequence.

[0059] S32: the motion decomposition Transformer module in the first estimator generates an initial residual deformation field based on the fifth layer moving feature map and the fifth layer fixed feature map.

[0060] Please refer to Figure 8 , Figure 8 A specific implementation of step S3 is shown as follows:

[0061] S321: obtain the feature mapping of the fifth layer moving feature map and the fifth layer fixed feature map through the motion decomposition Transformer module in the first estimator, to obtain the fifth layer moving mapping and the fifth layer fixed mapping.

[0062] S322: perform linear projection and layer normalization processing on the fifth layer moving mapping and the fifth layer fixed mapping, to obtain a query tensor and a key tensor.

[0063] S223: split the query tensor and the key tensor to obtain a split query tensor and a split key tensor.

[0064] S324: calculate a neighborhood attention map based on the split query tensor and the split key tensor, and generate the initial residual deformation field based on the neighborhood attention map.

[0065] In the deep learning-based registration network, the position p in the low-resolution feature map contains a large area of semantic information in the original image, so it usually has the possibility of multiple different motion modes. In order to simulate these possibilities, the multi-head neighborhood attention mechanism is used to decompose different motion modes at the low-resolution level.

[0066] Let F, M ∈ R c×h×w×l represent the fixed and moving feature mappings of the fifth layer from the sub-layer encoder, where h, w, l represent the size of the feature mapping, and c represents the channel number. The feature mappings F and M are subjected to linear projection (proj) and layer normalization (LN) to obtain a query tensor Q (query) and a key tensor K (key). The specific process is as follows:

[0067] Q = LN (proj (F)), K = LN (proj (M));

[0068] Q = {Q (1) ,Q (2) ,…,Q (S)};

[0069] K = {K(1) ,K (2) ,…,K (S)};

[0070] In this operation, the projection operation uses shared weights, and the weights are initialized from N(0,1e). -5 Sampling begins, with the bias initialized to 0. Then, the query tensor Q and key tensor K are segmented according to channels, and S represents the total number of heads.

[0071] Further, a specific implementation of step S324 is provided, including: filling the boundaries of the segmented key tensor using the zero-filling method to obtain a filled key tensor; performing calculation processing based on the segmented query tensor and the filled key tensor using a preset formula to obtain the neighborhood attention map; and obtaining multiple initial residual deformation fields based on the weighted regular displacement field in the neighborhood attention map.

[0072] In this embodiment, c(p) represents the neighborhood of voxel p. For a neighborhood of size n×n×n, ||c(p)||=n 3 The preset formula is as follows:

[0073]

[0074] Where NA(p,s) is the neighborhood attention map, and B∈R S×n×n×n This is a learnable relative positional bias, initialized to all zeros. In this formula, when p is at the boundary position of Q, the neighborhood range on the key tensor K will shrink. Therefore, in this embodiment, K is padded to unify the neighborhood size. Since the registration task sometimes requires voxels outside the field of view to move into the field of view, we use zero-padding to padded the boundaries of the key tensor K. The above formula shows how to calculate the neighborhood attention of the s-th head at position p, thereby decomposing the semantic information of low-resolution voxels, calculating similarity one by one, and preparing for modeling different motion patterns. In addition, the neighborhood attention operation reduces the scope of attention calculation, reduces the amount of computation, and is very friendly to volume processing.

[0075] Furthermore, by calculating the neighborhood attention graph-weighted regularized displacement field, multiple subfields of this layer are obtained: where φ s ∈R h×w×l×3 , V∈R n×n×n V(value) represents the relative position coordinates of the neighborhood centroids. No learning is required; the multi-head attention relationship can be naturally transformed into a multi-coordinate relationship. Through the above steps, a series of deformation subfields for this layer are obtained.

[0076] S33: The target residual deformation field is generated based on the initial residual deformation field by the regeneration module in the first estimator, and the target residual deformation field is used as the initial total residual deformation field.

[0077] Referring to Figure 9 , Figure 9 An embodiment of step S33 is shown as follows:

[0078] S331: Perform three-channel convolution processing on the initial residual deformation field based on the reconstruction module in the first estimator to obtain a convolution layer output result.

[0079] S332: Obtain a three-channel static velocity field by performing scaling and squaring processing on the convolution layer output result.

[0080] S333: Generate the target residual deformation field based on the three-channel static velocity field, and take the target residual deformation field as the initial total residual deformation field.

[0081] An embodiment of the present application includes a reconstruction module to reconstruct the deformation field. The reconstruction module includes a convolution layer with a kernel size greater than or equal to 3. The weights of the convolution layer are initialized as N(0, 1e -5 ), and the bias is initialized as 0. The input is the multiple sub-fields from the motion decomposition Transformer, and the output is a three-channel static velocity field. In this way, the neighborhood information is extracted when combining multiple motion modes. An additional scaling and squaring operation is added in the module, and the static velocity field is used to ensure the differential homeomorphism properties, such as derivability, reversibility, and topological preservation. The deformation field is defined as an ordinary differential equation with respect to time t:

[0082]

[0083] where φ (0) is the identity transformation when t = 0, and φ (1) is the deformation field required by the embodiment of the present application. Further, the scaling and squaring method is used to integrate φ (t) from 0 to 1. Specifically, it is obtained by the following loop:

[0084]

[0085] Specifically, for the output of the convolution layer in each reconstruction module, first multiply it by as the initial After T times of the above formula loop, the residual deformation field φ s of this level is finally obtained. After this module, the embodiment of the present application can obtain the residual field of this level, which fuses multiple motion methods and fully considers the neighborhood relationship.

[0086] S34: input the moving feature maps and the fixed feature maps of the first four layers into the corresponding last four estimators, and fuse the generated residual deformation field after upsampling with the total residual deformation field in each estimator to obtain the target total residual deformation field.

[0087] It should be noted that the motion decomposition Transformer module and the regeneration module in the last four estimators have the same processing process as steps S321-S324 and steps S331-S333 described above. To avoid repetition, details are not described here.

[0088] Please refer to Figure 10 , Figure 10 is a CUDA operator call structure diagram provided by the embodiment of the application.

[0089] For the motion decomposition Transformer module, since there is no better implementation in the existing deep learning framework, the CUDA programming is used for operator implementation in the embodiment of the application. As shown in Figure 10 , the embodiment of the application mainly describes the implementation method of the core forward propagation function and the backward propagation function.

[0090] In Figure 10 , the input of the f_fw_cu function is three tensor type constant references, which are query tensor, key tensor and relative position bias tensor rpb respectively, so that copying and modification can be avoided. The output is an attention score tensor attn. The default shapes of the input and output tensors are as follows:

[0091] query: B, heads, H, W, L, dims;

[0092] key: B, heads, H+2, W+2, L+2, dims;

[0093] rpb: heads, kernel_size, kernel_size, kernel_size;

[0094] Attn: B, heads, H, W, L, kernel_size 3 ;

[0095] Wherein, B represents batch size, heads represents the number of heads, H, W, L represent height, width and length in space respectively, dims represents the number of dimensions of each head, and kernel_size represents the size of a single dimension of the neighborhood. Here, the key is larger than the query because it is filled. This facilitates filling in any form.

[0096] Since the upper limit of the number of threads in a block in the GPU is 1024, the embodiment of the application creates a division of dimensions, and requires that the final product of each dimension cannot be greater than 1024. The size of the first dimension is kernel_size 3 , which is exactly the size of the range to be calculated by attention, the size of the second dimension is B*heads, and the size of the third dimension is H*W*L.

[0097] wherein the fw_kernel function internally writes a calculation function for a point in the output. The embodiment of the application obtains the current number and calculation range according to the three dimensions of the division, and directly returns when exceeding the maximum range of a certain dimension, thereby preventing the invocation of a cuda core that should not be invoked. According to the thread index formula of CUDA, the specific indexes x, y, and z of the thread are obtained. Next, they are used to decompose the index of the output tensor corresponding to the current cuda thread, and short division is adopted, for example, the x dimension represents the product of H, W, and L. First, H is divided by z to obtain the number of H dimensions, and then the remainder is used to divide W to obtain the number of W dimensions, and so on. Similarly, z represents the batch and head dimensions, and short division is also used to separate b and h. The final dimension of y is the cube of kernel_size, we assume that kernel_size is 3, then the last dimension represents which direction of the 27 directions relative to the current q that the current kernel function is calculating, and the specific operation of the last kernel function is to loop along the multi-head dimension dims to calculate the sum of the product of q and k. Finally, the corresponding rpb value is added to assign the passed-in attn tensor.

[0098] wherein the f_bw_cu function realizes back propagation, and is used to calculate the gradients of query, key, and rpb through the attention score gradient returned by the python program part. The function accepts four input parameters, which are the attention back propagation gradient tensor d_attn, the query tensor query, the key tensor key. The return value of the function is a vector containing three tensors, which are the back propagation gradient d_query of the query tensor, the back propagation gradient d_key of the key tensor, and the back propagation gradient d_rpb of the relative position bias. The shapes of the several gradient tensors are as follows:

[0099] d_key:B,heads,H+2,W+2,L+2,dims;

[0100] d_attn:B,heads,height,width,length,kernel_size3;

[0101] d_query:B,heads,H,W,L,dims;

[0102] d_rpb: B, kernel_size, kernel_size, kernel_size.

[0103] First, the zero tensor of the output is created for assignment. Because three tensors are involved in the back propagation calculation, it is difficult to balance the division of multiple dimensions of threads, so the dimensions of CUDA are not divided in the embodiment of the application, and each back propagation function uses a 1-dimensional block division. d_q uses 512 threads, d_k uses 512 threads, and d_rpb uses 64 threads. Then, the three kernel functions fdq_bw_kernel, fdk_bw_kernel and fdrpb_bw_kernel are distributed to the GPU through the floating-point tensor distributor. The back propagation derivation is as follows:

[0104]

[0105] Specifically, the specific kernel function calculation is as follows: d_query is calculated in the same way as the forward propagation logic. Only the key tensor that performs dot product with its neighborhood can affect the query gradient, so the input is the tensor of the back propagation attention gradient and the key tensor of the forward propagation. First, the short division is used to obtain the corresponding index of d_query in the output calculated by the current CUDA thread. In the calculation part, the kernel_size 3 of the query point in the key is multiplied by the last dimension of d_attn and then added. The calculation of d_key is more complex because it needs to calculate which positions in Q produce calculation for a point in K and consider the boundary conditions. Here, the starting position and the ending position of the window on Q corresponding to the value of a certain dimension of K are first calculated, and then a triple loop is performed according to the starting point and the ending point of the window to multiply query and d_attn in the corresponding direction and then add them. The calculation of the corresponding direction uses the coordinates of the current key minus the coordinates of the query. The calculation of d_rpb is simple because its forward propagation is addition, so the derivative of d_attn to rpb is 1, that is, the d_rpb in a domain is equal to the d_attn at the corresponding position. However, because rpb is shared in each neighborhood, it is equivalent to adding the d_attn at the corresponding position. This operation uses the primitive-level addition function of CUDA to accumulate, because it involves parallel writing of different kernel functions to the same memory unit.

[0106] It should be noted that all loops are expanded by the unroll command at compile time to reduce the number of logical judgments in the kernel.

[0107] S4: registering the moving feature map according to the target total residual deformation field to obtain a target registered image.

[0108] Specifically, the total loss of model training in the embodiment of the present application is as follows:

[0109]

[0110] wherein L total is the total loss, λ is the weight of the regularization term, I f is the fixed medical image, I m is the moving medical image, is the deformation field, L reg is the residual loss, if the single modality medical image is registered, L sim is the normalized mutual loss, if the multi-modality medical image is registered, L sim is the local mutual information loss.

[0111] Further, in the model training process, the Adam optimizer and the learning rate decay strategy of the present application are as follows:

[0112]

[0113] wherein lr m represents the learning rate of the mth round, lr init represents the learning rate of the initial round, and the learning round is determined by whether the loss decreases or not, and the training can be stopped when the convergence is reached.

[0114] In the embodiment of the present application, the fixed medical image and the moving medical image to be registered are collected, and the fixed medical image and the moving medical image to be registered are preprocessed to obtain initial medical image data; the structured image features in the initial medical image data are extracted through a shared weight encoding module to obtain a moving feature map and a fixed feature map, wherein the moving feature map and the fixed feature map each include five layers of feature maps; each layer of the moving feature map and the fixed feature map is sequentially input into a corresponding estimator, and the generated residual deformation field is fused with a total residual deformation field after upsampling in each estimator to obtain a target total residual deformation field b, wherein each estimator includes a motion decomposition Transformer module and a regeneration module; the moving feature map is registered according to the target total residual deformation field b to obtain a target registration image. The embodiment of the present application extracts features from medical image data through a shared weight encoding module, generates a target total residual deformation field b from a moving feature map and a fixed feature map through a motion decomposition Transformer module and a regeneration module, and finally registers the moving feature map according to the target total residual deformation field b to obtain a target registration image, thereby facilitating to improve the accuracy of medical image registration.

[0115] Further, the registration method in most existing image registration methods adopts a traditional iterative optimization method, which is time-consuming, the single-stage network precision is not enough, and the cascade network speed is also slow. The registration method of the medical image based on the deep learning pyramid structure provided in the embodiment of the application effectively solves the problem of long registration time and improves the precision. Through the shared weight encoder, the network of the embodiment of the application can perform semantic similar encoding on the moving image and the fixed image in the registration, which is convenient for subsequent deformation field estimation. The motion decomposition Transformer module neural network provided in the embodiment of the application solves and converts the corresponding relationship of multiple groups of features into a deformation field, and the embodiment of the application provides a CUDA accelerated version of the module, which is very fast. The re-generation module provided in the embodiment of the application regenerates the deformation field obtained by the motion decomposition Transformer module without considering the neighborhood relationship, and obtains a deformation field with a non-biholomorphic characteristic, so that the deformation is more reasonable. The embodiment of the application can be used for registering two-dimensional images, three-dimensional images, multi-modal medical images and multi-modal natural images, while some patents will limit the scope of application, such as only applicable to two-dimensional images or three-dimensional images.

[0116] Please refer to Figure 11 , as an implementation of the method shown in Figure 1 , the application provides an embodiment of a medical image registration device of a motion decomposition model, which corresponds to the method embodiment shown in Figure 1 , and the device can be applied to various electronic devices.

[0117] As shown in Figure 11 , the medical image registration device of the motion decomposition model of the embodiment includes a medical image data acquisition unit 51, an image feature extraction unit 52, a residual deformation field generation unit 53 and a registration image generation unit 54, wherein:

[0118] The medical image data acquisition unit 51 is used to acquire fixed medical images and moving medical images to be registered, and to pre-process the fixed medical images and moving medical images to be registered to obtain initial medical image data;

[0119] The image feature extraction unit 52 is used to extract structured image features in the initial medical image data through a shared weight encoding module to obtain a moving feature map and a fixed feature map, wherein the moving feature map and the fixed feature map each include five layers of feature maps;

[0120] The residual deformation field generation unit 53 is configured to sequentially input the moving feature map and the fixed feature map of each layer into a corresponding estimator, and fuse the generated residual deformation field with a total residual deformation field after upsampling in each estimator to obtain a target total residual deformation field, wherein each estimator comprises a motion decomposition Transformer module and a regeneration module.

[0121] The registration image generation unit 54 is configured to register the moving feature map according to the target total residual deformation field to obtain a target registration image.

[0122] Further, the medical image data acquisition unit 51 comprises:

[0123] The image selection unit is configured to sequentially select two images from the initial medical image data as a moving medical image and a fixed medical image, respectively.

[0124] The first image feature extraction unit is configured to sequentially input the moving medical image and the fixed medical image into an encoding module with shared weights, and perform feature extraction on the input moving medical image and the fixed medical image by a first layer encoder to obtain an initial first layer moving feature map and an initial first layer fixed feature map.

[0125] The channel dimension increasing unit is configured to perform feature processing on the initial first layer moving feature map and the initial first layer fixed feature map by two convolution blocks to obtain a first layer moving feature map and a first layer fixed feature map.

[0126] The second image feature extraction unit is configured to sequentially perform feature extraction on the moving medical image and the fixed medical image by four subsequent layer encoders to obtain four subsequent layer moving feature maps and four subsequent layer fixed feature maps.

[0127] Further, the second image feature extraction unit comprises:

[0128] The third image feature extraction unit is configured to sequentially perform feature extraction on the moving medical image and the fixed medical image by the four subsequent layer encoders to obtain an initial four subsequent layer moving feature map and an initial four subsequent layer fixed feature.

[0129] The down-sampling processing unit is configured to perform average pooling down-sampling processing on any of the initial four subsequent layer moving feature maps and the initial four subsequent layer fixed features to obtain four subsequent layer moving sampling maps and four subsequent layer fixed sampling maps.

[0130] The fourth image feature extraction unit is configured to perform feature extraction on the four subsequent layer moving sampling maps and the four subsequent layer fixed sampling maps by two convolution blocks, respectively, to obtain the four subsequent layer moving feature maps and the four subsequent layer fixed feature maps.

[0131] Further, the image feature extraction unit 52 comprises:

[0132] a feature map input unit, configured to sequentially input the moving feature maps and the fixed feature maps of each layer into the corresponding estimators;

[0133] an initial residual deformation field generation unit, configured to generate an initial residual deformation field based on the fifth layer moving feature map and the fifth layer fixed feature map by the motion decomposition Transformer module in the first estimator;

[0134] a residual deformation field regeneration unit, configured to generate a target residual deformation field based on the initial residual deformation field by the regeneration module in the first estimator, and take the target residual deformation field as the initial total residual deformation field;

[0135] a residual deformation field fusion unit, configured to input the moving feature maps and the fixed feature maps of the first four layers into the corresponding last four estimators, and fuse the generated residual deformation field with the total residual deformation field after upsampling in each of the estimators to obtain the target total residual deformation field.

[0136] Further, the initial residual deformation field generation unit comprises:

[0137] a feature mapping unit, configured to obtain feature mappings of the fifth layer moving feature map and the fifth layer fixed feature map by the motion decomposition Transformer module in the first estimator to obtain a fifth layer moving mapping map and a fifth layer fixed mapping map;

[0138] a layer normalization processing unit, configured to perform linear projection and layer normalization processing on the fifth layer moving mapping map and the fifth layer fixed mapping map to obtain a query tensor and a key tensor;

[0139] a tensor segmentation unit, configured to segment the query tensor and the key tensor to obtain a segmented query tensor and a segmented key tensor;

[0140] a neighborhood attention map calculation unit, configured to calculate a neighborhood attention map based on the segmented query tensor and the segmented key tensor, and generate the initial residual deformation field based on the neighborhood attention map.

[0141] Further, the neighborhood attention map calculation unit comprises:

[0142] a boundary padding unit, configured to perform boundary padding on the segmented key tensor by zero padding method to obtain a padded key tensor;

[0143] The neighborhood information calculation unit is configured to calculate and process the neighborhood attention map based on the segmented query tensor and the padded key tensor through a preset formula.

[0144] The regular displacement field calculation unit is configured to obtain the plurality of initial residual deformation fields based on the weighted regular displacement field in the neighborhood attention map.

[0145] Further, the residual deformation field regeneration unit comprises:

[0146] The convolution processing unit is configured to perform three-channel convolution processing on the initial residual deformation field based on the regeneration module in the first estimator to obtain a convolution layer output result.

[0147] The static velocity field generation unit is configured to obtain a three-channel static velocity field by performing scaling and squaring processing on the convolution layer output result.

[0148] The target residual deformation field generation unit is configured to generate the target residual deformation field based on the three-channel static velocity field, and take the target residual deformation field as the initial total residual deformation field.

[0149] To solve the above technical problems, the embodiment of the present application further provides a computer device. For details, please refer to Figure 12 , Figure 12 The basic structure block diagram of the computer device of the present embodiment is shown in the figure.

[0150] The computer device 8 comprises a memory 81, a processor 82 and a network interface 83 which are connected to each other through a system bus. It should be pointed out that only the computer device 8 with three components, i.e. the memory 81, the processor 82 and the network interface 83, is shown in the figure, but it should be understood that all the components shown are not required to be implemented, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device herein is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0151] The computer device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad or a voice control device, etc.

[0152] The memory 81 includes at least one type of readable storage medium, including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8. Of course, the memory 81 can also include both the internal storage unit and the external storage device of the computer device 8. In the present embodiment, the memory 81 is generally used to store an operating system and various application software installed on the computer device 8, such as program codes of the medical image registration method of the motion decomposition model, etc. In addition, the memory 81 can also be used to temporarily store various data that have been output or will be output.

[0153] The processor 82 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In the present embodiment, the processor 82 is used to run program codes or process data stored in the memory 81, such as running program codes of the medical image registration method of the motion decomposition model, to implement various embodiments of the medical image registration method of the motion decomposition model.

[0154] The network interface 83 can include a wireless network interface or a wired network interface, which is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0155] The present application also provides another implementation, i.e., to provide a computer readable storage medium, which stores a computer program, and the computer program can be executed by at least one processor to make the at least one processor execute the steps of the medical image registration method of the motion decomposition model as described above.

[0156] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method of each embodiment of the present application.

[0157] Obviously, the above-described embodiments are only some of the embodiments of the present application, not all the embodiments, and the drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing specific embodiments, or make equivalent replacements to some technical features. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the scope of the patent protection of the present application.

Claims

1. A method for medical image registration using a motion decomposition model, characterized in that, include: Acquire fixed and mobile medical images to be registered, and preprocess the fixed and mobile medical images to be registered to obtain initial medical image data; Structured image features are extracted from the initial medical image data by a weighted encoding module to obtain a moving feature map and a fixed feature map, wherein both the moving feature map and the fixed feature map include five layers of feature maps; Each layer of the moving feature map and the fixed feature map are sequentially input into the corresponding estimator. In each estimator, the generated residual deformation field is upsampled and fused with the total residual deformation field to obtain the target total residual deformation field. Each estimator includes a motion decomposition Transformer module and a regeneration module. The moving feature map is registered based on the target total residual deformation field to obtain the target registered image; The process involves sequentially inputting the moving feature map and the fixed feature map of each layer into the corresponding estimator, and in each estimator, upsampling the generated residual deformation field and fusing it with the total residual deformation field to obtain the target total residual deformation field, including: The moving feature map and the fixed feature map of each layer are sequentially input into the corresponding estimator; The motion decomposition Transformer module in the first estimator generates the initial residual deformation field based on the fifth-layer moving feature map and the fifth-layer fixed feature map; The target residual deformation field is generated by the regeneration module in the first estimator based on the initial residual deformation field, and the target residual deformation field is used as the initial total residual deformation field. The moving feature maps and fixed feature maps of the first four layers are input into the corresponding last four estimators. In each estimator, the generated residual deformation field is upsampled and fused with the total residual deformation field to obtain the target total residual deformation field. The step of generating a target residual deformation field based on the initial residual deformation field through the regeneration module in the first estimator, and using the target residual deformation field as the initial total residual deformation field, includes: Based on the regeneration module in the first estimator, the initial residual deformation field is subjected to three-channel convolution processing to obtain the output result of the convolutional layer. By scaling and squaring the output of the convolutional layer, a three-channel static velocity field is obtained. The target residual deformation field is generated based on the static velocity field of the three channels, and the target residual deformation field is used as the initial total residual deformation field.

2. The medical image registration method for the motion decomposition model according to claim 1, characterized in that, The encoding module includes a five-layer pyramid structure encoder with shared weights. The shared-weight encoding module extracts structured image features from the initial medical image data to obtain moving and fixed feature maps, including: Two images are selected sequentially from the initial medical image data, respectively serving as a mobile medical image and a stationary medical image; The mobile medical image and the fixed medical image are sequentially input into the encoding module with shared weights. The first-layer encoder extracts features from the input mobile medical image and the fixed medical image to obtain an initial first-layer mobile feature map and an initial first-layer mobile feature map. The initial first-layer moving feature map and the initial first-layer moving feature map are processed by two convolutional blocks to obtain the first-layer moving feature map and the first-layer fixed feature map. The mobile medical image and the fixed medical image are sequentially processed by the last four layers of encoders to extract features, resulting in the last four layers of mobile feature maps and the last four layers of fixed feature maps.

3. The medical image registration method for the motion decomposition model according to claim 2, characterized in that, The process involves sequentially extracting features from the mobile medical image and the stationary medical image using a four-layer encoder to obtain four layers of mobile feature maps and four layers of stationary feature maps, including: The mobile medical image and the fixed medical image are sequentially extracted by the last four layers of encoder to obtain the initial last four layers of mobile feature map and the initial last four layers of fixed feature map. The initial four-layer moving feature map and the initial four-layer fixed feature map are subjected to average pooling downsampling to obtain the four-layer moving sample map and the four-layer fixed sample map. Feature extraction is performed on the last four moving sample maps and the last four fixed sample maps using two convolutional blocks respectively, to obtain the last four moving feature maps and the last four fixed feature maps.

4. The medical image registration method for the motion decomposition model according to claim 1, characterized in that, The motion decomposition Transformer module in the first estimator generates the initial residual deformation field based on the fifth-layer moving feature map and the fifth-layer fixed feature map, including: The feature maps of the fifth-layer moving feature map and the fifth-layer fixed feature map are obtained through the motion decomposition Transformer module in the first estimator, resulting in the fifth-layer moving map and the fifth-layer fixed map. Linear projection and layer normalization are performed on the fifth-layer moving map and the fifth-layer fixed map to obtain the query tensor and the key tensor. The query tensor and the key tensor are split to obtain the split query tensor and the split key tensor; The neighborhood attention map is calculated based on the segmented query tensor and the segmented key tensor, and the initial residual deformation field is generated based on the neighborhood attention map.

5. The medical image registration method for the motion decomposition model according to claim 4, characterized in that, The step of calculating a neighborhood attention map based on the segmented query tensor and the segmented key tensor, and generating the initial residual deformation field based on the neighborhood attention map, includes: The partitioned key tensor is filled with zero-filling to obtain a filled key tensor. The neighborhood attention map is obtained by calculating the segmented query tensor and the padded key tensor using a preset formula. Based on the weighted regular displacement field in the neighborhood attention graph, multiple initial residual deformation fields are obtained.

6. A medical image registration device for a motion decomposition model, characterized in that, include: A medical image data acquisition unit is used to acquire fixed medical images and moving medical images to be registered, and to preprocess the fixed medical images and moving medical images to be registered to obtain initial medical image data; The image feature extraction unit is used to extract structured image features from the initial medical image data through a weighted encoding module to obtain a moving feature map and a fixed feature map, wherein both the moving feature map and the fixed feature map include five layers of feature maps; The residual deformation field generation unit is used to input the moving feature map and the fixed feature map of each layer into the corresponding estimator in sequence, and to fuse the generated residual deformation field with the total residual deformation field after upsampling in each estimator to obtain the target total residual deformation field. Each estimator includes a motion decomposition Transformer module and a regeneration module. A registration image generation unit is used to register the moving feature map according to the total residual deformation field of the target to obtain a target registration image; The residual deformation field generation unit includes: The feature map input unit is used to sequentially input the moving feature map and the fixed feature map of each layer into the corresponding estimator; The initial residual deformation field generation unit is used to generate the initial residual deformation field based on the fifth layer moving feature map and the fifth layer fixed feature map in the motion decomposition Transformer module of the first estimator. The residual deformation field regeneration unit is used to generate a target residual deformation field based on the initial residual deformation field through the regeneration module in the first estimator, and use the target residual deformation field as the initial total residual deformation field; The residual deformation field fusion unit is used to input the moving feature map and the fixed feature map of the first four layers into the corresponding last four estimators. In each estimator, the generated residual deformation field is upsampled and fused with the total residual deformation field to obtain the target total residual deformation field. The residual deformation field regeneration unit includes: The convolution processing unit is used to perform three-channel convolution processing on the initial residual deformation field based on the regeneration module in the first estimator to obtain the output result of the convolution layer. A static velocity field generation unit is used to obtain a three-channel static velocity field by scaling and squaring the output of the convolutional layer. The target residual deformation field generation unit is used to generate the target residual deformation field based on the static velocity field of the three channels, and to use the target residual deformation field as the initial total residual deformation field.

7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the medical image registration method of the motion decomposition model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the medical image registration method of the motion decomposition model as described in any one of claims 1 to 5.

Images