A brain image registration method, device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present application disclose a brain image registration method and device, electronic equipment, storage medium and program product. The method comprises: obtaining an individual brain image and a template brain image, and a pre-trained registration model; inputting the individual brain image and the template brain image into the registration model, to register the template brain image onto the individual brain image based on the following steps by the registration model: using an affine registration network to perform affine registration associated with the individual brain image and the template brain image, to obtain an affine transformation matrix, and using a first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix, to obtain an intermediate brain image; using a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image, to obtain a deformation field, and using a second spatial transformation layer to perform spatial transformation of the intermediate brain image based on the deformation field, to obtain a target brain image. The problem that the brain image registration process is relatively cumbersome is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image registration technology, and in particular to a brain image registration method, apparatus, electronic device, storage medium and program product. Background Technology

[0002] Brain image registration is a core technology in the field of neuroimaging, and it is of great significance for the early diagnosis of diseases, the evaluation of treatment plans, and the exploration of the relationship between brain function and structure in neuroscience research.

[0003] However, current brain image registration schemes suffer from a cumbersome registration process, which urgently needs to be addressed. Summary of the Invention

[0004] This invention provides a brain image registration method, apparatus, electronic device, storage medium, and program product, which solves the problem of the cumbersome brain image registration process.

[0005] According to one aspect of the present invention, a brain image registration method is provided, which may include: acquiring an individual brain image and a template brain image, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer; inputting the individual brain image and the template brain image into the registration model, so as to register the template brain image onto the individual brain image through the registration model based on the following steps: using the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and using the first spatial transformation layer to perform a spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image; using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and using the second spatial transformation layer to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0006] According to another aspect of the present invention, a brain image registration apparatus is provided, which may include: a registration model acquisition module for acquiring an individual brain image and a template brain image, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module including an affine registration network and a first spatial transformation layer, and the fine registration module including a deformation registration network and a second spatial transformation layer; and a template brain image input module for inputting the individual brain image and the template brain image into the registration model, so as to register the template brain image based on the following two sub-modules through the registration model. Registration onto individual brain images: The intermediate brain image acquisition submodule is used to perform affine registration with the individual brain image and the template brain image using an affine registration network to obtain an affine transformation matrix. Then, using the first spatial transformation layer, based on the affine transformation matrix, it performs a spatial transformation of the template brain image to obtain the intermediate brain image. The target brain image acquisition submodule is used to perform deformation registration on the intermediate brain image and the individual brain image using a deformation registration network to obtain a deformation field. Then, using the second spatial transformation layer, based on the deformation field, it performs a spatial transformation of the intermediate brain image to obtain the target brain image.

[0007] According to another aspect of the present invention, an electronic device is provided, which may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to implement the brain image registration method provided in any embodiment of the present invention when executed by the at least one processor.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon for causing a processor to execute and implement the brain image registration method provided in any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the brain image registration method provided in any embodiment of the present invention.

[0010] The technical solution of this invention acquires an individual brain image and a template brain image, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, while the fine registration module includes a deformation registration network and a second spatial transformation layer. This enables the acquisition of the individual brain image, the template brain image, and the registration model for brain image registration. By inputting the individual brain image and the template brain image into the registration model, the template brain image is registered onto the individual brain image using the following steps. The registration model achieves end-to-end brain image registration. It utilizes an affine registration network to perform affine registration associated with individual and template brain images, obtaining an affine transformation matrix. A first spatial transformation layer then performs a spatial transformation of the template brain image based on the affine transformation matrix, resulting in an intermediate brain image globally aligned by the coarse registration module. A deformation registration network then performs deformation registration on the intermediate and individual brain images, obtaining a deformation field. A second spatial transformation layer then performs a spatial transformation of the intermediate brain image based on the deformation field, resulting in the target brain image precisely registered by the fine registration module. This technical solution, through the registration model, enables brain image registration with only a simple end-to-end process, thus solving the problem of the cumbersome brain image registration process.

[0011] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a brain image registration method provided according to an embodiment of the present invention.

[0014] Figure 2 This is a flowchart of another brain image registration method provided according to an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram illustrating the principle of calculating the affine transformation matrix in another brain image registration method provided by an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of a key point detection network in another brain image registration method provided by an embodiment of the present invention.

[0017] Figure 5 This is a flowchart of another brain image registration method provided according to an embodiment of the present invention.

[0018] Figure 6 This is a flowchart of another brain image registration method provided according to an embodiment of the present invention.

[0019] Figure 7 This is a schematic diagram of a deformation registration network in another brain image registration method provided according to an embodiment of the present invention.

[0020] Figure 8 This is a schematic diagram of a residual block in another brain image registration method provided according to an embodiment of the present invention.

[0021] Figure 9 This is a schematic diagram of attention gating in another brain image registration method provided by an embodiment of the present invention.

[0022] Figure 10 This is a schematic diagram of the training phase in an optional example of another brain image registration method provided according to an embodiment of the present invention.

[0023] Figure 11 This is a schematic diagram of the test phase in an optional example of another brain image registration method provided according to an embodiment of the present invention.

[0024] Figure 12 This is a schematic diagram of the visualization result in an optional example of another brain image registration method provided according to an embodiment of the present invention.

[0025] Figure 13 This is a structural block diagram of a brain image registration device provided according to an embodiment of the present invention.

[0026] Figure 14 This is a schematic diagram of the structure of an electronic device that implements the brain image registration method of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The same applies to "target," "original," etc., and will not be repeated here. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Before introducing the embodiments of the present invention, the application scenarios of the embodiments of the present invention, the significance of implementing the embodiments of the present invention, the implementation process of the relevant brain image registration scheme, and the reasons for the problem that the brain image registration process is relatively cumbersome will be explained by way of example, so as to better understand why the solution proposed by the embodiments of the present invention solves the problem of the relatively cumbersome brain image registration process.

[0030] Template brain images provide a standardized reference framework, enabling the comparison and analysis of anatomical structures from different individuals in a consistent manner. However, despite the common anatomical structure of the brain, each person's brain structure is individualized. Therefore, in specific situations, it may be necessary to generate individualized template brain images to more accurately reflect the uniqueness of each person's brain structure, which is crucial for understanding inter-individual differences. Quantitative methods based on template brain images assume that, at a certain level of representation, the topological structure of the brain is invariant in normal individuals, and the differences between individuals lie only in the details of the shape of the individual's brain structure. With these assumptions, the problem of locating the anatomical structure of an individual's template brain image becomes a template-individual brain image registration problem. Specifically, the goal is to map the template brain image to the space of the individual brain image through a series of spatial transformations, taking into account local shape differences.

[0031] Brain image registration, by precisely aligning brain images from different individuals, time points, and / or even different imaging techniques, enables researchers and clinicians to analyze brain changes at a fine level. This provides crucial technical support for revealing the patterns and mechanisms of brain development, aging, and disease states, contributing to a deeper understanding of brain science. Template-individual brain image registration helps identify specific regions or structures in the brain while preserving the complete anatomical and topological information of the individual image. It plays a vital role in clinical practice; for example, in neurosurgery or the treatment of neurological diseases, accurate region-of-interest localization is crucial for surgical planning and treatment outcomes. In particular, the process of precisely aligning individual brain images with template brain images assists doctors in more accurately determining the location of regions of interest and clarifying their relative positions to functional areas of the brain. This is the significance of implementing the embodiments of this invention.

[0032] Relevant brain image registration schemes include optimization-based iterative registration techniques. While these schemes have made some progress in handling various deformations, they are highly dependent on basic registration, resulting in a cumbersome computational process, high computational cost, and long computation time. To improve upon these schemes, a deep learning-based template-individual registration model has been proposed. Although this scheme has achieved certain advantages in computational efficiency and registration performance, it still requires basic registration using pre-alignment tools before inputting the model. Otherwise, it is difficult to capture large or complex deformations, thus affecting the registration effect. However, this scheme still suffers from the cumbersome brain image registration process due to the need for pre-alignment tools.

[0033] To address this issue, this invention employs a registration model for brain image registration, enabling registration through a simple end-to-end process, thus resolving the problem of the cumbersome nature of brain image registration. This will be explained in detail below.

[0034] Figure 1 This is a flowchart of a brain image registration method provided in an embodiment of the present invention. This embodiment is applicable to brain image registration. The method can be executed by the brain image registration device provided in this embodiment of the present invention. The device can be implemented by software and / or hardware, and can be integrated into an electronic device, which can be various user terminals or servers.

[0035] See Figure 1 The method of this invention specifically includes the following steps.

[0036] S110. Acquire individual brain images and template brain images, as well as a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer.

[0037] Here, an individual brain image can be understood as an image of an individual's brain; an individual brain image is, for example, a brain MRI image or a brain computed tomography (CT) image, etc., without being specifically limited here; an individual brain image can be represented in grayscale form.

[0038] A template brain image is an image that serves as a template for understanding an individual's brain image; the image type of a template brain image can be the same as that of an individual brain image; a template brain image can be represented in grayscale.

[0039] A registration model can be understood as a pre-trained model for registering brain images. Registration models can adaptively register individual brain images and template brain images with varying degrees of appearance differences, and achieve precise localization of brain regions through registration. This can provide strong technical support for future medical image analysis tasks, especially in applications requiring efficient and accurate registration, such as radiotherapy planning, disease progression monitoring, and multimodal image fusion.

[0040] The coarse registration module can be understood as a module used for coarse registration of brain images.

[0041] Correspondingly, the fine registration module can be understood as a module used for precise registration of brain images. The fine registration module can be a non-linear deep registration network to further refine the local registration.

[0042] Affine registration networks can be understood as networks used for affine registration with individual brain images and template brain images. The first spatial transformation layer can be understood as a network used for spatial transformation of the template brain image. Deformation registration networks can be understood as networks used for deformation registration of intermediate brain images and individual brain images. The first spatial transformation layer can be understood as a network used for spatial transformation of intermediate brain images.

[0043] In this embodiment of the invention, individual brain images and template brain images can be acquired, as well as a registration model including a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer.

[0044] In this embodiment of the invention, standard preprocessing steps can be performed on the individual brain image and the template brain image to perform skull stripping, cut off non-brain regions and unnecessary black background parts in the image, resample the image to a size of 128×128×128, normalize the gray values ​​to the range of [0,1], and update the individual brain image and the template brain image based on the obtained preprocessing results. Accordingly, at least one of the following images mentioned later, such as the individual gradient map, the template gradient map, the sample individual brain image, and the sample template brain image, can be subjected to the above preprocessing.

[0045] S120. Input the individual brain image and the template brain image into the registration model, and through the registration model, execute steps S1201~S1202 to register the template brain image onto the individual brain image.

[0046] In this embodiment of the invention, an individual brain image and a template brain image can be input into a registration model. The template brain image is then registered onto the individual brain image by performing the steps S1201 to S1202 described below, thereby obtaining the target brain image.

[0047] S1201. Using an affine registration network, perform affine registration with the individual brain image and the template brain image to obtain an affine transformation matrix. Then, using the first spatial transformation layer, perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0048] The affine transformation matrix can be understood as the matrix obtained by affine registration associated with the individual brain image and the template brain image.

[0049] The intermediate brain image can be understood as an image obtained by spatial transformation of a template brain image.

[0050] In this embodiment of the invention, the affine registration network can perform affine registration with the individual brain image and the template brain image to obtain an affine transformation matrix, and the first spatial transformation layer can perform spatial transformation on the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0051] For example, a first spatial transformation layer can be used, based on an affine transformation matrix. Through formula Perform template brain image Spatial transformation yields mesobrain images .in," " indicates a spatial transformation operation.

[0052] For example, affine registration can be performed to associate individual brain images with template brain images to obtain affine transformation matrices. And using the first spatial transformation layer, the template brain image Through affine transformation matrix Linear registration to individual brain images The individual space is used to obtain the intermediate brain image, thereby achieving a rough global alignment between the template and the individual.

[0053] S1202. Using a deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. Then, using a second spatial transformation layer, spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image.

[0054] The deformation field can be understood as the location obtained by deformation registration of the intermediate brain image and the individual brain image.

[0055] The target brain image can be understood as the registration result obtained from brain image registration.

[0056] In this embodiment of the invention, the deformation registration network can perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and the second spatial transformation layer can perform spatial transformation on the intermediate brain image based on the deformation field to obtain the target brain image.

[0057] The technical solution of this invention acquires an individual brain image and a template brain image, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, while the fine registration module includes a deformation registration network and a second spatial transformation layer. This enables the acquisition of the individual brain image, the template brain image, and the registration model for brain image registration. By inputting the individual brain image and the template brain image into the registration model, the template brain image is registered onto the individual brain image using the following steps. The registration model achieves end-to-end brain image registration. It utilizes an affine registration network to perform affine registration associated with individual and template brain images, obtaining an affine transformation matrix. A first spatial transformation layer then performs a spatial transformation of the template brain image based on the affine transformation matrix, resulting in an intermediate brain image globally aligned by the coarse registration module. A deformation registration network then performs deformation registration on the intermediate and individual brain images, obtaining a deformation field. A second spatial transformation layer then performs a spatial transformation of the intermediate brain image based on the deformation field, resulting in the target brain image precisely registered by the fine registration module. This technical solution, through the registration model, enables brain image registration with only a simple end-to-end process, thus solving the problem of the cumbersome brain image registration process.

[0058] Figure 2This is a flowchart of another brain image registration method provided in this embodiment of the invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, the coarse registration module further includes a keypoint detection network to register the template brain image to the individual brain image using the registration model and based on the following steps: using the keypoint detection network to detect individual keypoints from the individual brain image, and using the keypoint detection network to detect template keypoints from the template brain image; correspondingly, using an affine registration network, affine registration associated with the individual brain image and the template brain image is performed to obtain an affine transformation matrix, including: using an affine registration network to perform affine registration from template keypoints to individual keypoints to obtain an affine transformation matrix. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0059] See Figure 2 The method in this embodiment may specifically include the following steps.

[0060] S210. Acquire individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The coarse registration module also includes a key point detection network.

[0061] Among them, the key point detection network can be understood as a network used to detect key points in an image.

[0062] Understandably, the high heterogeneity between individual brain images and template brain images poses a significant challenge to the registration of registration models. To address this challenge, a keypoint detection network can be designed into the registration model. This keypoint detection network can learn features in the image that are useful for registration. These features are highly resistant to noise and changes in image quality, which can make the registration model more robust.

[0063] S220. Input the individual brain image and the template brain image into the registration model, and through the registration model, execute steps S2201 to S2203 to register the template brain image onto the individual brain image.

[0064] S2201. Using a keypoint detection network, detect individual keypoints from individual brain images, and use a keypoint detection network to detect template keypoints from template brain images.

[0065] Individual keypoints can be understood as keypoints in an individual brain image, and specifically, individual keypoints can be understood as keypoints that characterize features in an individual brain image that are useful for registration.

[0066] Template keypoints can be understood as keypoints in a template brain image, and specifically, they can be understood as keypoints that characterize features in a template brain image that are useful for registration.

[0067] It is understandable that the expressions such as the first key point, the second key point, the third key point, and the fourth key point in the whole text are all key points. They are simply named differently to distinguish key points in different application scenarios, and are not specific limitations on their substantive content.

[0068] In this embodiment of the invention, global coarse registration can be guided by detecting key points. Since closed-form key points are independent of their initial positions and are robust to spatial misalignment, key points can be closed-form.

[0069] In this embodiment of the invention, the key point detection network can extract key points that are useful for registration from individual brain images and template brain images, so that a more effective affine transformation matrix can be obtained in the subsequent process through the key points.

[0070] In this embodiment of the invention, individual key points can be detected from individual brain images using a key point detection network, and template key points can be detected from template brain images using a key point detection network.

[0071] In this embodiment of the invention, individual key points and template key points can be represented in the form of point sets. For example, the key point set corresponding to the template brain image can be detected. Key point set corresponding to individual brain images This set of key points Includes template key points, this set of key points Including individual key points, where the key point set and All are of size The matrix, that is, the matrix from which each image can be extracted. There are 1 key points, and each key point is a... Dimensional vector.

[0072] S2202. Using an affine registration network, perform affine registration from template keypoints to individual keypoints to obtain an affine transformation matrix. Then, using the first spatial transformation layer, based on the affine transformation matrix, perform spatial transformation of the template brain image to obtain an intermediate brain image.

[0073] In this embodiment of the invention, an affine registration network can be used to perform affine registration from template keypoints to individual keypoints to obtain an affine transformation matrix.

[0074] For example, see Figure 3 By using a keypoint detection network, a keypoint set can be obtained. and key point set ,in, , The dimension representing the keypoints in the keypoint set, i.e. Each keypoint represents a 3D vector. Based on the key point set and key point set You can get Key points ;Will Key points Input is fed into an affine registration network to utilize the affine registration network, based on... Key points Using a differentiable closed-form expression, affine registration is performed from template keypoints to individual keypoints, and the affine transformation matrix is ​​calculated. To achieve alignment of key points; utilizing the first spatial transformation layer, based on the affine transformation matrix. Spatial transformation of the template brain image is performed to obtain the intermediate brain image.

[0075] In this embodiment of the invention, the affine transformation matrix is ​​calculated. The process is a dynamic learning of the optimal affine transformation matrix. The objective function for learning the optimal affine transformation matrix can be... ,in, This represents the F-norm (Frobenius). In a homogeneous coordinate system ,Right now , here This refers to the affine transformation matrix currently being calculated during the learning process, while A represents the affine transformation matrix calculated historically during the learning process, such as the previously calculated affine transformation matrix. It can be understood that the above objective function can be derived using the following formula: ; ,in, of This can be understood as the objective function mentioned above. .

[0076] S2203. Using a deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. Then, using a second spatial transformation layer, spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image.

[0077] The technical solution of this invention utilizes a keypoint detection network to detect keypoints and obtains an affine transformation matrix based on the detected keypoints. This allows for registration based on stable keypoints, independent of differences in global pixel intensity. This enables the registration model to learn salient features useful for registration, enhancing its robustness under complex deformations, large deformations, noise, and uneven grayscale distribution, thus giving the registration model high robustness.

[0078] An optional technical solution is that the key point detection network includes multiple first convolutional layers, multiple second convolutional layers, and pooling layers used in conjunction with each first convolutional layer. The multiple first convolutional layers and multiple second convolutional layers are alternately set in the key point detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0079] The first convolutional layer can be understood as a convolutional layer with a stride smaller than that of the second convolutional layer.

[0080] The second convolutional layer can be understood as a convolutional layer with a stride larger than that of the first convolutional layer.

[0081] It is understandable that the terms "first convolutional layer," "second convolutional layer," "third convolutional layer," and "fourth convolutional layer" used throughout the text are all essentially convolutional layers. These different names are used to distinguish convolutional layers with different application scenarios or parameters, and are not intended to limit their actual content.

[0082] Pooling layers are pooling layers used in conjunction with the first convolutional layer.

[0083] In this embodiment of the invention, the keypoint detection network includes multiple first convolutional layers, multiple second convolutional layers, and multiple pooling layers. The multiple pooling layers and the multiple first convolutional layers can correspond to each other. Each first convolutional layer is paired with one of the multiple pooling layers corresponding to it. The multiple first convolutional layers and the multiple second convolutional layers are alternately arranged in the keypoint detection network, that is, the multiple first convolutional layers and the multiple second convolutional layers can be interleaved with each other. The stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0084] Understandably, since the centroid layer can calculate the centroid of the activation map in each channel of the keypoint detection network output, which is the center position of the feature estimated by calculating the weighted average of voxel intensities, the keypoint detection network can also include a centroid (Center of Mass, CoM) layer as the output layer, whose output centroid can be used for keypoints.

[0085] For example, see Figure 4The keypoint detection network consists of five first convolutional layers with a stride of 1, four second convolutional layers with a stride of 2, pooling layers used in conjunction with each first convolutional layer, and one centroid layer. Each convolutional layer employs... The convolutional kernels are configured such that each convolutional layer uses the instance normalized and rectified linear unit (ReLU) activation function. Understandably, the second convolutional layer with a stride of 2, compared to the first convolutional layer with a stride of 1, can reduce the size of the feature map by half, replacing pooling layers to achieve downsampling. Furthermore, since the second convolutional layer with a stride of 2 does not discard any feature information, it makes the registration model more robust, reduces the number of parameters in the registration model, and improves convergence speed.

[0086] In this embodiment of the invention, by making the key point detection network include multiple first convolutional layers, multiple second convolutional layers, and pooling layers used in conjunction with each first convolutional layer, and by alternately setting the multiple first convolutional layers and multiple second convolutional layers in the key point detection network, and by making the stride of the first convolutional layer smaller than the stride of the second convolutional layer, the registration model can be made more robust, while reducing the number of registration model parameters and improving the convergence speed.

[0087] Another optional technical solution involves pre-training the registration model as follows: For an original detection network with the same network structure as the keypoint detection network, the sample individual image is input into the original detection network to obtain the first keypoint; For a randomized random affine transformation matrix, the sample individual image is spatially transformed using the random affine transformation matrix to obtain a transformed brain image, and the first keypoint is spatially transformed using the random affine transformation matrix to obtain the second keypoint; The transformed brain image is input into the original detection network to obtain the third keypoint; Based on at least the second and third keypoints, the parameters in the original detection network are adjusted to obtain pre-trained parameters; The pre-trained parameters are used as initialization parameters for the registration model during training to train the registration model.

[0088] The original detection network can be understood as an untrained network with the same network structure as the keypoint detection network.

[0089] The sample individual image can be understood as the individual brain image used as a sample.

[0090] The first keypoint can be understood as a keypoint detected from the sample individual image using the original detection network. The second keypoint can be understood as a keypoint obtained by spatially transforming the first keypoint. The third keypoint can be understood as a keypoint detected from the transformed brain image using the original detection network.

[0091] A random affine transformation matrix can be understood as an affine transformation matrix obtained by randomization.

[0092] Transformed brain images can be understood as images obtained by spatially transforming individual sample images.

[0093] Pre-trained parameters can be understood as parameters obtained by adjusting the parameters in the original detection network based at least on the second and third keypoints.

[0094] Initialization parameters can be understood as the parameters initialized during the training of the registration model.

[0095] In this embodiment of the invention, in order to enhance the robustness of the registration model, a self-supervised pre-training strategy can be adopted for the original detection network to obtain pre-training parameters, which are then used as initialization parameters for the registration model during the training process to train the registration model.

[0096] In this embodiment of the invention, a sample individual image can be input into the original detection network to obtain a first key point; a spatial transformation of the sample individual image can be performed using a random affine transformation matrix to obtain a transformed brain image, and a spatial transformation of the first key point can be performed using a random affine transformation matrix to obtain a second key point; the transformed brain image can be input into the original detection network to obtain a third key point; the parameters in the original detection network can be adjusted based on at least the second and third key points to obtain pre-training parameters; and the pre-training parameters can be used as initialization parameters to train a registration model.

[0097] For example, for the original detection network, the sample individual images The input is fed into the original detection network so that the original detection network can detect individual sample images. Uniform sampling is performed on the coordinate grid to select a set of random key points as the first key point. For the randomized affine transformation matrix T obtained from the uniform distribution in the parameter space, the individual sample images are processed using the randomized affine transformation matrix T. Spatial transformations are performed to obtain transformed brain images, and the first keypoint is determined using a random affine transformation matrix T. The spatial transformation yields the second keypoint; the transformed brain image is input into the original detection network to obtain the third keypoint; based at least on the second and third keypoints, the objective function is minimized. The parameters in the original detection network are adjusted to obtain pre-trained parameters; these pre-trained parameters are then used as initialization parameters for the registration model during training to train the registration model.

[0098] In this embodiment of the invention, pre-trained parameters are used as initialization parameters for the registration model during training to obtain the registration model. For example, pre-trained parameters can be used as initialization parameters for the registration model during training to obtain an intermediate model, and then the intermediate model as a whole is trained to obtain the registration model. For example, the intermediate model can be trained for 400 rounds, with each round containing 200 iterations. The initial learning rate is set to 0.0005, and then every 60 rounds of training, the learning rate is reduced to 0.5 times the previous rate. The loss function uses four parameters. The optimal value can be determined using a grid search method, which yields the result when... The best results were achieved when the model was trained on a single NVIDIA TITIAN Xp GPU and Intel(R) Xeon(R) CPU (v4 @2.40 GHz), and the model that achieved the highest Dice score on the validation set was selected as the registered model after training.

[0099] In this embodiment of the invention, to enhance the robustness of the model to image quality, such as robustness in the presence of noise, noise can be added to a preset number or preset proportion of training samples used to pre-train the registration model. For example, random Gaussian noise with a uniform sampling variance range of [0, 0.25] can be added to half of the training samples, and each training sample is updated according to the noise addition result. The training samples may include at least individual sample images.

[0100] In this embodiment of the invention, after the registration model is trained, it can be tested. For example, multiple test samples can be obtained. For each test sample, the test template image in the test sample can be segmented into 35 brain regions using FreeSurfer. The test sample is then updated based on the segmentation results. Multiple test samples can be used to evaluate the registration performance of the registration model during the testing phase. For example, the accuracy of the affine transformation matrix and deformation field determined by the registration model can be evaluated.

[0101] In this embodiment of the invention, by determining the second key point based on the obtained first key point, and obtaining pre-training parameters for use as initialization parameters based on the second key point and the third key point, the robustness of the registration model can be enhanced.

[0102] Figure 5This is a flowchart of another brain image registration method provided in this embodiment of the invention. This embodiment is based on the above-mentioned technical solutions and optimized. In this embodiment, optionally, the brain image registration method further includes: obtaining an individual gradient map corresponding to an individual brain image, and obtaining a template gradient map corresponding to a template brain image; inputting the individual brain image and the template brain image into a registration model, including: inputting the individual brain image, the individual gradient map, the template brain image, and the template gradient map into the registration model; using a keypoint detection network to detect individual keypoints from the individual brain image, and using a keypoint detection network to detect template keypoints from the template brain image, including: using a keypoint detection network to detect individual keypoints from the individual brain image based on the individual gradient map, and using a keypoint detection network to detect template keypoints from the template brain image based on the template gradient map. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0103] See Figure 5 The method in this embodiment may specifically include the following steps.

[0104] S310. Acquire individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The coarse registration module also includes a key point detection network.

[0105] S320. Obtain the individual gradient map corresponding to the individual brain image, and obtain the template gradient map corresponding to the template brain image.

[0106] In this embodiment of the invention, individual gradient maps and template gradient maps can be obtained for subsequent key point detection using the individual gradient maps and template gradient maps.

[0107] S330. Input the individual brain image, individual gradient map, template brain image and template gradient map into the registration model, and execute steps S3301 to S3303 to register the template brain image onto the individual brain image.

[0108] In this embodiment of the invention, the individual gradient map and the template gradient map will be collectively referred to as gradient maps in the following text.

[0109] Understandably, in order to enhance registration accuracy and improve the registration model's sensitivity to image details, gradient maps can also be used as input to the registration model. This can effectively highlight edge features in the image, and edge features can highlight the contours and boundaries of anatomical structures. Emphasizing edge features can help the registration model more accurately identify and align corresponding structures between different images, thereby promoting the keypoint detection network to learn a more uniform and meaningful set of points.

[0110] S3301. Using a keypoint detection network, based on individual gradient maps, detect individual keypoints from individual brain images; and using a keypoint detection network, based on template gradient maps, detect template keypoints from template brain images.

[0111] In this embodiment of the invention, an individual keypoint can be detected from an individual brain image based on an individual gradient map using a keypoint detection network, and a template keypoint can be detected from a template brain image based on a template gradient map using a keypoint detection network.

[0112] For example, individual brain images can be used. Individual gradient map Template brain images and template gradient map The data is input into the keypoint detection network so that the keypoint detection network can detect data based on individual gradient maps. From individual brain images Individual key points were detected, and based on template gradient maps... From template brain images Key points of the template were detected.

[0113] S3302. Using an affine registration network, perform affine registration from template keypoints to individual keypoints to obtain an affine transformation matrix. Then, using the first spatial transformation layer, based on the affine transformation matrix, perform spatial transformation of the template brain image to obtain an intermediate brain image.

[0114] S3303. Using a deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. Then, using a second spatial transformation layer, spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image.

[0115] The technical solution of this invention detects individual key points from individual brain images based on individual gradient maps and detects template key points from template brain images using template gradient maps. In other words, using gradient maps as a factor to obtain key points can improve the sensitivity of the key point detection network to image details, thereby enhancing the registration accuracy of the registration model.

[0116] An optional technical solution is that the total loss function used in the training process of the registration model is at least based on the gradient similarity loss function, wherein the gradient similarity loss function is obtained from the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by using the affine transformation matrix to perform a spatial transformation of the template gradient map.

[0117] The total loss function can be understood as the total loss function applied during the training process of the registration model.

[0118] The gradient similarity loss function can be understood as a similarity loss function that calculates the loss associated with the gradient.

[0119] In this embodiment of the invention, an intermediate gradient map can be obtained by spatial transformation of the template gradient map using an affine transformation matrix. A gradient similarity loss function can be obtained based on the individual gradient map and the intermediate gradient map. The total loss function can be obtained based at least on the gradient similarity loss function.

[0120] In this embodiment of the invention, the total loss function can also be calculated based on at least one of the brain image similarity loss function, fine registration loss function, and regularization loss function.

[0121] For example, referring to the examples above, during the training of the registration model, the Normalized Cross-Correlation (NCC) function can be used as the similarity measure function between the brain image and the gradient map for both the coarse and fine registration modules. A regularized loss function can be used to control the smoothness of the deformation field. Specifically, for the affine registration network, the brain image similarity loss function used is... The gradient similarity loss function used is: ,in, and These represent the individual brain image and the template brain image, respectively. and These represent the individual gradient map and the template gradient map, respectively. and These represent the intermediate brain image and intermediate gradient map of the sample, respectively. The affine transformation matrix A used is obtained based on keypoint pairs, i.e. , This represents a local window centered at voxel x; for the affine registration network, the fine registration loss function used includes... as well as ,in, These are the target brain image and the template gradient map, respectively, involving the deformation field. It is based on the intermediate brain images of the samples and the individual brain images of the samples, as well as the gradient maps corresponding to both. , The regularization loss function can be... ,in, This represents the dense deformation field output by the registration model. Represents the deformation field The gradient at voxel x; based on the above loss functions, the total loss function can be obtained as follows: .

[0122] In this embodiment of the invention, the total loss function can be obtained at least based on the gradient similarity loss function, thereby enabling the gradient map to be used as a factor in obtaining the target brain image. This can improve the sensitivity of the trained registration model to image details, thereby further enhancing the registration accuracy of the registration model.

[0123] Figure 6 This is a flowchart of another brain image registration method provided in this embodiment of the invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, the deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is implemented based on an attention gating mechanism. The explanations of terms that are the same as or corresponding to those in the above embodiments are not repeated here.

[0124] See Figure 6 The method in this embodiment may specifically include the following steps.

[0125] S410. Acquire individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The deformation registration network includes an encoder and a decoder. The skip connection between the encoder and the decoder is implemented based on an attention gating mechanism.

[0126] Skip Connections can be understood as an architectural design in deep learning, mainly used to solve the vanishing gradient problem in registration models and to facilitate more direct information transmission.

[0127] Attention gate (AG) can be understood as a mechanism used in a registration model to dynamically focus on a specific part of the input data.

[0128] In this embodiment of the invention, the deformation registration network may include an encoder and a decoder. Skip connections between the encoder and decoder can be achieved based on an attention gating mechanism. Through the attention gating mechanism, local registration can be dynamically adjusted to learn a dense deformation field, focusing on the important features of the input data of the deformation registration network and suppressing irrelevant information. That is, it can focus on learning the local deformation of the input data of the deformation registration network. By assigning a "soft" attention weight to each feature, the deformation registration network can highlight important features while suppressing unimportant features, thereby enabling the deformation registration network to focus on local areas that are difficult to register, achieving one-step refinement of local details in registration, and thus achieving accurate registration of brain images.

[0129] For example, see Figure 7The deformation registration network consists of an encoder and a decoder; the encoder includes one input layer, three residual blocks, and three third convolutional layers with a stride of 2; the decoder includes three residual blocks, three upsampling layers, and one output layer. See [link to documentation]. Figure 8 Each residual block may include two consecutive fourth convolutional layers using the Leaky Rectified Linear Unit (LeakyReLU) activation function; the three-step skip connection between the encoder and decoder is each fitted with an attention gating. See also... Figure 9 Attention gating can be applied to the input features obtained by the decoder. That is, the feature map output by the smaller residual block is used to perform... The convolution operation, while... The corresponding downsampling features of the encoder part That is, the feature map output by the larger residual block, is used to perform... The convolution operation; attention gating can add the two output features after convolution, then perform ReLU activation, and then perform [further processing] on the activation result. The convolution operation linearly transforms the number of channels in the resulting feature map to 1; then, the linearly transformed result is activated using a sigmoid function, and the activation result is resampled to obtain a 1D weight matrix with the same size as the original feature map; the weight matrix is ​​then compared with the input feature map. Multiplication returns a new feature map. As the output of attention gating.

[0130] In this embodiment of the invention, the deformation registration network includes an encoder and a decoder. The skip connection between the encoder and the decoder is realized based on the attention gating mechanism, which makes the deformation registration network pay more attention to the local features related to registration, thereby outputting a more accurate deformation field and improving the convergence speed of the registration model.

[0131] S420. Input the individual brain image and the template brain image into the registration model, and through the registration model, execute steps S4201 to S4202 to register the template brain image onto the individual brain image.

[0132] S4201. Using an affine registration network, perform affine registration with the individual brain image and the template brain image to obtain an affine transformation matrix. Then, using the first spatial transformation layer, perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0133] S4202. Using a deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. Then, using a second spatial transformation layer, spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image.

[0134] The technical solution of this invention realizes the skip connection between the encoder and decoder based on the attention gating mechanism, which can dynamically adjust the local registration, thereby ensuring that even uncommon deformations can be accurately registered. This allows the registration model to learn dense deformation fields, improves the computational efficiency and accuracy of the registration model, and enables the registration model to quickly and accurately capture local deformations, thereby achieving fast and accurate registration of brain images.

[0135] Based on any of the above embodiments, an optional technical solution utilizes a deformation registration network to perform deformation registration on intermediate brain images and individual brain images to obtain a deformation field, including: using a deformation registration network to perform deformation registration on intermediate brain images and individual brain images to obtain a velocity field, and performing differential processing on the velocity field based on the differential homeomorphism mechanism to obtain the deformation field.

[0136] The velocity field can be understood as a description of the velocity distribution obtained by deformation registration of the intermediate brain image and the individual brain image.

[0137] The differential homeomorphism mechanism can be understood as an invertible mapping between two smooth manifolds, where both the mapping and its inverse are smooth.

[0138] In an embodiment of the invention, for example, the velocity field can be differentiated by the transformation based on the differential homeomorphism mechanism in the micro-layer of the deformation registration network to obtain the deformation field.

[0139] In this embodiment of the invention, a deformation registration network can be used to perform deformation registration on intermediate brain images and individual brain images to obtain a velocity field. Based on the differential homeomorphism mechanism, the velocity field is differentiated to obtain a deformation field, thereby making the obtained deformation field smoother.

[0140] Based on any of the above embodiments, another optional technical solution utilizes a deformation registration network to perform deformation registration on intermediate brain images and individual brain images to obtain a deformation field. This includes: using a deformation registration network, based on intermediate gradient maps and individual gradient maps, to perform deformation registration on intermediate brain images and individual brain images to obtain a deformation field; wherein, the individual gradient map is the gradient map corresponding to the individual brain image, and the intermediate gradient map is obtained by using an affine transformation matrix to perform a spatial transformation of the template gradient map, and the template gradient map is the gradient map corresponding to the template brain image.

[0141] The individual gradient map can be understood as the gradient map corresponding to an individual brain image. Specifically, the individual gradient map can be understood as the image obtained by gradient processing of an individual brain image.

[0142] The intermediate gradient map can be understood as an image obtained by spatial transformation of the template gradient map using an affine transformation matrix.

[0143] The template gradient map can be understood as the gradient map corresponding to the template brain image. Specifically, the template gradient map can be understood as the image obtained by performing gradient processing on the template brain image.

[0144] It is understandable that the terms "individual gradient graph," "intermediate gradient graph," and "template gradient graph" used throughout the text are all essentially gradient graphs. They are simply different names used to distinguish gradient graphs in different application scenarios, and are not specific limitations on their actual content.

[0145] In this embodiment of the invention, by utilizing a deformation registration network, deformation registration is performed on intermediate brain images and individual brain images based on intermediate gradient maps and individual gradient maps to obtain a deformation field. In other words, the gradient map is used as a factor in obtaining the deformation field, which can improve the sensitivity of the deformation registration network to image details, thereby enhancing the registration accuracy of the registration model.

[0146] To better understand the technical solutions of the above embodiments of the present invention, an optional example is provided herein. For example, see... Figure 10 and Figure 11 The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes a key point detection network, an affine registration network, and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The registration model can be a model based on sparse key points and attention mechanisms. The registration model is an end-to-end two-stage integrated registration framework.

[0147] See Figure 10 During the training phase of the registration model, in the coarse registration phase using the coarse registration module, the individual brain image, the template brain image, the individual gradient map, and the template gradient map, as well as the registration model to be trained, can be acquired. The individual brain image, the individual gradient map, the template brain image, and the template gradient map can be input into the registration model. The registration model then registers the template brain image onto the individual brain image using the following steps: using a keypoint detection network, based on the individual gradient map, to detect keypoints of the individual brain image; and using a keypoint detection network, based on the template gradient map, to detect keypoints of the template brain image; and using an affine registration network, through the formula... Affine registration is performed from key points of the sample template to key points of individual samples to obtain the sample affine transformation matrix. The system utilizes a first spatial transformation layer to perform spatial transformation of the sample template brain image based on the affine transformation matrix A, obtaining the intermediate brain image of the sample. It also performs spatial transformation of the template gradient map based on the affine transformation matrix A, obtaining the intermediate gradient map. In the fine registration stage using the fine registration module, a deformation registration network is employed, i.e. Figure 10The deformable registration process involves performing deformation registration on the intermediate brain image and the individual brain image based on the intermediate gradient map and the individual gradient map to obtain the deformation field φ. Then, using a second spatial transformation layer, a spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image. Additionally, a spatial transformation of the intermediate gradient map is performed based on the deformation field to obtain the target gradient map. Finally, image similarity loss is calculated using image similarity loss functions based on the intermediate brain image and the individual brain image, as well as gradient similarity loss functions based on the intermediate gradient map and the individual gradient map, and finally, image similarity loss is calculated using image similarity loss functions based on the target brain image and the individual brain image. The registration model is trained using the results obtained from these calculations.

[0148] See Figure 11 During the testing phase of the registration model, the registration model can process the input and output data in the same way as the registration model in the training phase. In addition, the atlas of the sample template brain image can also be processed in the same way as the sample template brain image to obtain the intermediate atlas and the target atlas. Thus, the registration model can be evaluated based on at least one of the intermediate atlas and the target atlas.

[0149] In this embodiment of the invention, in order to evaluate the performance of the trained registration model, mainstream registration algorithms and registration models can be selected for comprehensive comparison. Mainstream registration algorithms may include, for example, Symmetric Normalization (SyN) algorithm based on the Advanced Normalization Tools (ANTs) toolkit, Free-form Ddeformation (FFD) algorithm based on Nifty image registration (Nifty_Reg), Diffeomorphic Modelling (D.Demons) algorithm, a widely used deep learning algorithm VoxelMorph, and a recently proposed keypoint-based morphological transformation registration algorithm (KeyMorph).

[0150] See Tables 1 and 2 below. Tables 1 and 2 respectively show the Dice score and Jacobian determinant values ​​of the registration model compared to mainstream registration algorithms on the Open Access Series of Imaging Studies (OASIS) dataset and the Internet Brain Segmentation Repository (IXI) dataset. The registration model achieved the highest Dice value among all methods. Compared to FFD, the registration model improved the registration accuracy by more than 20%. Although the SyN and D. Demons algorithms used differential homeomorphism constraints, which improved the registration performance to some extent, the performance of the registration model still significantly surpassed the aforementioned algorithms. Compared to VoxelMorph, the registration model improved the Dice score by more than 10% and the Jacobian non-negative percentage by more than 90%, which is due to the addition of differential homeomorphism constraints. Compared to KeyMorph, the algorithm most similar to the registration model, the registration model improved the registration accuracy by more than 6%, and the smoothness of the deformation field was comparable to KeyMorph.

[0151] Table 1. Quantitative registration results of mainstream registration algorithms on the OASIS dataset.

[0152]

[0153] Table 2. Quantitative registration results of mainstream registration algorithms on the IXI dataset.

[0154]

[0155] See Figure 12 , Figure 12 This section showcases the template-individual registration visualization results obtained from registration models and mainstream registration algorithms. The first column displays the individual brain image and a comparison image of the individual brain image overlaid with the template brain image. The second column displays the template brain image, the template-individual difference map, and the brain region atlas corresponding to the template brain image. The third column and subsequent columns represent the registration results obtained from different mainstream registration algorithms. The first row shows the registered target brain image, the second row shows the difference map between the registered target brain image and the individual brain image (brighter images indicate greater differences), and the third row shows the registered segmentation atlas (i.e., the atlas corresponding to the template brain image is registered onto the individual brain image). Within the atlas, the same brain region is colored the same. Figure 12It can be seen that the spatial misalignment between the individual brain image and the template brain image before registration is very obvious. The registration result obtained by the registration model is the most reasonable. The registration result obtained by VoxelMorph has obvious cracks. This is due to the folding of the deformation field. Among the mainstream registration algorithms, especially D.Demons, there are large errors. It is difficult to learn such large and complex deformations. For the atlas of deformation to individual space, if the same brain region (same color) has a better clustering effect, the deformation error is smaller. It can be seen that the registration model and the KeyMorph algorithm based on key point registration have obtained better results. This is because KeyMorph introduces the bending energy function in thin plate splines, which ensures the consistency of the topological structure and the smoothness of the deformation field. The registration model ensures the correctness of registration and the smoothness of the deformation field through the constraint of differential homeomorphism and attention gating.

[0156] Figure 13 This is a structural block diagram of a brain image registration apparatus provided in an embodiment of the present invention. This apparatus is used to execute the brain image registration method provided in any of the above embodiments. This apparatus and the brain image registration methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the brain image registration apparatus can be found in the embodiments of the brain image registration methods described above. See also... Figure 13 The device may specifically include: a registration model acquisition module 510, a template brain image input module 520, an intermediate brain image acquisition submodule 5201, and a target brain image acquisition submodule 5202.

[0157] The registration model acquisition module 510 is used to acquire individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, while the fine registration module includes a deformation registration network and a second spatial transformation layer. The template brain image input module 520 is used to input the individual brain image and the template brain image into the registration model, so that the template brain image is registered onto the individual brain image based on the following two sub-modules: intermediate brain image... The submodule 5201 is used to perform affine registration with the individual brain image and the template brain image using an affine registration network to obtain an affine transformation matrix, and then use a first spatial transformation layer to perform a spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image. The submodule 5202 is used to perform deformation registration with the intermediate brain image and the individual brain image using a deformation registration network to obtain a deformation field, and then use a second spatial transformation layer to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0158] Optionally, the coarse registration module also includes a keypoint detection network. The template brain image input module 520 can also be used to register the template brain image onto the individual brain image using a registration model based on the following sub-modules: a template keypoint detection sub-module, used to detect individual keypoints from the individual brain image using the keypoint detection network, and to detect template keypoints from the template brain image using the keypoint detection network; correspondingly, the intermediate brain image acquisition sub-module 5201 may include: an affine transformation matrix acquisition unit, used to perform affine registration from template keypoints to individual keypoints using an affine registration network to obtain an affine transformation matrix.

[0159] Optionally, based on the above-mentioned device, the key point detection network includes multiple first convolutional layers, multiple second convolutional layers, and pooling layers used in conjunction with each first convolutional layer. The multiple first convolutional layers and multiple second convolutional layers are alternately arranged in the key point detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0160] Optionally, based on the above-described apparatus, the apparatus may further include: a template gradient map acquisition module, used to acquire an individual gradient map corresponding to an individual brain image, and to acquire a template gradient map corresponding to a template brain image; a template brain image input module 520, which may include: a template brain image input submodule, used to input the individual brain image, the individual gradient map, the template brain image, and the template gradient map into a registration model; and a template keypoint detection submodule, which may include: a template keypoint detection unit, used to detect individual keypoints from the individual brain image based on the individual gradient map using a keypoint detection network, and to detect template keypoints from the template brain image based on the template gradient map using a keypoint detection network.

[0161] Optionally, based on the above device, the total loss function used in the training process of the registration model is at least based on the gradient similarity loss function, wherein the gradient similarity loss function is obtained from the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by using the affine transformation matrix to perform a spatial transformation of the template gradient map.

[0162] Optionally, based on the above-described device, the device may further include the following modules for pre-training the registration model: a first keypoint acquisition module, used to input the sample individual image into the original detection network with the same network structure as the keypoint detection network to obtain the first keypoint; a second keypoint acquisition module, used to perform spatial transformation of the sample individual image using the randomized random affine transformation matrix to obtain a transformed brain image, and to perform spatial transformation of the first keypoint using the randomized affine transformation matrix to obtain the second keypoint; a third keypoint acquisition module, used to input the transformed brain image into the original detection network to obtain the third keypoint; a pre-training parameter acquisition module, used to adjust the parameters in the original detection network based at least on the second and third keypoints to obtain pre-training parameters; and an initialization parameter module, used to use the pre-training parameters as initialization parameters for the registration model during training to train the registration model.

[0163] Optionally, the deformation registration network includes an encoder and a decoder, with the skip connections between the encoder and decoder implemented based on an attention gating mechanism.

[0164] Optionally, the target brain image acquisition submodule 5202 may include: a first deformation field acquisition unit, used to perform deformation registration on intermediate brain images and individual brain images using a deformation registration network to obtain a velocity field, and to perform differential processing on the velocity field based on the differential homeomorphism mechanism to obtain a deformation field.

[0165] Optionally, the target brain image acquisition submodule 5202 includes: a first deformation field acquisition unit, used to perform deformation registration on intermediate brain images and individual brain images based on intermediate gradient maps and individual gradient maps using a deformation registration network to obtain a deformation field; wherein, the individual gradient map is the gradient map corresponding to the individual brain image, and the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using an affine transformation matrix, and the template gradient map is the gradient map corresponding to the template brain image.

[0166] The brain image registration device provided in this embodiment of the invention acquires an individual brain image and a template brain image, as well as a pre-trained registration model, through a registration model acquisition module. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, while the fine registration module includes a deformation registration network and a second spatial transformation layer, thereby acquiring the individual brain image, the template brain image, and the registration model for brain image registration. A template brain image input module inputs the individual brain image and the template brain image into the registration model. Based on the following two sub-modules, the registration model registers the template brain image onto the individual brain image. End-to-end brain image registration is achieved through a registration model. A sub-module obtaining an intermediate brain image uses an affine registration network to perform affine registration with the individual brain image and the template brain image, obtaining an affine transformation matrix. A first spatial transformation layer then performs a spatial transformation of the template brain image based on the affine transformation matrix, resulting in an intermediate brain image globally aligned by the coarse registration module. A sub-module obtaining a target brain image uses a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image, obtaining a deformation field. A second spatial transformation layer then performs a spatial transformation of the intermediate brain image based on the deformation field, resulting in a target brain image precisely registered by the fine registration module. This device, through a registration model, enables brain image registration with only a simple end-to-end process, thus solving the problem of the cumbersome brain image registration process.

[0167] The brain image registration device provided in the embodiments of the present invention can execute the brain image registration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0168] It is worth noting that in the embodiments of the brain image registration device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0169] Figure 14 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0170] like Figure 14 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0171] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0172] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as brain image registration methods.

[0173] In some embodiments, the brain image registration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the brain image registration method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the brain image registration method by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0179] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0180] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A brain image registration method, characterized in that, include: The system acquires individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network, a first spatial transformation layer, and a keypoint detection network. The fine registration module includes a deformation registration network and a second spatial transformation layer. The keypoint detection network includes multiple first convolutional layers, multiple second convolutional layers, and pooling layers used in conjunction with each first convolutional layer. The multiple first convolutional layers and multiple second convolutional layers are alternately arranged in the keypoint detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer. The individual brain image and the template brain image are input into the registration model, and the template brain image is registered to the individual brain image through the registration model based on the following steps: The keypoint detection network is used to detect individual keypoints from the individual brain image, and the keypoint detection network is also used to detect template keypoints from the template brain image. The affine registration network is used to perform affine registration from the template keypoints to the individual keypoints, dynamically learning the optimal affine transformation matrix, and the first spatial transformation layer is used to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image. Using the deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. Then, using the second spatial transformation layer, spatial transformation of the intermediate brain image is performed based on the deformation field to obtain the target brain image. The registration model is pre-trained in the following manner: The individual sample image is input into the original detection network with the same network structure as the key point detection network to obtain the first key point; Using the randomized affine transformation matrix, the spatial transformation of the sample individual image is performed to obtain a transformed brain image; and using the randomized affine transformation matrix, the spatial transformation of the first key point is performed to obtain the second key point. The transformed brain image is input into the original detection network to obtain the third key point; Based at least on the second key point and the third key point, the parameters in the original detection network are adjusted to obtain pre-trained parameters; The pre-trained parameters are used as initialization parameters for the registration model during the training process to train the registration model.

2. The method according to claim 1, characterized in that, Also includes: Obtain the individual gradient map corresponding to the individual brain image, and obtain the template gradient map corresponding to the template brain image; The step of inputting the individual brain image and the template brain image into the registration model includes: The individual brain image, the individual gradient map, the template brain image, and the template gradient map are input into the registration model; The step of detecting individual key points from the individual brain image using the key point detection network, and detecting template key points from the template brain image using the key point detection network, includes: Using the keypoint detection network, individual keypoints are detected from the individual brain image based on the individual gradient map, and template keypoints are detected from the template brain image based on the template gradient map using the keypoint detection network.

3. The method according to claim 2, characterized in that, The total loss function used in the training process of the registration model is at least based on the gradient similarity loss function, wherein the gradient similarity loss function is obtained based on the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using the affine transformation matrix.

4. The method according to claim 1, characterized in that, The deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is implemented based on an attention gating mechanism.

5. The method according to claim 1, characterized in that, The process of using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field includes: Using the deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a velocity field. Based on the differential homeomorphism mechanism, the velocity field is differentiated to obtain a deformation field.

6. The method according to claim 1, characterized in that, The process of using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field includes: Using the deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image based on the intermediate gradient map and the individual gradient map to obtain the deformation field; Wherein, the individual gradient map is the gradient map corresponding to the individual brain image, and the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using the affine transformation matrix, wherein the template gradient map is the gradient map corresponding to the template brain image.

7. A brain image registration device, characterized in that, include: The registration model acquisition module is used to acquire individual brain images and template brain images, as well as a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. A template brain image input module is used to input the individual brain image and the template brain image into the registration model, so as to register the template brain image onto the individual brain image through the registration model based on the following two sub-modules: The intermediate brain image acquisition submodule is used to perform affine registration with the individual brain image and the template brain image using the affine registration network to obtain an affine transformation matrix, and to perform spatial transformation of the template brain image based on the affine transformation matrix using the first spatial transformation layer to obtain the intermediate brain image. The target brain image acquisition submodule is used to perform deformation registration on the intermediate brain image and the individual brain image using the deformation registration network to obtain a deformation field, and to perform spatial transformation of the intermediate brain image based on the deformation field using the second spatial transformation layer to obtain the target brain image. The coarse registration module further includes a keypoint detection network. The template brain image input module is further configured to register the template brain image onto the individual brain image using the registration model, based on the following sub-modules: a template keypoint detection sub-module, configured to detect individual keypoints from the individual brain image using the keypoint detection network, and to detect template keypoints from the template brain image using the keypoint detection network; correspondingly, the intermediate brain image obtaining sub-module includes: an affine transformation matrix obtaining unit, configured to perform affine registration from the template keypoints to the individual keypoints using the affine registration network, and dynamically learn the optimal affine transformation matrix; The keypoint detection network includes multiple first convolutional layers, multiple second convolutional layers, and pooling layers used in conjunction with each first convolutional layer. The multiple first convolutional layers and multiple second convolutional layers are alternately arranged in the keypoint detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer. The device may further include the following modules for pre-training the registration model: a first keypoint acquisition module, used to input a sample individual image into an original detection network with the same network structure as the keypoint detection network to obtain a first keypoint; a second keypoint acquisition module, used to perform a spatial transformation of the sample individual image using a randomized random affine transformation matrix to obtain a transformed brain image, and to perform a spatial transformation of the first keypoint using the randomized affine transformation matrix to obtain a second keypoint; a third keypoint acquisition module, used to input the transformed brain image into the original detection network to obtain a third keypoint; a pre-training parameter acquisition module, used to adjust the parameters in the original detection network based at least on the second keypoint and the third keypoint to obtain pre-training parameters; and an initialization parameter module, used to use the pre-training parameters as initialization parameters for the registration model during training to train the registration model.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to cause the at least one processor to perform the brain image registration method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the brain image registration method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the brain image registration method according to any one of claims 1-6.

Images