Brain image registration method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention discloses a brain image registration method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring an individual brain image, a template brain image and a pre-trained registration model; inputting the individual brain image and the template brain image into a registration model, and registering the template brain image to the individual brain image through the registration model based on the following steps: carrying out affine registration associated with the individual brain image and the template brain image by using an affine registration network to obtain an affine transformation matrix, and carrying out a first spatial transformation layer to obtain a second spatial transformation layer; based on the affine transformation matrix, performing spatial transformation of the template brain image to obtain an intermediate brain image; and performing deformation registration on the intermediate brain image and the individual brain image by using a deformation registration network to obtain a deformation field, and performing spatial transformation of the intermediate brain image based on the deformation field by using a second spatial transformation layer to obtain a target brain image. The problem that the brain image registration process is tedious is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of image registration, and in particular to a brain image registration method, device, electronic device, storage medium and program product. Background Art

[0002] Brain image registration is a core technology in the field of neuroimaging, and is of great significance for the early diagnosis of diseases, the evaluation of treatment plans, and the exploration of the relationship between brain function and structure in neuroscience research.

[0003] However, the current brain image registration scheme has the problem that the brain image registration process is relatively cumbersome, which needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a brain image registration method, device, electronic device, storage medium and program product, which solve the problem that the brain image registration process is relatively complicated.

[0005] According to one aspect of the present invention, a brain image registration method is provided, which may include: acquiring an individual brain image and a template brain image, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer; inputting the individual brain image and the template brain image into the registration model, so as to register the template brain image to the individual brain image through the registration model based on the following steps: using the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and using the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image; using the deformable registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and using the second spatial transformation layer to perform spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0006] According to another aspect of the present invention, there is provided a brain image registration device, which may include: a registration model acquisition module for acquiring an individual brain image, a template brain image, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer; a template brain image input module for inputting the individual brain image and the template brain image into the registration model, so as to register the template brain image onto the individual brain image through the registration model based on the following two sub-modules: an intermediate brain image obtaining sub-module for performing affine registration associated with the individual brain image and the template brain image by using the affine registration network to obtain an affine transformation matrix, and performing spatial transformation of the template brain image based on the affine transformation matrix by using the first spatial transformation layer to obtain an intermediate brain image; a target brain image obtaining sub-module for performing deformation registration on the intermediate brain image and the individual brain image by using the deformation registration network to obtain a deformation field, and performing spatial transformation of the intermediate brain image based on the deformation field by using the second spatial transformation layer to obtain a target brain image.

[0007] According to another aspect of the present invention, there is provided an electronic device, which may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is caused to implement the brain image registration method provided by any embodiment of the present invention.

[0008] According to another aspect of the present invention, there is provided a computer-readable storage medium having computer instructions stored thereon, and when the computer instructions are executed by a processor, the computer instructions are used to implement the brain image registration method provided by any embodiment of the present invention.

[0009] According to another aspect of the present invention, there is provided a computer program product including a computer program, and when the computer program is executed by a processor, the computer program is used to implement the brain image registration method provided by any embodiment of the present invention.

[0010] The technical solution of the embodiment of the present invention is to obtain an individual brain image, a template brain image, and a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer, so as to realize the acquisition of the individual brain image, the template brain image, and the registration model for brain image registration; by inputting the individual brain image and the template brain image into the registration model, through the registration model, based on the following steps, the template brain image is registered onto the individual brain image, so as to realize end-to-end brain image registration through the registration model; by using the affine registration network, affine registration associated with the individual brain image and the template brain image is performed to obtain an affine transformation matrix, and by using the first spatial transformation layer, based on the affine transformation matrix, spatial transformation of the template brain image is performed to obtain an intermediate brain image globally aligned by the coarse registration module; by using the deformation registration network, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field, and by using the second spatial transformation layer, based on the deformation field, spatial transformation of the intermediate brain image is performed to obtain a target brain image precisely registered by the fine registration module. The above technical solution can realize brain image registration through only a simple end-to-end process by using the registration model, thus solving the problem that the brain image registration process is relatively cumbersome.

[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0013] Figure 1 is a flowchart of a method for brain image registration provided according to an embodiment of the present invention.

[0014] Figure 2 is a flowchart of another method for brain image registration provided according to an embodiment of the present invention.

[0015] Figure 3 is a schematic diagram of calculating an affine transformation matrix in another method for brain image registration provided according to an embodiment of the present invention.

[0016] Figure 4 is a schematic diagram of a key point detection network in another method for brain image registration provided according to an embodiment of the present invention.

[0017] Figure 5 It is a flowchart of another brain image registration method provided according to an embodiment of the present invention.

[0018] Figure 6 It is a flowchart of yet another brain image registration method provided according to an embodiment of the present invention.

[0019] Figure 7 It is a schematic diagram of a deformation registration network in yet another brain image registration method provided according to an embodiment of the present invention.

[0020] Figure 8 It is a schematic diagram of a residual block in yet another brain image registration method provided according to an embodiment of the present invention.

[0021] Figure 9 It is a schematic diagram of an attention gate in yet another brain image registration method provided according to an embodiment of the present invention.

[0022] Figure 10 It is a schematic diagram of the training stage in an optional example of yet another brain image registration method provided according to an embodiment of the present invention.

[0023] Figure 11 It is a schematic diagram of the testing stage in an optional example of yet another brain image registration method provided according to an embodiment of the present invention.

[0024] Figure 12 It is a schematic diagram of the visualization result in an optional example of yet another brain image registration method provided according to an embodiment of the present invention.

[0025] Figure 13 It is a structural block diagram of a brain image registration device provided according to an embodiment of the present invention.

[0026] Figure 14 It is a schematic structural diagram of an electronic device for implementing the brain image registration method according to an embodiment of the present invention. Detailed implementation manners

[0027] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description, claims and the above drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. The same is true for "target", "original", etc., which will not be elaborated here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Before introducing the embodiments of the present invention, an exemplary description is first given of the application scenarios of the embodiments of the present invention, the significance of implementing the embodiments of the present invention, the implementation process of related brain image registration schemes, and the reasons for the problem that the brain image registration process is relatively cumbersome, so as to better understand the reasons why the scheme proposed by the embodiments of the present invention solves the problem that the brain image registration process is relatively cumbersome.

[0030] The template brain image provides a standardized reference framework, enabling the anatomical structures from different individuals to be compared and analyzed in a consistent form. However, although the brain has a common anatomical structure, the brain structure of each person is individualized. Therefore, in certain cases, it may be necessary to generate an individualized template brain image to more accurately reflect the uniqueness of each person's brain structure, which is crucial for understanding the differences between individuals. The assumption of the quantitative method based on the template brain image is that at a certain level of representation, the topological structure of the brain is invariant in normal individuals, and the differences between individuals are only reflected in the details of the shape of the individual brain structure. With these assumptions, the problem of localizing the anatomical structure of the individual's template brain image becomes the problem of template-to-individual brain image registration. Specifically, the goal is to map the template brain image to the space of the individual brain image through a series of spatial transformations that take into account local shape differences during the process.

[0031] Brain image registration can precisely align brain images from different individuals, different time points, and / or even different imaging techniques, enabling researchers and clinicians to analyze brain changes at a fine level. It provides important technical support for revealing the laws and mechanisms of brain development, aging, and changes under disease states, and helps promote a deeper understanding of brain science. Template-to-individual brain image registration can help identify specific regions or structures in the brain while maintaining the complete anatomical and topological information of individual images. It plays an important role in clinical practice. For example, in neurosurgery or the treatment of neurological diseases, accurate localization of the region of interest is decisive for surgical planning and treatment effects. In particular, the process of precisely aligning individual brain images with template brain images can assist doctors in more accurately determining the location of the region of interest and clarifying the relative positions of these regions with respect to brain functional areas. This is the significance of implementing the embodiments of the present invention.

[0032] Related brain image registration schemes include those based on optimized iterative registration techniques. Although such schemes have made some progress in dealing with various deformations, they are highly dependent on basic registration, resulting in a cumbersome calculation process, large computational volume, and long calculation time. To improve the above scheme, related brain image registration schemes have proposed a scheme based on a deep learning-based template-to-individual registration model. Although this scheme has achieved certain advantages in computational efficiency and registration performance, it still requires prior basic registration using a pre-alignment tool before inputting into the model. Otherwise, it is difficult to capture large or complex deformations, which affects the registration effect. However, since this scheme still requires prior basic registration using a pre-alignment tool, there is still the problem of a cumbersome brain image registration process.

[0033] In response to this, the embodiments of the present invention can perform brain image registration through a registration model, achieving brain image registration with only a simple end-to-end process, thus solving the problem of a cumbersome brain image registration process. The following will elaborate on this in detail.

[0034] Figure 1 It is a flowchart of a brain image registration method provided in the embodiments of the present invention. This embodiment is applicable to the situation of brain image registration. This method can be executed by the brain image registration device provided in the embodiments of the present invention. The device can be implemented in software and / or hardware, and can be integrated on an electronic device, which can be various user terminals or servers.

[0035] See Figure 1 , the method of the embodiments of the present invention specifically includes the following steps.

[0036] S110. Obtain an individual brain image, a template brain image, and a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer.

[0037] Among them, the individual brain image can be understood as the brain image of an individual; the individual brain image is, for example, a magnetic resonance imaging (MRI) brain image or a computed tomography (CT) brain image, etc., and no specific limitation is made here; the individual brain image can be represented in the form of a grayscale image.

[0038] The template brain image is understood as an image that serves as a template for the individual brain image; the image type of the template brain image can be the same as that of the individual brain image; the template brain image can be represented in the form of a grayscale image.

[0039] The registration model can be understood as a pre-trained model for brain image registration. The registration model can adaptively register individual brain images and template brain images with different degrees of appearance differences, and achieve precise localization of brain regions through registration, which can provide strong technical support for future medical image analysis tasks, especially in applications that require efficient and precise registration, such as radiotherapy planning, disease progression monitoring, and multimodal image fusion.

[0040] The coarse registration module can be understood as a module for coarse registration of brain images.

[0041] Correspondingly, the fine registration module can be understood as a module for precise registration of brain images. The fine registration module can be a non-linear deep registration network to further adjust local registration.

[0042] The affine registration network can be understood as a network for performing affine registration associated with the individual brain image and the template brain image. The first spatial transformation layer can be understood as a network for performing spatial transformation of the template brain image. The deformation registration network can be understood as a network for performing deformation registration on the intermediate brain image and the individual brain image. The first spatial transformation layer can be understood as a network for performing spatial transformation of the intermediate brain image.

[0043] In the embodiment of the present invention, an individual brain image, a template brain image, and a registration model including a coarse registration module and a fine registration module can be obtained. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer.

[0044] In the embodiments of the present invention, standard preprocessing steps can be performed on the individual brain image and the template brain image for skull stripping, cutting off the non-brain regions and unnecessary black background parts in the image, resampling the image to a size of 128×128×128, normalizing the gray value to the range of [0,1], and updating the individual brain image and the template brain image according to the obtained preprocessing results. Correspondingly, at least one of the images such as the subsequent-mentioned individual gradient map, template gradient map, sample individual brain image, and sample template brain image can be preprocessed as described above.

[0045] S120. Input the individual brain image and the template brain image into the registration model, and through the registration model, perform the steps of S1201~S1202 to register the template brain image onto the individual brain image.

[0046] In the embodiments of the present invention, the individual brain image and the template brain image can be input into the registration model, and through the registration model, based on the execution of the following steps of S1201~S1202, the template brain image is registered onto the individual brain image to obtain the target brain image.

[0047] S1201. Use the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and use the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0048] Among them, the affine transformation matrix can be understood as the matrix obtained by performing affine registration associated with the individual brain image and the template brain image.

[0049] The intermediate brain image can be understood as the image obtained by performing spatial transformation on the template brain image.

[0050] In the embodiments of the present invention, the affine registration network can be made to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and the first spatial transformation layer can be made to perform spatial transformation on the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0051] Exemplarily, the first spatial transformation layer can be used to perform spatial transformation of the template brain image based on the affine transformation matrix , through the formula , to perform spatial transformation of the template brain image to obtain the intermediate brain image . Among them, " " represents the spatial transformation operation.

[0052] Exemplarily, affine registration associated with the individual brain image and the template brain image can be performed to obtain an affine transformation matrix , and the first spatial transformation layer can be used to register the template brain image , through an affine transformation matrix , linearly registered to the individual brain image in the individual space to obtain an intermediate brain image, thereby achieving a global rough alignment between the template and the individual.

[0053] S1202. Use a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and use a second spatial transformation layer to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0054] Among them, the deformation field can be understood as the position obtained by performing deformation registration on the intermediate brain image and the individual brain image.

[0055] The target brain image can be understood as the registration result obtained by registering the brain image.

[0056] In the embodiment of the present invention, the deformation registration network can be used to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and the second spatial transformation layer can be used to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0057] The technical solution of the embodiment of the present invention is to obtain an individual brain image, a template brain image, and a pre-trained registration model. Among them, the registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer, so as to realize the acquisition of the individual brain image, the template brain image, and the registration model for brain image registration; by inputting the individual brain image and the template brain image into the registration model, through the registration model, based on the following steps, the template brain image is registered onto the individual brain image, so as to realize end-to-end brain image registration through the registration model; by using the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and using the first spatial transformation layer to perform a spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image globally aligned by the coarse registration module; by using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and using the second spatial transformation layer to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image precisely registered by the fine registration module. The above technical solution can realize brain image registration by only performing a simple end-to-end process through the registration model, thereby solving the problem that the brain image registration process is relatively cumbersome.

[0058] Figure 2It is a flowchart of another brain image registration method provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the rough registration module further includes a key point detection network. Based on the registration model, and further based on the following steps, the template brain image is registered onto the individual brain image: using the key point detection network to detect individual key points from the individual brain image, and using the key point detection network to detect template key points from the template brain image; correspondingly, using the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, including: using the affine registration network to perform affine registration from the template key points to the individual key points to obtain the affine transformation matrix. Among them, the explanations of the same or corresponding terms as those in the above embodiments will not be elaborated here.

[0059] See Figure 2 , the method of this embodiment may specifically include the following steps.

[0060] S210. Obtain an individual brain image, a template brain image, and a pre-trained registration model. Among them, the registration model includes a rough registration module and a fine registration module. The rough registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The rough registration module further includes a key point detection network.

[0061] Among them, the key point detection network can be understood as a network for detecting key points in an image.

[0062] It can be understood that due to the high heterogeneity between the individual brain image and the template brain image, there will be great challenges in the registration of the registration model. To address the above challenges, a key point detection network can be designed in the registration model. This key point detection network can learn the features useful for registration in the image. These features have strong resistance to noise and image quality changes, and it can make the registration model have strong robustness.

[0063] S220. Input the individual brain image and the template brain image into the registration model to perform steps S2201 to S2203 through the registration model to register the template brain image onto the individual brain image.

[0064] S2201. Use the key point detection network to detect individual key points from the individual brain image, and use the key point detection network to detect template key points from the template brain image.

[0065] Among them, the individual key points can be understood as the key points in the individual brain image, and the individual key points can be specifically understood as the key points representing the features useful for registration in the individual brain image.

[0066] The template key points can be understood as the key points in the template brain image. Specifically, the template key points can be understood as the key points that characterize the features useful for registration in the template brain image.

[0067] It can be understood that the essence of related expressions such as the first key point, the second key point, the third key point, and the fourth key point throughout the text is the key point. Here, different names are only used to distinguish the key points in different application scenarios, rather than specific limitations on their essential content.

[0068] In the embodiments of the present invention, global coarse registration can be guided by detecting key points. Since the closed-form key points can be independent of the initial position of the key points and are robust to spatial misalignment, the key points can be in closed form.

[0069] In the embodiments of the present invention, the key point detection network can extract key points useful for registration from the individual brain image and the template brain image, so as to obtain a more effective affine transformation matrix through the key points in the subsequent process.

[0070] In the embodiments of the present invention, the individual key points can be detected from the individual brain image through the key point detection network, and the template key points can be detected from the template brain image through the key point detection network.

[0071] In the embodiments of the present invention, the individual key points and the template key points can be represented in the form of point sets. For example, the key point set corresponding to the template brain image can be detected and the key point set corresponding to the individual brain image , the key point set includes the template key points, the key point set includes the individual key points, where the key point sets and are both matrices of size , that is, each image can extract key points, and each key point is a -dimensional vector.

[0072] S2202. Use the affine registration network to perform affine registration from the template key points to the individual key points to obtain an affine transformation matrix, and use the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0073] In the embodiments of the present invention, affine registration from the template key points to the individual key points can be performed through the affine registration network to obtain an affine transformation matrix.

[0074] Exemplarily, referring to Figure 3 , using the key point detection network, the key point set can be obtained and the key point set , where , represents the dimension of the key points in the key point set, that is represents that each key point is a 3D vector ; according to the key point set and the key point set , we can obtain key point pairs ; input the key point pairs into the affine registration network to utilize the affine registration network. According to the key point pairs , perform affine registration from the template key points to the individual key points through a differentiable closed expression, and calculate the affine transformation matrix to achieve the alignment of the key points; utilize the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain the intermediate brain image.

[0075] In the embodiment of the present invention, the process of calculating the affine transformation matrix is a process of dynamically learning the optimal affine transformation matrix. The objective function for learning the optimal affine transformation matrix can be , where represents the Frobenius norm, is in the homogeneous coordinate system, that is . Here, is the affine transformation matrix currently calculated during the learning process, and A here is the affine transformation matrix calculated historically during the learning process, for example, it can be the affine transformation matrix calculated last time. It can be understood that the above objective function can be derived from the following formula: ; , where of can be understood as in the above objective function.

[0076] S2203. Utilize the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain the deformation field, and utilize the second spatial transformation layer to perform spatial transformation of the intermediate brain image based on the deformation field to obtain the target brain image.

[0077] The technical solution of the embodiment of the present invention performs key point detection by using a key point detection network, obtains an affine transformation matrix based on the detected key points, and performs registration based on the stable key points, without relying on the differences in global pixel intensity, enabling the registration model to learn significant features useful for registration and enhancing the robustness of the registration model in the case of complex deformations, large deformations, and the presence of noise and uneven gray-scale distributions, thereby making the registration model have high robustness.

[0078] An alternative technical solution is that the key point detection network includes a plurality of first convolutional layers, a plurality of second convolutional layers, and pooling layers respectively used in conjunction with each first convolutional layer. The plurality of first convolutional layers and the plurality of second convolutional layers are alternately arranged in the key point detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0079] Among them, the first convolutional layer can be understood as a convolutional layer with a stride smaller than that of the second convolutional layer.

[0080] The second convolutional layer can be understood as a convolutional layer with a stride larger than that of the first convolutional layer.

[0081] It can be understood that the essence of related expressions such as the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer throughout the text is a convolutional layer. Here, different names are only used to distinguish convolutional layers in different application scenarios or with different parameters, rather than specific limitations on their essential content.

[0082] The pooling layer is a pooling layer respectively used in conjunction with the first convolutional layer.

[0083] In the embodiment of the present invention, the key point detection network includes a plurality of first convolutional layers, a plurality of second convolutional layers, and a plurality of pooling layers. The plurality of pooling layers can correspond to the plurality of first convolutional layers. Each first convolutional layer is respectively used in conjunction with a corresponding one of the plurality of pooling layers; the plurality of first convolutional layers and the plurality of second convolutional layers are alternately arranged in the key point detection network, that is, the plurality of first convolutional layers and the plurality of second convolutional layers can be interspersed with each other, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0084] It can be understood that since the center of mass layer can calculate the center of mass of the activation map in each channel output by the key point detection network, and the center of mass is the central position of the feature estimated by calculating the weighted average of the voxel intensities, the key point detection network can also include a center of mass (CoM) layer as the output layer, and the center of mass output by this center of mass layer can be used for key points.

[0085] Exemplarily, see Figure 4, the key point detection network includes 5 first convolutional layers with a stride of 1, 4 second convolutional layers with a stride of 2, pooling layers respectively used in conjunction with each first convolutional layer, and 1 centroid layer. Each convolutional layer uses a convolution kernel. Each convolutional layer uses instance normalization and the rectified linear unit (ReLU) activation function. It can be understood that compared with the first convolutional layer with a stride of 1, the second convolutional layer with a stride of 2 can reduce the size of the feature map by half, replacing the pooling layer to achieve the effect of downsampling. And since the second convolutional layer with a stride of 2 does not discard any feature information, the registration model can be made more robust, while reducing the number of parameters of the registration model and improving the convergence speed.

[0086] In the embodiment of the present invention, by making the key point detection network include multiple first convolutional layers, multiple second convolutional layers, and pooling layers respectively used in conjunction with each first convolutional layer, the multiple first convolutional layers and the multiple second convolutional layers are alternately arranged in the key point detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer, the registration model can be made more robust, while reducing the number of parameters of the registration model and improving the convergence speed.

[0087] In another alternative technical solution, the registration model is pre-trained in the following manner: for the original detection network with the same network structure as the key point detection network, input the sample individual image into the original detection network to obtain the first key points; for the randomly obtained random affine transformation matrix, use the random affine transformation matrix to perform spatial transformation on the sample individual image to obtain the transformed brain image, and use the random affine transformation matrix to perform spatial transformation on the first key points to obtain the second key points; input the transformed brain image into the original detection network to obtain the third key points; at least based on the second key points and the third key points, adjust the parameters in the original detection network to obtain the pre-trained parameters; use the pre-trained parameters as the initialization parameters during the training process of the registration model to train the registration model.

[0088] Among them, the original detection network can be understood as an untrained network with the same network structure as the key point detection network.

[0089] The sample individual image can be understood as an individual brain image used as a sample.

[0090] The first key points can be understood as the key points detected from the sample individual image using the original detection network. The second key points can be understood as the key points obtained by performing spatial transformation on the first key points. The third key points can be understood as the key points detected from the transformed brain image using the original detection network.

[0091] The random affine transformation matrix can be understood as an affine transformation matrix obtained by randomization.

[0092] The transformed brain image can be understood as an image obtained by performing a spatial transformation on the individual sample image.

[0093] The pre-trained parameters can be understood as parameters obtained by adjusting the parameters in the original detection network based at least on the second key point and the third key point.

[0094] The initialization parameters can be understood as the parameters initialized for the registration model during the training process.

[0095] In an embodiment of the present invention, in order to enhance the robustness of the registration model, a self-supervised pre-training strategy can be adopted for the original detection network to obtain pre-trained parameters, and then the pre-trained parameters are used as the initialization parameters for the registration model during the training process to train the registration model.

[0096] In an embodiment of the present invention, the individual sample image can be input into the original detection network to obtain the first key point; a random affine transformation matrix is used to perform a spatial transformation on the individual sample image to obtain the transformed brain image, and a random affine transformation matrix is used to perform a spatial transformation on the first key point to obtain the second key point; the transformed brain image is input into the original detection network to obtain the third key point; the parameters in the original detection network are adjusted based at least on the second key point and the third key point to obtain the pre-trained parameters; the pre-trained parameters are used as the initialization parameters to train the registration model.

[0097] Exemplarily, for the original detection network, the individual sample image is input into the original detection network, so that the original detection network uniformly samples on the coordinate grid where the individual sample image is located to select a set of random key points as the first key point ; for the random affine transformation matrix T randomly obtained from the uniform distribution in the parameter space, the random affine transformation matrix T is used to perform a spatial transformation on the individual sample image to obtain the transformed brain image, and the random affine transformation matrix T is used to perform a spatial transformation on the first key point to obtain the second key point; the transformed brain image is input into the original detection network to obtain the third key point; the parameters in the original detection network are adjusted based at least on the second key point and the third key point by minimizing the objective function to obtain the pre-trained parameters; the pre-trained parameters are used as the initialization parameters for the registration model during the training process to train the registration model.

[0098] In an embodiment of the present invention, the pre-trained parameters are used as the initialization parameters during the training of the registration model to train the registration model. For example, the pre-trained parameters can be used as the initialization parameters during the training of the registration model to obtain an intermediate model, and then the entire intermediate model is trained to train the registration model. For example, the intermediate model can be trained for 400 rounds, each round containing 200 iterations, the initial learning rate is set to 0.0005, and then after every 60 rounds of training, the learning rate is reduced to 0.5 times that of the previous time. The four parameters in the loss function used can be determined by grid search to obtain the optimal values, and it can be obtained that when the best result is achieved. The model training can be performed on a single NVIDIA TITIAN Xp GPU and an Intel(R) Xeon(R) CPU (v4 @2.40 GHz), and the model with the highest Dice score in the validation set is selected as the trained registration model.

[0099] In an embodiment of the present invention, to enhance the robustness of the model to image quality, such as the robustness in the presence of noise, for each training sample used to pre-train the registration model, noise can be added to a preset number or a preset proportion of the training samples. For example, random Gaussian noise within the variance range of uniform sampling [0, 0.25] can be added to one-half of the training samples, and each training sample is updated according to the result of adding noise. Among them, the training samples can at least include individual sample images.

[0100] In an embodiment of the present invention, after training the registration model, the registration model can also be tested. For example, multiple test samples can be obtained. For each test sample, 35 brain regions can be segmented from the test template image in the test sample using FreeSurfer, and the test sample is updated according to the segmentation result. The multiple test samples can be used to evaluate the registration performance of the registration model during the test phase. For example, it can be evaluated whether the affine transformation matrix and the deformation field determined by the registration model are accurate.

[0101] In an embodiment of the present invention, by determining the second key point according to the obtained first key point and obtaining the pre-trained parameters used as the initialization parameters based on the second key point and the third key point, the robustness of the registration model can be enhanced.

[0102] Figure 5It is a flowchart of another brain image registration method provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the brain image registration method further includes: obtaining an individual gradient map corresponding to the individual brain image, and obtaining a template gradient map corresponding to the template brain image; inputting the individual brain image and the template brain image into the registration model, including: inputting the individual brain image, the individual gradient map, the template brain image, and the template gradient map into the registration model; using a key point detection network to detect individual key points from the individual brain image, and using the key point detection network to detect template key points from the template brain image, including: using the key point detection network to detect individual key points from the individual brain image based on the individual gradient map, and using the key point detection network to detect template key points from the template brain image based on the template gradient map. Among them, the explanations of the same or corresponding terms in the above embodiments will not be repeated here.

[0103] See Figure 5 , the method of this embodiment may specifically include the following steps.

[0104] S310. Obtain an individual brain image, a template brain image, and a pre-trained registration model, where the registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, the fine registration module includes a deformation registration network and a second spatial transformation layer, and the coarse registration module further includes a key point detection network.

[0105] S320. Obtain an individual gradient map corresponding to the individual brain image, and obtain a template gradient map corresponding to the template brain image.

[0106] In an embodiment of the present invention, an individual gradient map and a template gradient map can be obtained to perform key point detection through the individual gradient map and the template gradient map in the subsequent process.

[0107] S330. Input the individual brain image, the individual gradient map, the template brain image, and the template gradient map into the registration model, and execute steps S3301 to S3303 to register the template brain image onto the individual brain image.

[0108] In an embodiment of the present invention, the individual gradient map and the template gradient map may be collectively referred to as gradient maps hereinafter.

[0109] It can be understood that, in order to enhance the registration accuracy and improve the sensitivity of the registration model to image details, the gradient map can also be used as an input to the registration model, which can effectively highlight the edge features in the image, and the edge features can highlight the contours and boundaries of anatomical structures. Emphasizing the edge features can help the registration model more accurately identify and align the corresponding structures between different images, thereby promoting the key point detection network to learn a more uniform and meaningful point set.

[0110] S3301. Detect individual key points from the individual brain image based on the individual gradient map using a key point detection network, and detect template key points from the template brain image based on the template gradient map using the key point detection network.

[0111] In the embodiments of the present invention, individual key points can be detected from the individual brain image based on the individual gradient map through the key point detection network, and template key points can be detected from the template brain image based on the template gradient map through the key point detection network.

[0112] Exemplarily, the individual brain image , the individual gradient map , the template brain image and the template gradient map can be input into the key point detection network, so that the key point detection network, based on the individual gradient map , detects individual key points from the individual brain image , and, based on the template gradient map , detects template key points from the template brain image .

[0113] S3302. Use an affine registration network to perform affine registration from the template key points to the individual key points to obtain an affine transformation matrix, and use a first spatial transformation layer to perform a spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0114] S3303. Use a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and use a second spatial transformation layer to perform a spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

[0115] The technical solution of the embodiments of the present invention, by detecting individual key points from the individual brain image based on the individual gradient map and detecting template key points from the template brain image through the template gradient map, that is, using the gradient map as a factor for obtaining the detected key points, can improve the sensitivity of the key point detection network to image details, thereby enhancing the registration accuracy of the registration model.

[0116] An optional technical solution is that the total loss function applied during the training process of the registration model is at least obtained based on a gradient similarity loss function, where the gradient similarity loss function is obtained according to the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using the affine transformation matrix.

[0117] Among them, the total loss function can be understood as the total loss function applied during the training process of the registration model.

[0118] The gradient similarity loss function can be understood as a similarity loss function that calculates the loss associated with the gradient.

[0119] In an embodiment of the present invention, an affine transformation matrix can be used to perform a spatial transformation on the template gradient map to obtain an intermediate gradient map. The gradient similarity loss function can be obtained based on the individual gradient map and the intermediate gradient map, and the total loss function can be obtained at least based on the gradient similarity loss function.

[0120] In an embodiment of the present invention, the total loss function can also be calculated based on at least one of a brain image similarity loss function, a fine registration loss function, and a regularization loss function.

[0121] Exemplarily, referring to the above various examples, during the training process of the registration model, the normalized cross-correlation (NCC) function can be used as the similarity metric function for both the brain image and the gradient map in the coarse registration module and the fine registration module. The regularization loss function can be used to control the smoothness of the deformation field. Specifically, for the affine registration network, the brain image similarity loss function used is And the gradient similarity loss function used is , where, and respectively represent the sample individual brain image and the sample template brain image, and respectively represent the individual gradient map and the template gradient map, and respectively represent the sample intermediate brain image and the intermediate gradient map. The affine transformation matrix A used is obtained based on the key point pairs, that is, , represents the local window centered on the voxel x; for the affine registration network, the fine registration loss function used includes and , where, are respectively the sample target brain image and the template gradient map. The deformation field involved is obtained based on the sample intermediate brain image, the sample individual brain image, and their respective corresponding gradient maps, that is, , ; the regularization loss function can be , where, represents the dense deformation field finally output by the registration model, represents the gradient of the deformation field at the voxel x; according to the above various loss functions, the total loss function can be .

[0122] In an embodiment of the present invention, the total loss function can be obtained based at least on the gradient similarity loss function, so that the gradient map can be used as a factor for obtaining the target brain image, thereby improving the sensitivity of the trained registration model to image details, and further enhancing the registration accuracy of the registration model.

[0123] Figure 6 FIG. 4 is a flowchart of another brain image registration method provided in an embodiment of the present invention. This embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is implemented based on the attention gate mechanism. The explanations of the same or corresponding terms as those in the above embodiments are not repeated here.

[0124] See Figure 6 , and the method of this embodiment can specifically include the following steps.

[0125] S410. Obtain an individual brain image, a template brain image, and a pre-trained registration model. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is implemented based on the attention gate mechanism.

[0126] Among them, skip connections can be understood as an architecture design in deep learning, mainly used to solve the problem of gradient disappearance in the registration model and help the more direct transmission of information.

[0127] The attention gate (AG) mechanism can be understood as a mechanism used to dynamically focus on specific parts of the input data in the registration model.

[0128] In an embodiment of the present invention, the deformation registration network can include an encoder and a decoder. The skip connection between the encoder and the decoder can be implemented based on the attention gate mechanism. Through the attention gate mechanism, the local registration can be dynamically adjusted, so as to learn a dense deformation field, focus on the important features of the input data of the deformation registration network, and suppress irrelevant information. That is, it can focus on the learning of local deformations of the input data of the deformation registration network. By assigning a "soft" attention weight to each feature, the deformation registration network can highlight important features and at the same time suppress unimportant features, so that the deformation registration network focuses on difficult-to-register local areas, realizes the registration of refining local details in one step, and further realizes the accurate registration of brain images.

[0129] Exemplarily, see Figure 7, the deformation registration network includes an encoder and a decoder; the encoder includes 1 input layer, 3 residual blocks, and 3 third convolutional layers with a stride of 2; the decoder includes 3 residual blocks, 3 upsampling layers, and 1 output layer, see Figure 8 , each residual block may include two consecutive fourth convolutional layers using the Leaky Rectified Linear Unit (LeakyReLU) activation function; three skip connections between the encoder and the decoder respectively add an attention gate. Among them, see Figure 9 , the attention gate can perform operations on the input features obtained from the decoder part , that is, the feature map output by the smaller residual block, and perform convolutional operations, and at the same time perform on the downsampled features of the corresponding encoder part , that is, the feature map output by the larger residual block, and perform convolutional operations; the attention gate can add the two output features after convolution, then perform ReLU activation, and perform convolutional operations on the activation result, linearly transform the number of channels of the obtained feature map to 1; then perform sigmoid activation on the obtained linear transformation result, and resample the activation result to obtain a 1D weight matrix with the same size as the original feature; multiply the weight matrix by the input feature to return a new feature map as the output of the attention gate.

[0130] In the embodiment of the present invention, the deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is realized based on the attention gate mechanism, which can make the deformation registration network pay more attention to the local features related to registration, so as to output a more accurate deformation field, and at the same time can improve the convergence speed of the registration model.

[0131] S420. Input the individual brain image and the template brain image into the registration model, and perform the steps of S4201~S4202 through the registration model to register the template brain image onto the individual brain image.

[0132] S4201. Use the affine registration network to perform affine registration related to the individual brain image and the template brain image to obtain an affine transformation matrix, and use the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image.

[0133] S4202. Use the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, and use the second spatial transformation layer to perform spatial transformation of the intermediate brain image based on the deformation field to obtain the target brain image.

[0134] In the technical solution of the embodiment of the present invention, by implementing a skip connection between the encoder and the decoder based on the attention gating mechanism, the local registration can be dynamically adjusted, so as to ensure that even uncommon deformations can be accurately registered, enabling the registration model to learn a dense deformation field, improving the computational efficiency and accuracy of the registration model, enabling the registration model to quickly and accurately capture local deformations, and further realizing the fast and accurate registration of brain images.

[0135] Based on any of the above embodiments, an alternative technical solution is to use a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, including: using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a velocity field, and based on the diffeomorphic mechanism, performing differential processing on the velocity field to obtain a deformation field.

[0136] Among them, the velocity field can be understood as a description of the velocity distribution obtained by performing deformation registration on the intermediate brain image and the individual brain image.

[0137] The diffeomorphic mechanism can be understood as a reversible mapping between two smooth manifolds, and both the mapping and its inverse mapping are smooth.

[0138] In the embodiment of the present invention, for example, the velocity field can be differentially processed through the conversion of the differential layer in the deformation registration network based on the diffeomorphic mechanism to obtain a deformation field.

[0139] In the embodiment of the present invention, a deformation registration network can be used to perform deformation registration on the intermediate brain image and the individual brain image to obtain a velocity field, and based on the diffeomorphic mechanism, perform differential processing on the velocity field to obtain a deformation field, so that the obtained deformation field can be smoother.

[0140] Based on any of the above embodiments, another alternative technical solution is to use a deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image to obtain a deformation field, including: using the deformation registration network to perform deformation registration on the intermediate brain image and the individual brain image based on the intermediate gradient map and the individual gradient map to obtain a deformation field; among them, the individual gradient map is the gradient map corresponding to the individual brain image, and the intermediate gradient map is obtained by performing a spatial transformation on the template gradient map using an affine transformation matrix, and the template gradient map is the gradient map corresponding to the template brain image.

[0141] Among them, the individual gradient map can be understood as the gradient map corresponding to the individual brain image, and specifically, the individual gradient map can be understood as the image obtained by performing gradient processing on the individual brain image.

[0142] The intermediate gradient map can be understood as the image obtained by performing a spatial transformation on the template gradient map using an affine transformation matrix.

[0143] The template gradient map can be understood as the gradient map corresponding to the template brain image. Specifically, the template gradient map can be understood as the image obtained by performing gradient processing on the template brain image.

[0144] It can be understood that the essence of related expressions such as the individual gradient map, the intermediate gradient map, and the template gradient map in the full text is the gradient map. Here, different names are given only to distinguish the gradient maps in different application scenarios, rather than specific limitations on their essential content.

[0145] In the embodiment of the present invention, by using a deformation registration network, based on the intermediate gradient map and the individual gradient map, deformation registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field. That is, the gradient map is used as a factor for obtaining the deformation field, which can improve the sensitivity of the deformation registration network to image details, thereby enhancing the registration accuracy of the registration model.

[0146] To better understand the technical solution of the above embodiment of the present invention, an optional example is provided here. Exemplarily, referring to Figure 10 and Figure 11 , the registration model includes a coarse registration module and a fine registration module. The coarse registration module includes a key point detection network, an affine registration network, and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The registration model can be a model based on sparse key points and an attention mechanism, and the registration model is an end-to-end two-stage integrated registration framework.

[0147] Referring to Figure 10 , in the training stage of the registration model, in the coarse registration stage using the coarse registration module, a sample individual brain image, a sample template brain image, an individual gradient map, a template gradient map, and the registration model to be trained can be obtained; the sample individual brain image, the individual gradient map, the sample template brain image, and the template gradient map can be input into the registration model, so that through the registration model, based on the following steps, the sample template brain image is registered onto the sample individual brain image: using the key point detection network, based on the individual gradient map, detecting sample individual key points from the sample individual brain image, and using the key point detection network, based on the template gradient map, detecting sample template key points from the sample template brain image; using the affine registration network, through the formula , performing affine registration from the sample template key points to the sample individual key points to obtain a sample affine transformation matrix , and using the first spatial transformation layer, based on the affine transformation matrix A, performing spatial transformation on the sample template brain image to obtain a sample intermediate brain image, and, based on the affine transformation matrix A, performing spatial transformation on the template gradient map to obtain an intermediate gradient map; in the fine registration stage using the fine registration module, using the deformation registration network, that is Figure 10In the deformable registration, according to the intermediate gradient map and the individual gradient map, deformable registration is performed on the sample intermediate brain image and the sample individual brain image to obtain a deformation field 𝝓. Then, using the second spatial transformation layer, based on the deformation field, spatial transformation of the sample intermediate brain image is performed to obtain the sample target brain image, and based on the deformation field, spatial transformation of the intermediate gradient map is performed to obtain the target gradient map; based on the sample intermediate brain image and the sample individual brain image, image similarity loss calculation is performed through an image similarity loss function, based on the intermediate gradient map and the individual gradient map, gradient similarity loss calculation is performed through a gradient similarity loss function, based on the individual gradient map and the template gradient map, gradient similarity loss calculation is performed through a gradient similarity loss function, based on the sample target brain image and the sample individual brain image, image similarity loss calculation is performed through an image similarity loss function, and based on the calculation results obtained from the above calculations, a registration model is trained.

[0148] See Figure 11 , in the test stage of the registration model, the way the registration model processes the input and output data can be the same as the way the registration model processes the input and output data in the above training stage. In addition, the atlas of the sample template brain image can also be processed in the same process as the sample template brain image is processed to obtain an intermediate atlas and a target atlas, so that the registration model can be evaluated based on at least one of the intermediate atlas and the target atlas.

[0149] In the embodiments of the present invention, in order to evaluate the performance of the trained registration model, mainstream registration algorithms can be selected for comprehensive comparison with the registration model. Mainstream registration algorithms can include, for example, the Symmetric Normalization (SyN) algorithm based on the Advanced Normalization Tools (ANTs) toolkit, the Free-form Ddeformation (FFD) algorithm based on Nifty Registration (Nifty_Reg), the Diffeomorphic Modelling (D.Demons) algorithm, a widely used deep learning algorithm VoxelMorph, and a recently proposed method based on keypoint registration, the Keypoint-based Morphing Registration Algorithm (KeyMorph).

[0150] See Table 1 and Table 2 below. Table 1 and Table 2 respectively show the Dice scores and Jacobian determinant values of the registration model and the mainstream registration algorithms on the Open Access Series of Imaging Studies (OASIS) dataset and the Internet Brain Segmentation Repository (IXI) dataset. The registration model achieved the highest Dice value among all methods. Compared with FFD, the registration accuracy of the registration model was improved by more than 20%. Although the two algorithms, SyN and D.Demons, used the constraint of diffeomorphism and improved the registration performance to a certain extent, the performance of the registration model still significantly exceeded the above algorithms. Compared with VoxelMorph, the registration model was improved by more than 10% in Dice score and more than 90% in the percentage of non-negative Jacobians. This was because the registration model added the constraint of diffeomorphism. Compared with the algorithm KeyMorph, which was the most similar to the registration model, the registration accuracy of the registration model was improved by more than 6%, and the smoothness of the deformation field was comparable to that of KeyMorph.

[0151] Table 1 Quantitative registration results of mainstream registration algorithms on the OASIS dataset

[0152]

[0153] Table 2 Quantitative registration results of mainstream registration algorithms on the IXI dataset

[0154]

[0155] See Figure 12 , Figure 12 shows the visualization results of template-to-individual registration obtained by the registration model and the mainstream registration algorithms. Among them, the first column shows the individual brain images and the superimposed comparison images of the individual brain images and the template brain images respectively. The second column shows the template brain image, the template-individual difference image, and the brain region atlas corresponding to the template brain image. The third column and subsequent columns show the registration result images obtained by different mainstream registration algorithms. The first row represents the registered target brain image, the second row represents the difference image between the registered target brain image and the individual brain image (the brighter the greater the difference), and the third row is the registered segmentation atlas (that is, the atlas corresponding to the template brain image is registered to the individual brain image). The same brain region has the same color in the atlas; from Figure 12It can be seen that the spatial misalignment between the individual brain image and the template brain image before registration is very obvious. The registration result obtained by the registration model is the most reasonable. There are obvious cracks in the registration result obtained by VoxelMorph, which is caused by the folding of the deformation field. In the mainstream registration algorithms, especially D.Demons, there are large errors and it is difficult to learn such large and complex deformations. For the atlas deformed to the individual space, if the clustering effect of the same brain region (the same color) is better, the deformation error is smaller. It can be seen that the registration model and the KeyMorph algorithm based on key point registration have obtained better results. This is because KeyMorph introduces the bending energy function in the thin plate spline to ensure the consistency of the topological structure and the smoothness of the deformation field. The registration model ensures the correctness of registration and the smoothness of the deformation field through the constraint of diffeomorphism and attention gating.

[0156] Figure 13 FIG. is a structural block diagram of a brain image registration device provided by an embodiment of the present invention. The device is used to execute the brain image registration method provided in any of the above embodiments. The device and the brain image registration methods in the above embodiments belong to the same inventive concept. For the details not described in detail in the embodiment of the brain image registration device, reference can be made to the embodiments of the above brain image registration method. Refer to Figure 13 , and the device may specifically include: a registration model acquisition module 510, a template brain image input module 520, an intermediate brain image obtaining sub-module 5201, and a target brain image obtaining sub-module 5202.

[0157] Among them, the registration model acquisition module 510 is configured to acquire an individual brain image and a template brain image, and a pre-trained registration model, where the registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer. The fine registration module includes a deformation registration network and a second spatial transformation layer. The template brain image input module 520 is configured to input the individual brain image and the template brain image into the registration model, so as to register the template brain image onto the individual brain image through the registration model based on the following two sub-modules: The intermediate brain image obtaining sub-module 5201 is configured to perform affine registration associated with the individual brain image and the template brain image by using the affine registration network to obtain an affine transformation matrix, and perform spatial transformation of the template brain image based on the affine transformation matrix by using the first spatial transformation layer to obtain an intermediate brain image. The target brain image obtaining sub-module 5202 is configured to perform deformation registration on the intermediate brain image and the individual brain image by using the deformation registration network to obtain a deformation field, and perform spatial transformation of the intermediate brain image based on the deformation field by using the second spatial transformation layer to obtain a target brain image.

[0158] Optionally, the rough registration module further includes a key point detection network. The template brain image input module 520 can also be used to register the template brain image to the individual brain image through the registration model based on the following sub-modules: a template key point detection sub-module, which is used to detect individual key points from the individual brain image by using the key point detection network, and detect template key points from the template brain image by using the key point detection network; correspondingly, the intermediate brain image obtaining sub-module 5201 may include: an affine transformation matrix obtaining unit, which is used to perform affine registration from the template key points to the individual key points by using the affine registration network to obtain an affine transformation matrix.

[0159] Optionally, on the basis of the above device, the key point detection network includes a plurality of first convolutional layers, a plurality of second convolutional layers, and a pooling layer used in combination with each first convolutional layer respectively. The plurality of first convolutional layers and the plurality of second convolutional layers are alternately arranged in the key point detection network, and the stride of the first convolutional layer is smaller than the stride of the second convolutional layer.

[0160] Optionally, on the basis of the above device, the device may further include: a template gradient map acquisition module, which is used to acquire the individual gradient map corresponding to the individual brain image and the template gradient map corresponding to the template brain image; the template brain image input module 520 may include: a template brain image input sub-module, which is used to input the individual brain image, the individual gradient map, the template brain image, and the template gradient map into the registration model; the template key point detection sub-module may include: a template key point detection unit, which is used to detect individual key points from the individual brain image based on the individual gradient map by using the key point detection network, and detect template key points from the template brain image based on the template gradient map by using the key point detection network.

[0161] Optionally, on the basis of the above device, the total loss function applied in the training process of the registration model is at least obtained based on the gradient similarity loss function, where the gradient similarity loss function is obtained according to the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by performing spatial transformation of the template gradient map by using the affine transformation matrix.

[0162] Optionally, based on the above device, the device may further include the following modules for pre-training a registration model: a first key point obtaining module, configured to input a sample individual image into an original detection network with the same network structure as the key point detection network to obtain a first key point; a second key point obtaining module, configured to perform a spatial transformation on the sample individual image by using a randomly generated random affine transformation matrix to obtain a transformed brain image, and perform a spatial transformation on the first key point by using the random affine transformation matrix to obtain a second key point; a third key point obtaining module, configured to input the transformed brain image into the original detection network to obtain a third key point; a pre-training parameter obtaining module, configured to adjust parameters in the original detection network based on at least the second key point and the third key point to obtain pre-training parameters; and an initial parameter as module, configured to use the pre-training parameters as the initial parameters during the training of the registration model to train the registration model.

[0163] Optionally, the deformation registration network includes an encoder and a decoder, and the skip connection between the encoder and the decoder is implemented based on an attention gating mechanism.

[0164] Optionally, the target brain image obtaining sub-module 5202 may include: a first deformation field obtaining unit, configured to perform deformation registration on the intermediate brain image and the individual brain image by using the deformation registration network to obtain a velocity field, and perform differential processing on the velocity field based on a diffeomorphic mechanism to obtain a deformation field.

[0165] Optionally, the target brain image obtaining sub-module 5202 includes: a first deformation field obtaining unit, configured to perform deformation registration on the intermediate brain image and the individual brain image based on an intermediate gradient map and an individual gradient map by using the deformation registration network to obtain a deformation field; where the individual gradient map is the gradient map corresponding to the individual brain image, and the intermediate gradient map is obtained by performing a spatial transformation on a template gradient map by using an affine transformation matrix, and the template gradient map is the gradient map corresponding to the template brain image.

[0166] The brain image registration device provided by the embodiment of the present invention obtains an individual brain image, a template brain image, and a pre-trained registration model through a registration model acquisition module. The registration model includes a coarse registration module and a fine registration module. The coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformation registration network and a second spatial transformation layer, so as to realize the acquisition of the individual brain image, the template brain image, and the registration model for brain image registration; through the template brain image input module, the individual brain image and the template brain image are input into the registration model, so that through the registration model, based on the following two sub-modules, the template brain image is registered onto the individual brain image, so as to realize end-to-end brain image registration through the registration model; through the intermediate brain image obtaining sub-module, an affine registration associated with the individual brain image and the template brain image is performed using the affine registration network to obtain an affine transformation matrix, and using the first spatial transformation layer, based on the affine transformation matrix, a spatial transformation of the template brain image is performed to obtain an intermediate brain image globally aligned by the coarse registration module; through the target brain image obtaining sub-module, a deformation registration is performed on the intermediate brain image and the individual brain image using the deformation registration network to obtain a deformation field, and using the second spatial transformation layer, based on the deformation field, a spatial transformation of the intermediate brain image is performed to obtain a target brain image precisely registered by the fine registration module. The above device can perform brain image registration through the registration model, and can realize brain image registration by only performing a simple end-to-end process, thus solving the problem that the brain image registration process is relatively cumbersome.

[0167] The brain image registration device provided by the embodiment of the present invention can execute the brain image registration method provided by any embodiment of the present invention, and has corresponding function modules and beneficial effects for executing the method.

[0168] It should be noted that in the embodiments of the above brain image registration device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0169] Figure 14 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0170] As shown Figure 14 in FIG. 1, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0171] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0172] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the brain image registration method.

[0173] In some embodiments, the brain image registration method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the brain image registration method described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the brain image registration method by any other appropriate means (e.g., by means of firmware).

[0174] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0175] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0176] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0178] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0179] The computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0180] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0181] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A brain image registration method, characterized in that: include: Acquire an individual brain image and a template brain image, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformable registration network and a second spatial transformation layer; The individual brain image and the template brain image are input into the registration model, so as to register the template brain image to the individual brain image through the registration model based on the following steps: Using the affine registration network, perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and using the first spatial transformation layer, based on the affine transformation matrix, perform spatial transformation of the template brain image to obtain an intermediate brain image; The deformable registration network is used to perform deformable registration on the intermediate brain image and the individual brain image to obtain a deformation field, and the second spatial transformation layer is used to perform spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

2. The method according to claim 1, characterized in that The coarse registration module further includes a key point detection network to register the template brain image to the individual brain image through the registration model, further based on the following steps: Detecting individual key points from the individual brain image using the key point detection network, and detecting template key points from the template brain image using the key point detection network; Accordingly, the affine registration network is used to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, including: The affine registration network is used to perform affine registration from the template key points to the individual key points to obtain an affine transformation matrix.

3. The method according to claim 2, characterized in that The key point detection network includes multiple first convolutional layers, multiple second convolutional layers and a pooling layer used in conjunction with each of the first convolutional layers. The multiple first convolutional layers and the multiple second convolutional layers are alternately arranged in the key point detection network, and the step size of the first convolutional layer is smaller than the step size of the second convolutional layer.

4. The method according to claim 2, characterized in that: Also includes: Acquiring an individual gradient map corresponding to the individual brain image, and acquiring a template gradient map corresponding to the template brain image; The step of inputting the individual brain image and the template brain image into the registration model comprises: inputting the individual brain image, the individual gradient map, the template brain image, and the template gradient map into the registration model; The step of detecting individual key points from the individual brain image using the key point detection network, and detecting template key points from the template brain image using the key point detection network, comprises: Using the key point detection network, based on the individual gradient map, individual key points are detected from the individual brain image, and using the key point detection network, based on the template gradient map, template key points are detected from the template brain image.

5. The method according to claim 4, characterized in that The total loss function applied during the training process of the registration model is obtained based on at least a gradient similarity loss function, wherein the gradient similarity loss function is obtained based on the individual gradient map and the intermediate gradient map, and the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using the affine transformation matrix.

6. The method according to claim 2, characterized in that The registration model is pre-trained in the following way: For an original detection network having the same network structure as the key point detection network, inputting the sample individual image into the original detection network to obtain a first key point; For the random affine transformation matrix obtained by randomization, using the random affine transformation matrix, performing spatial transformation of the sample individual image to obtain a transformed brain image, and using the random affine transformation matrix, performing spatial transformation of the first key point to obtain a second key point; Inputting the transformed brain image into the original detection network to obtain a third key point; At least based on the second key point and the third key point, adjusting parameters in the original detection network to obtain pre-trained parameters; The pre-training parameters are used as initialization parameters of the registration model during the training process to train the registration model.

7. The method according to claim 1, characterized in that The deformable registration network includes an encoder and a decoder, and the jump connection between the encoder and the decoder is implemented based on an attention gating mechanism.

8. The method according to claim 1, characterized in that: The method of using the deformable registration network to perform deformable registration on the intermediate brain image and the individual brain image to obtain a deformation field includes: The deformable registration network is used to perform deformable registration on the intermediate brain image and the individual brain image to obtain a velocity field, and based on a differential homeomorphism mechanism, the velocity field is differentiated to obtain a deformation field.

9. The method according to claim 1, characterized in that: The method of using the deformable registration network to perform deformable registration on the intermediate brain image and the individual brain image to obtain a deformation field includes: Using the deformable registration network, based on the intermediate gradient map and the individual gradient map, deformable registration is performed on the intermediate brain image and the individual brain image to obtain a deformation field; The individual gradient map is a gradient map corresponding to the individual brain image, the intermediate gradient map is obtained by performing a spatial transformation of the template gradient map using the affine transformation matrix, and the template gradient map is a gradient map corresponding to the template brain image.

10. A brain image registration device, characterized in that: include: A registration model acquisition module, used to acquire individual brain images and template brain images, and a pre-trained registration model, wherein the registration model includes a coarse registration module and a fine registration module, the coarse registration module includes an affine registration network and a first spatial transformation layer, and the fine registration module includes a deformable registration network and a second spatial transformation layer; The template brain image input module is used to input the individual brain image and the template brain image into the registration model, so as to register the template brain image to the individual brain image through the registration model based on the following two submodules: The intermediate brain image obtaining submodule is used to use the affine registration network to perform affine registration associated with the individual brain image and the template brain image to obtain an affine transformation matrix, and use the first spatial transformation layer to perform spatial transformation of the template brain image based on the affine transformation matrix to obtain an intermediate brain image; The target brain image obtaining submodule is used to use the deformable registration network to perform deformable registration on the intermediate brain image and the individual brain image to obtain a deformation field, and use the second spatial transformation layer to perform spatial transformation of the intermediate brain image based on the deformation field to obtain a target brain image.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the brain image registration method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the brain image registration method according to any one of claims 1 to 9 when executed.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the brain image registration method according to any one of claims 1 to 9.

Images